@@ -8,7 +8,8 @@ This directory provides production-grade scripts to back up Qdrant snapshots and
   a newer one via Homebrew for local dev/testing. The `pharia-helper` container image ships
   Bash 5.x.)
 - **curl**
-- **jq** (>= 1.6 — the per-shard manifest/config-mapping jq filters use features not present in older jq;
+- **jq** (>= 1.7 — REQUIRED and gated at per-shard task entry: jq 1.6 parses 64-bit peer ids
+  as IEEE doubles and rounds above 2^53, so every peer lookup fails against real clusters;
   1.7+ is recommended for precise handling of large integers such as peer ids. The
   `pharia-helper` image ships jq 1.8.x.)
 - **mc** (MinIO Client)
@@ -386,16 +387,14 @@ manifest's `backup_set_id` only.
 
 ### Per-shard restore peer discovery
 
-**One target cluster per run:** only the FIRST `QDRANT_RESTORE_HOSTS` entry's cluster is
-restored to; extra entries are ignored with a WARNING naming the count. Run once per
-cluster (legacy `recover_snap`, by contrast, fanned out to every listed host).
+**One target cluster per run:** only the FIRST `QDRANT_RESTORE_HOSTS` entry is restored
+to (extras warned and ignored) — run once per cluster. See RUNBOOK.md, Restore procedure.
 
 `recover_snap_shards` always discovers the **target** cluster's peers via
 `QDRANT_RESTORE_HOSTS`, unconditionally — it does not consult `GET_PEERS_FROM_CLUSTER_INFO` at
 all (that variable only still affects the legacy dispatch path). If you deploy this on
-Kubernetes, `k8s/restore-job.yaml` sets `GET_PEERS_FROM_CLUSTER_INFO=true` for exactly this
-reason; see that file's comment if you also run the legacy `recover_snap` task from the same
-manifest.
+Kubernetes, use `k8s/restore-per-shard-job.yaml` (the self-consistent per-shard restore
+manifest); `k8s/restore-job.yaml` stays fully legacy.
 
 ## Common Workflows
 
 
@@ -159,10 +159,9 @@ This becomes the **quarterly canary** thereafter — do not treat it as a one-ti
 
 ### Step 4 — CronJob cutover
 
-Only after Steps 1–3 have passed. This repo's `k8s/backup-cronjob.yaml` already carries the env
-and Job-policy changes staged (S3 credentials, `MC_CONFIG_DIR`, `backoffLimit: 0`,
-`activeDeadlineSeconds`, `restartPolicy: Never`) — the cutover itself is the one-line command
-flip:
+Only after Steps 1–3 have passed. This repo's `k8s/backup-cronjob.yaml` carries the per-shard
+ENV staged (S3 credentials, `MC_CONFIG_DIR`) but deliberately keeps the LEGACY Job policy —
+the cutover is the command flip PLUS the three Job-policy values, applied together:
 
 ```diff
               args:
@@ -171,6 +170,13 @@ flip:
 +                - "cd scripts && ./qdrant_backup_recovery.sh create_snap_shards"
 ```
 
+and in the same edit set `backoffLimit: 0`, `restartPolicy: Never`, and
+`activeDeadlineSeconds: 7200`. Why together: per-shard exits nonzero on any partial failure
+while still pruning the collections that succeeded, so a retrying Job (`backoffLimit > 0` or
+`OnFailure`) can create+prune several sets back-to-back within ONE firing and collapse
+retention history to near-duplicates minutes apart; conversely those values must never apply
+while the legacy command still runs (legacy relies on retries and has no runtime ceiling).
+
 **Pre-flight gate — before touching anything, confirm the deployed secret is ready:**
 
 ```bash
@@ -184,8 +190,8 @@ deployment.** This file carries repo placeholders — `QDRANT_SOURCE_HOSTS:
 URL), and the secret name `qdrant-credentials` — and `apply` replaces the entire `env:` list, so
 a site that customized its headless-service name or S3 endpoint would have those values
 **reverted to placeholders**, and the first per-shard firing would fail at `mc alias set`. **Port the
-command flip above, together with the env and Job-policy additions already staged in this
-file, into your site's own deployed manifest or values overlay, then apply *that*.**
+command flip + Job-policy trio above, together with the env additions staged in this file,
+into your site's own deployed manifest or values overlay, then apply *that*.**
 
 This single swap also activates automatic retention (`create_snap_shards` runs `prune_snap`'s
 retention pass on every successful run) — there is no separate retention deploy step.
@@ -241,16 +247,15 @@ any flag:**
 | Exists, empty, **or** non-empty with resume history matching THIS exact set id | Resumes — creation skipped, only shards not yet recovered are recovered. | None. |
 | Exists, non-empty, no matching history for this set (or config-incompatible) | Aborts. | `QDRANT_RESTORE_FORCE=true` — destructive: deletes and recreates the collection. |
 
-**Launching a restore in Kubernetes — flip BOTH together, or neither.** The shipped
-`k8s/restore-job.yaml`'s `args` is still legacy `recover_snap` while its env is staged for per-shard.
-Running legacy `recover_snap` as shipped will exit immediately at dispatch —
-`GET_PEERS_FROM_CLUSTER_INFO=true` combined with `QDRANT_SOURCE_HOSTS=""` leaves legacy peer
-discovery with no host to query (see the manifest's own comment). To actually run
-`recover_snap_shards`:
+**Launching a restore in Kubernetes.** The per-shard restore ships as its own
+internally-consistent manifest — no field flipping required:
+
+```bash
+kubectl -n <namespace> create -f k8s/restore-per-shard-job.yaml
+```
 
-1. Set `args` to run `recover_snap_shards` instead of `recover_snap`.
-2. Confirm `GET_PEERS_FROM_CLUSTER_INFO=true` (already the case in the shipped file).
-3. `kubectl -n <namespace> create -f k8s/restore-job.yaml`
+(`k8s/restore-job.yaml` remains the fully-legacy `recover_snap` manifest for legacy-era
+snapshots; the two are independent and each runs its own default command as shipped.)
 
 **Restore requirements — per-shard restore has no static-hosts mode.** `recover_snap_shards`
 hard-requires (a) a **cluster-mode** target whose `GET /cluster` peer discovery works on the
 
@@ -265,13 +265,9 @@ documented in the main [README.md](../README.md#per-shard-backup--restore); the
 sequence (day-0 unblock, restore procedure, verification-failure remediation, transition rules)
 is in [RUNBOOK.md](../RUNBOOK.md).
 
-**Cutover status:** `backup-cronjob.yaml` and `restore-job.yaml` ship with the per-shard env vars and
-Job-policy fields already in place (S3 credentials, `MC_CONFIG_DIR`, `backoffLimit: 0`,
-`activeDeadlineSeconds`, `restartPolicy: Never`, and — on the restore side —
-`GET_PEERS_FROM_CLUSTER_INFO=true`), but the `args:` command in both is still the legacy task.
-Flipping `create_snap` → `create_snap_shards` (and, when ready, `recover_snap` →
-`recover_snap_shards`) is a deliberate, gated step — see RUNBOOK.md — not something that
-happens just because the env is staged for it.
+**Cutover status:** `backup-cronjob.yaml` stages per-shard env but keeps legacy args and
+Job policy — the cutover flip applies both together (RUNBOOK.md Step 4). The per-shard
+restore is its own manifest, `restore-per-shard-job.yaml`; `restore-job.yaml` stays legacy.
 
 ### Per-shard environment variables
 
 
@@ -5,12 +5,8 @@ metadata:
   labels:
     app: qdrant-backup
 spec:
-  # Cron schedule (min hour day month weekday)
-  # "0 0 */7 * 0" restricts BOTH day-of-month and day-of-week — cron ORs them when both are
-  # restricted, so this actually fires at 00:00 on the 1st/8th/15th/22nd/29th AND every Sunday
-  # (~9-10x/month), NOT weekly (the schedule VALUE itself is intentionally
-  # unchanged — changing it is deferred to the cutover decision). For a plain weekly schedule, use
-  # "0 0 * * 0" instead (day-of-month unrestricted, fires only on Sunday).
+  # NOTE: cron ORs restricted day-of-month AND day-of-week, so this fires
+  # ~9-10x/month, not weekly ("0 0 * * 0" would be plain weekly); value kept.
   schedule: "0 0 */7 * 0"
 
   # Optional: prevent overlapping jobs
@@ -27,51 +23,22 @@ spec:
   failedJobsHistoryLimit: 3
   jobTemplate:
     spec:
-      # backoffLimit MUST stay 0 once this CronJob runs create_snap_shards (per-shard):
-      # create_snap_shards exits nonzero on ANY partial-collection failure but
-      # still runs its automatic retention pass on the collections that DID
-      # succeed. With backoffLimit > 0, the Job controller retries by
-      # spawning fresh pods within the SAME scheduled firing — each retry
-      # creates another backup set and prunes again, so a single flaky
-      # firing can create+prune several sets back-to-back and collapse
-      # QDRANT_BACKUP_RETENTION_SETS worth of history down to near-duplicate
-      # sets from minutes apart instead of separate nights (found in review).
-      # 0 means: one attempt, ever; a failure is reported, not silently
-      # retried into a retention-collapsing loop.
-      backoffLimit: 0
-      # Hard ceiling on how long one firing may run, independent of
-      # CURL_TIMEOUT (below): CURL_TIMEOUT bounds a SINGLE curl call and is
-      # retried up to ~3x per shard on timeout/`accepted`, so a
-      # hung peer can consume a multiple of CURL_TIMEOUT before that shard's
-      # collection is abandoned; activeDeadlineSeconds is the outer backstop
-      # so a genuinely stuck run is killed rather than blocking the next
-      # scheduled firing indefinitely (concurrencyPolicy: Forbid above).
-      activeDeadlineSeconds: 7200
+      # legacy values — at per-shard cutover set backoffLimit 0 / restartPolicy Never / activeDeadlineSeconds 7200 (RUNBOOK Step 4)
+      backoffLimit: 3
       template:
         metadata:
           labels:
             app: qdrant-backup
         spec:
-          # Never (not OnFailure): OnFailure lets the kubelet respawn the
-          # SAME container in place on failure, which is exactly the kind of
-          # extra attempt backoffLimit: 0 above is meant to rule out for this
-          # job. Matches backup-job.yaml/restore-job.yaml, which already pair
-          # backoffLimit: 0 with restartPolicy: Never for the same reason.
-          restartPolicy: Never
+          restartPolicy: OnFailure
           containers:
             - name: qdrant-backup
               image: ghcr.io/aleph-alpha/shared-images/pharia-helper:latest
               imagePullPolicy: IfNotPresent
               securityContext:
                 readOnlyRootFilesystem: true
-              # Gated cutover (runbook Day-0 sequence): this still
-              # runs the legacy per-node `create_snap` task. The per-shard
-              # task is `create_snap_shards` (adds manifest + automatic
-              # retention). Flipping the command below to
-              # `create_snap_shards` is the actual cutover step — do NOT
-              # flip it in this commit; it is gated on the rollout gates
-              # in RUNBOOK.md (day-0 legacy prune -> shadow run ->
-              # restore canary -> THEN this flip).
+              # still LEGACY create_snap — the flip to create_snap_shards is the
+              # gated cutover step (RUNBOOK Day-0 sequence)
               command:
                 - /bin/bash
               args:
@@ -91,24 +58,9 @@ spec:
                   value: "true"
                 - name: CURL_TIMEOUT
                   value: "3000" # increase if more time is needed for a single backup
-                # S3 credentials: needed once the command above is cut over to
-                # create_snap_shards (manifest upload, `mc stat` verification,
-                # prune_snap). Staging these now is what makes the
-                # cutover a pure command-string change (see the args comment
-                # above) instead of a second env-adding deploy.
-                #
-                # DELIBERATE DEVIATION:
-                # the three refs below are `optional: true`. This env is staged
-                # for per-shard while `args` above still runs LEGACY create_snap, which
-                # never needed these keys — a deployed `qdrant-credentials`
-                # secret that does not yet carry them must not
-                # CreateContainerConfigError the still-running legacy backup.
-                # The Step 4 pre-flight secret gate + Step 2 shadow run
-                # (RUNBOOK.md) catch missing creds before cutover to
-                # create_snap_shards, which DOES need them non-optionally at
-                # that point. QDRANT_API_KEY above keeps its required
-                # (non-optional) semantics — it was already required by the
-                # legacy task this CronJob still runs.
+                # S3 creds staged for the per-shard cutover; optional: true so a
+                # secret without them cannot break the still-legacy backup (the
+                # RUNBOOK Step-4 gate verifies them before the flip)
                 - name: QDRANT_S3_ACCESS_KEY_ID
                   valueFrom:
                     secretKeyRef:
@@ -144,14 +96,7 @@ spec:
                 subPath: qdrant_backup_recovery.sh
               - name: scripts
                 mountPath: /scripts
-              # Defense-in-depth alongside the script's own TMPDIR default
-              # (TMPDIR defaults to $PWD == /scripts, already writable via
-              # the "scripts" emptyDir above): readOnlyRootFilesystem means a
-              # bare /tmp write from `mc`, curl, or any tool that doesn't go
-              # through the script's TMPDIR-aware mktemp calls would
-              # otherwise fail hard (empirically confirmed: "mktemp: :
-              # Read-only file system" with no /tmp mount under this
-              # securityContext).
+              # under readOnlyRootFilesystem, tools writing bare /tmp need this mount
               - name: tmp
                 mountPath: /tmp
           volumes:
 
@@ -29,22 +29,22 @@ spec:
             - name: QDRANT_S3_ACCESS_KEY_ID
               valueFrom:
                 secretKeyRef:
-                  name: qdrant-credentials-minio # change this to your qdrant kubernetes secret name
+                  name: qdrant-credentials # change this to your qdrant kubernetes secret name
                   key: QDRANT_S3_ACCESS_KEY_ID
             - name: QDRANT_S3_SECRET_ACCESS_KEY
               valueFrom:
                 secretKeyRef:
-                  name: qdrant-credentials-minio # change this to your qdrant kubernetes secret name
+                  name: qdrant-credentials # change this to your qdrant kubernetes secret name
                   key: QDRANT_S3_SECRET_ACCESS_KEY
             - name: QDRANT_S3_BUCKET_NAME
               valueFrom:
                 secretKeyRef:
-                  name: qdrant-credentials-minio # change this to your qdrant kubernetes secret name
+                  name: qdrant-credentials # change this to your qdrant kubernetes secret name
                   key: QDRANT_S3_BUCKET_NAME
             - name: QDRANT_API_KEY
               valueFrom:
                 secretKeyRef:
-                  name: qdrant-credentials-minio # change this to your qdrant kubernetes secret name
+                  name: qdrant-credentials # change this to your qdrant kubernetes secret name
                   key: QDRANT_API_KEY
             - name: QDRANT_S3_ENDPOINT_URL
               value: "http://minio.default.svc.cluster.local:9000" # change this to your s3 endpoint url