[NOT READY YET] Don't block on (or restore from) third-party snapshot copies

[huydhn]

Follow-up to #226, kept separate as discussed. #226 stopped the Lambda blocking on a pending snapshot; this fixes which snapshots we consider ours in the first place.

Why

CopySnapshot copies the source snapshot's tags onto the copy. The Wiz security scanner copies gpu-dev disks, so its copies arrive tagged gpu-dev-user + disk_name + ManagedBy=gpu-dev-cli and our disk logic treats them as ours. Two failures, both observed in prod:

1. Blocking. The pending-snapshot gate matched the scanner's copy, which can sit pending for hours — one lived 10h (snap-01da8673…, CloudTrail: CopySnapshot by WizScannerRole, description auto-generated by Wiz). On 2026-08-17 that produced 3 failed reservations for drisspg. A scan batch at 17:38 UTC on 08-18 armed the same trap for 8 users simultaneously.

2. Silent stale restores — the more dangerous one. A completed copy also matched, and its StartTime is the copy time, so it can win max(StartTime) and be picked as the restore source — returning a point-in-time older than it appears. Simulated against prod data, 7 of the 8 affected users would have restored from a scanner copy on their next reservation.

What

• is_foreign_snapshot() — ignore snapshots we didn't author, in both the pending gate and the restore-source selection.
• Only wait on a pending snapshot newer than the newest completed one. An older pending snapshot has nothing to offer; the 08-17 blocker was 6h15m older than a complete, ready restore point.

Matched on copy markers (wiz* tags, VolumeId == vol-ffffffff, description) rather than requiring a gpu-dev tag. A whitelist looked cleaner but 133 of 6612 real snapshots carry no SnapshotType tag, including 5 hand-made recovery snapshots (mreso, nshulga, ezyang) it would have orphaned.

Validation

Against all 6612 production snapshots: 6596/6596 real kept, 16/16 scanner copies excluded, zero false exclusions.

Replaying the new logic over the 8 currently-affected users — all go blocked → ok, and 7 get corrected restore sources:

user                    before      after       restore source
drisspg@meta.com        BLOCKED     ok          snap-0f2342ae1205868a8
simonlayton@meta.com    BLOCKED     ok          snap-0dd5f5005e704886e  (was a WIZ COPY)
eellison@meta.com       BLOCKED     ok          snap-01ef00259731e6c5c  (was a WIZ COPY)
vasiliy@meta.com        BLOCKED     ok          snap-092b7e618ec648584  (was a WIZ COPY)
tarindu@meta.com        BLOCKED     ok          snap-04b77d2f3f3917576  (was a WIZ COPY)
zainhuda@meta.com       BLOCKED     ok          snap-08fd9cb46fbf06400  (was a WIZ COPY)
calebkim@meta.com       BLOCKED     ok          snap-03b6b318de711ee99  (was a WIZ COPY)
felipemello@meta.com    BLOCKED     ok          snap-04c57fe289b87a8d6  (was a WIZ COPY)

Test plan

Written TDD — 5 new tests in tests/unit/lambda_fn/test_foreign_snapshots.py, all confirmed failing first (incl. one reproducing the stale-restore path as assert 'snap-wiz' == 'snap-ours').

env -u SSH_AUTH_SOCK uv run pytest -m "not integration"
# 1220 passed, 3 xfailed

tofu validate not run — my local provider cache is empty; the tf change is a one-character LAMBDA_VERSION bump to 0.6.7.

Notes

• Needs tofu apply (prod) to take effect. Until then the 8 users above stay blocked; --no-persist is the interim workaround.
• Deliberately out of scope: the unbounded queued state from Retry pending disk snapshots without blocking Lambda #226 (a stuck snapshot still queues forever with no deadline) — separate PR.
• shared/snapshot_utils.py::get_latest_snapshot has the same hole, but its only caller get_latest_user_snapshot → migrate_ebs_across_az is dead code, so I left it. Worth deleting or fixing if it's ever revived.
• Worth raising the tag-inheritance behaviour with the Wiz integration owners — it affects anything keying off gpu-dev-user/disk_name, not just this gate.