[marwen-abid]

What

• New registry file runner/internal/packgen/generations.json. It lists the dataset pack generations, newest first: packs-v6, packs-v5, packs-v4. Each entry has a reader_marker (a git grep -E pattern and paths, _test.go excluded), an optional reader_absent, a writer_commit and a one-line format_change. The runner embeds the file, and the tests parse it.
• run.DetectPackGeneration picks the first entry whose markers match the built commit. If no entry matches, the generation is unknown: the runner does not rewrite the location, and the read check still runs.
• Markers:
  • packs-v6: groupHasCounts uint8 = (stellar-rpc a130b9e2 cmd/stellar-rpc/internal/rpcv2/packfile/index.go:29, added by c8d48764).
  • packs-v5: trailerSize = 44 (4d30c258 packfile/packfile.go:14), and dirEntryLen = must not match (a130b9e2 index.go:40, added by 4ca7bd1c). This keeps a commit with the new index directory but without per-group counts from being taken as v5.
  • packs-v4: trailerSize = 76 (1c0572da packfile/packfile.go:15).
• plan.json records pack_generation as the full name (for example "packs-v5", was "v5") and the new pack_marker. The dry-run note gives both.
• Location rewrite: in a packs-s3 location, each path segment that names a registered generation changes to the chosen generation. configured_location stays as in runner: choose the dataset pack generation the ref reads, and refuse a mismatch before ingest #43.
• Availability: before the build, campaign run lists each packs-s3 location that it will fetch. If a location is empty, the campaign stops. The message gives the missing prefix and the generation that the registry chose. --no-preflight skips this check.
• Readability: after each packs-local, packs-gs or packs-s3 dataset is ready and before any leg, the runner runs the built binary on one ledger. The command is <bin> --enable-rpcv2 bench ingest hot --source=pack --pack-dir=<root>/ledgers --start-chunk=<first chunk> --num-ledgers=1 --close-interval=0 --hot-dir=<scratch>/hot --out=<scratch>/out. It runs in $BENCH_ROOT/scratch/readcheck-<name>, which the runner removes before and after the check, and it gets the same GOMEMLIMIT as the legs. If the check fails, the campaign stops, also without --fail-fast. The error gives the binary's own last lines and the text dataset <name> is <generation>; the registry chose it for <sha8> by marker <…>. Only plans with static_libs (--enable-rpcv2) have this check. Older refs check nothing.
• This replaces the trailer-bytes check of runner: choose the dataset pack generation the ref reads, and refuse a mismatch before ingest #43 (packcheck.go and its fixture packs).
• runner/README.md documents the registry and how to add a generation.

Why

stellar-rpc a130b9e2 (4d30c258 + packfile 4ca7bd1c, 5546d795, c8d48764) changes the layout of the ledger pack index, but the trailer stays 44 bytes. It rejects packs-v5 at open with packfile: corrupt file: index group 0 starts at item …. The trailer check of #43 cannot tell packs-v5 from packs-v6. With this change, a new generation needs only a freeze, an upload and a registry entry, and no code change. The read check uses the same reader as the legs, so it finds any format mismatch, not only a trailer mismatch.

Campaigns read only <root>/ledgers from a golden dataset. The query legs read the stores that the campaign's own ingest builds. Thus the markers identify the ledger pack reader only.

packs-v6 is in the registry now, but it is not uploaded yet. It becomes usable when the packs-v6 freeze is uploaded for the three phase-3b C1 profiles (sac-9000, custom_token-5400, soroswap-2700). Until then, a campaign at a ref that reads packs-v6 stops at the availability check, before the build.

Testing

• cd runner && go vet ./... && go test -count=1 ./...: all 11 packages pass, 409 tests and subtests. New and changed tests:
  • Registry: the embedded file parses, and 11 bad registries are rejected.
  • Selection: git repos with commits shaped like v4, v5, the 4ca7bd1c-only directory (selects none), v6, and markers in test files only. Also a new top entry (packs-v7) that the runner selects with no code change.
  • Rewrite: v4, v5 and v6 locations and refs, and unregistered segments that stay unchanged.
  • Read check: a fake binary that passes (the legs run, GOMEMLIMIT reaches it, the scratch dir is removed) and one that fails (the walk stops, and the error has the binary's error and the registry's choice).
  • Availability: a stub listing with an empty location, a listing error, a fetched root, and a packs-gs dataset.
• make test (converter): 161 tests, OK (13 skipped).
• make smoke: 4758 passed, 0 failed (35 runs).
• Dry run: campaign plan with the phase-3b config, against git clone --shared checkouts of stellar-rpc:
  • 1c0572da: pack generation: packs-v4, the runner fetches …/packs-v4/cold.
  • 4d30c258: packs-v5, the runner fetches …/packs-v5/cold.
  • a130b9e2: packs-v6, the runner fetches …/packs-v6/cold.
  • Each dataset step prints the read check.
• Read check on the dev box with the stock binaries, soroswap-2700 chunk 1:
  • 4d30c258 on packs-v5: exit 0 in 0.48 s.
  • 4d30c258 on packs-v4: exit 1 in 0.11 s, packfile: corrupt file: invalid magic number.
  • a130b9e2 on packs-v5: exit 1 in 0.58 s, packfile: corrupt file: index group 0 starts at item 3426353155.

Not covered

• No full campaign run on a box. I did not run the availability check against the real bucket, because there are no AWS credentials on the local machine.
• I did not run a130b9e2 on packs-v6 data, because the freeze is still in progress.
• A campaign plan of a ref that the clone cannot resolve now assumes the newest registry entry (packs-v6), not packs-v5.

https://claude.ai/code/session_01S2XwYtZzcJnyxHU9TC3oLh

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-10-23 05:34 UTC