Epic: Pre-GA changes to cold artifacts #1119

Parent: #1052. Spike write-up and discussion: #1052 (comment) and the replies below it.

Why

The spike measured a backfill of 600,000 recent ledgers. The results:

• Backfill is CPU-bound.
• Building ledger packs takes ~54% of the time. Building event packs takes ~25%.
• A node that receives finished ledgers/*.pack and events.pack files can skip that work.
• Two nodes produce byte-identical events.pack files from the same source ledgers.
• Two nodes do not produce byte-identical ledger packs. Core emits nondeterministic metadata. The SDF GCS lake and the AWS lake already differ for the same ledger.

Each item below is one task. Convert an item to a sub-issue when someone picks it up.

Before GA

These items change bytes on disk or the read path. Do them before the format freezes.

• Remove the content hash from backfill. The trailer SHA-256 costs ~11% of backfill CPU and no production code reads it. Compute a hash on demand later, after we agree on a normalization step. Closed by Remove the content hash from cold artifacts #1127.
• Record producer provenance in the artifacts. Store the zstd version, the zstd level, and the meta flag set in the pack app-data or in a sidecar file. A reader can then tell whether two packs are comparable before it compares bytes. Closed by Record the zstd version + settings in ledger packs and events.pack #1169
• Add size limits to the packfile and zstd readers. The readers trust three size fields from the file. These are the zstd frame's declared size, the trailer recordCount, and the trailer itemsPerRecord. A small malformed file can make the reader allocate many gigabytes. Add caps and negative tests. This needs no format change.
• Align the captive-core meta flags between RPC v2 and galexie. The lakes emit LedgerCloseMetaExt v1 and RPC's captive core does not (EMIT_LEDGER_CLOSE_META_EXT_V1). List every flag that changes meta bytes, choose the set we want, and record where galexie differs on purpose. Closed by Align the captive-core meta flags between RPC v2 and galexie #1165.
• Decide whether ledger packs keep diagnostic events. The data lake ledgers contain them. If we remove them, the backfill path must strip them, and the transaction API must deprecate its diagnostic-event fields, see Proposal: store and serve ledgers in RPC v2 without diagnostic events #1149.

After GA

These items add verification and distribution features. They do not change the frozen format.

• Define a canonical form for the captive-core config. Sort fields lexicographically. Operators can then compare configs byte-for-byte and hash them.
• Add opt-in canonical sorting of ledger-entry changes. Apply core's normalizeMeta comparator (ledger key, type order, sha256) at ingest or on the galexie side. The SDK's existing sort drops STATE entries and has no tie-break, so it is not sufficient.
• Define the normalized ledger form and the on-demand content hash. Zero invoke_time_nsecs or strip diagnostic events, sort changes as above, clear scpInfo, then hash each ledger. Peers and trusted sources compare this hash. Depends on the two diagnostic-event and sorting items.
• Train a per-chunk zstd dictionary for events.pack. An outer compression pass over an existing events pack saves 40%, which shows large cross-record redundancy. A dictionary applied per record saved 13–20% on disk in tests and keeps random access.
• Define the distribution encoding for shared packs. Distribution is out of scope for Decide what the cold artifacts must carry to be verifiable #1052, but the encoding follows from it. Ship .pack and events.pack as-is under an outer zstd -9 --long wrapper (~12% and ~40% smaller). The receiver gets the exact original bytes back. Record the wrapper's zstd version with the file.

Not in scope

• Sharing index.pack, index.hash, or txhash files. Each node keys these with its own secret. A receiving node rebuilds them from events.pack and the ledger packs.
• The distribution mechanism (where files live, how peers find them).