[tamirms]

What

The hot loop now starts a ledger's zstd encode before the ledger's transaction walk, and joins it as the last step queued into the ledger's batch. The encode runs alongside the walk and the other queue steps instead of between them.

• HotService.Ingest calls hotchunk.DB.StartCompress before ExtractLedgerTxParts. IngestLedger takes the pending compression and joins it on every path. PhaseLedgers is now the wait for the encode plus the put.
• The encode uses two libzstd threads (zstd.WithWorkers). The output is one standard frame, but its bytes differ from a single-threaded encode of the same input. The freeze of ledgers from the hot DB, later in this stack, makes the cold pack writer encode the same way, so a copied pack matches a built one.
• The store keeps one compressor and one output buffer, which keeps the capacity of the largest ledger compressed so far. Before, it borrowed a compressor from a pool that a garbage collection could empty, and encoded into a new buffer for every ledger.
• The ledger writes are single-flight: a second StartCompress while one is in flight panics.

Tests: TestStartCompress_OneInFlight requires that panic, and that joining the first compression frees the store for the next. TestStartCompress_ReusesItsBuffer requires every ledger to compress into the one buffer, across garbage collections.

Why

A sac-9000 ledger, with its diagnostic events, is 43 MB of XDR and 3.35 MB compressed. On the events stack, the ledgers phase, which compresses the ledger on the ingest goroutine and queues it, takes 47.2 ms of a 153.4 ms ledger at p50 (1,000 unpaced ledgers, mean of two runs). The encode needs only the ledger's bytes, so it can run while the walk does. Measured before this stack was reordered, on the stack through the SDK walk PR, which is now its top PR, the same phase takes 1.3 ms at p50 (1,000 unpaced ledgers, mean of four runs).

🤖 Generated with Claude Code