Proposal: store and serve ledgers in RPC v2 without diagnostic events #1149

Parent: #1147.

This proposes that RPC v2 ingest, store and serve ledgers without Soroban diagnostic events. It hasn't been decided yet. #1119 tracks the decision in its item "Decide whether ledger packs keep diagnostic events", and #1052 has the discussion. This issue sets out what dropping them would require, what we would gain, and how it could be done.

Diagnostic events are the contract host's debug log. Core only writes them into ledger meta when ENABLE_SOROBAN_DIAGNOSTIC_EVENTS is on, and RPC v2 turns that option on today. At sac-9000 density they make up 41.5% of each ledger's raw bytes, and ingestion has to parse every one of them.

Requirements, if we go ahead

• No ledger that RPC v2 stores has diagnostic events, whether it came from live ingestion, a backfill or a freeze. Hot and cold data then give the same answer for every transaction.
• sendTransaction still returns core's diagnostics when core rejects a Soroban transaction.
• simulateTransaction still returns the events from its simulation.
• It lands before GA, so no released node writes ledger packs with diagnostic events.

Expected gain

• Faster ingestion. With the new ingestion stack (Hot ingestion: meet the newest Phase 3 target of 105 ms p99 at sac-9000 density #1148), dropping diagnostic events cuts hot ingestion from 88.2 ms to 80.8 ms at p99, and from 67.4 ms to 57.5 ms at p50.
• Smaller ledgers in memory, but barely on disk. A sac-9000 ledger shrinks from 43.5 MB to 25.4 MB raw, but only from 3.36 MB to 3.20 MB compressed.
• Pubnet today: diagnostic events are 18.2% of a raw ledger and 4.3% of a compressed one.

How these were measured: the ingestion numbers come from a c6id.8xlarge with local NVMe ingesting 1,000 sac-9000 ledgers as fast as it can, four runs with diagnostic events and two without. The sizes come from the 10,000 ledgers of sac-9000's first chunk. The pubnet shares come from 1,000 SDF data lake ledgers near ledger 64,810,000.

What would change for API users

• getTransaction and getTransactions stop returning diagnosticEventsXdr and diagnosticEventsJson.
• The transaction meta they return, and the ledger meta getLedgers returns, has empty diagnostic event lists.
• Nothing else changes. getEvents never served diagnostic events. sendTransaction's diagnostics come from a separate core option, ENABLE_DIAGNOSTICS_FOR_TX_SUBMISSION. simulateTransaction's events come from RPC's own simulation.

Suggested approach

• Captive core. Turn off ENABLE_SOROBAN_DIAGNOSTIC_EVENTS and keep ENABLE_DIAGNOSTICS_FOR_TX_SUBMISSION on. RPC v2 sets the SDK's EnforceSorobanDiagnosticEvents today, which turns on both, so RPC v2 would set the two options itself. It could also refuse a captive core config that turns diagnostic events on.
• Backfill from the data lake. The lake's ledgers have diagnostic events, so backfill would strip them before writing. A prototype copies each ledger with its diagnostic event lists emptied. Its output is byte for byte what decoding, clearing the lists and re-encoding gives, and it took about 0.3 ms per pubnet ledger on an M1 Max.
• Docs. Mark diagnosticEventsXdr and diagnosticEventsJson deprecated in go-stellar-sdk's protocols/rpc and in the getTransaction and getTransactions docs.

Links

• Epic: Pre-GA changes to cold artifacts #1119's item "Align the captive-core meta flags between RPC v2 and galexie" should record this as a deliberate difference, since the lake keeps diagnostic events.

🤖 Generated with Claude Code