Benchmark full-history backfill — time and disk footprint on minimum hardware #895

TL;DR

Backfill the entire pubnet history with the production backfill path, on minimum hardware, and publish two numbers partners need to size an RPC v2 deployment:

1. Time — wall-clock to backfill full history on minimum hardware (a conservative upper bound).
2. Disk — total on-disk footprint, broken down by data type (ledgers vs transaction hashes vs events), as absolute size, as a percentage split, and as per-chunk unit costs partners can extrapolate from.

This is a real-data run. It reads live pubnet ledgers from a public datastore and drives the daemon's own backfill. It does not use synthetic datasets.

Why

Partners must provision hardware and storage before they run RPC v2 with full history. Two questions block that decision: how long does the initial backfill take, and how much disk does full history consume. The compressed input lake from genesis to tip is approximately 4.9 TB (measured 2026-08-10 by listing the public S3 lake). The cold-store footprint by data type is not yet measured. This run measures both and gives partners an extrapolation rule for shorter retention windows.

What "full history" means here

Full history is organized in chunks of exactly 10,000 ledgers. chunk_id = (ledger_seq − 2) / 10_000; the chain starts at ledger 2.

• Start chunk = the genesis chunk = chunk 0 (ledgers 2–10,001).
• Number of chunks = every complete chunk up to the tip. At the current tip (63,890,102 ledgers measured 2026-08-10), that is 6,389 complete chunks (chunk 0 through 6,388). Compute the exact count from the live tip on run day: floor((tip_ledger − 1) / 10_000). The final in-flight chunk is incomplete and is not a backfill target; it is the live/hot chunk.
• Chunk growth — approximately 1.7 chunks per day at the current ledger close rate.

History is front-light and back-heavy: early ledgers are nearly empty, recent Soroban ledgers are large and carry almost all events. Per-chunk time and per-chunk size both rise across the range. The whole-range run is the only faithful total; any early-chunk sample underestimates both cost and event share.

Method

Use stellar-rpc bench-ingest cold. Cold is the backfill path — it runs one RunBackfill over the chunk range: per-chunk freezes (ledger packs, txhash bins, events cold segments), the cross-window txhash index builds, and the shared worker pool. Backfill full history in a single run so the cross-window index builds (one per 1,000-chunk window, approximately 6–7 windows over full history) are included in the measured total. A chunk-by-chunk loop would not exercise them.

Read ledgers from the datastore, not from local packs, so the run reflects a real backfill:

• Source: the public S3 lake s3://aws-public-blockchain/v1.1/stellar/ledgers/pubnet (region us-east-2, anonymous-read — no credentials). Total size approximately 4.9 TB compressed.
• Run the box in us-east-2 so datastore reads stay in-region. This benchmark measures local ingest throughput and disk cost, not cross-region download latency. An out-of-region box measures the wrong thing.
• Alternative source: SDF's GCS lake (--datastore-type=GCS, --bucket-path=sdf-ledger-close-meta/v1/ledgers/pubnet). It needs auth. Match the box region to the bucket.

Backfill streams ledgers via BufferedStorageBackend with an in-memory prefetch buffer. No local staging of the lake occurs. Disk holds the cold-store output only.

Illustrative command (confirm exact flags against the harness help):

stellar-rpc bench-ingest cold \
  --source=bsb --datastore-type=S3 --region=us-east-2 \
  --bucket-path=aws-public-blockchain/v1.1/stellar/ledgers/pubnet \
  --start-chunk=0 --num-chunks=6389 --workers=8 \
  --cold-out-dir=/mnt/data/cold --catalog-dir=/mnt/data/cold \
  --out=/mnt/data/bench-out

Put --cold-out-dir and --catalog-dir on the disk being measured. --out (CSV report) can be anywhere.

Default settings

Run RPC v2 as a partner would out of the box. Change only what the box dictates.

• Workers = number of cores. The daemon defaults backfill concurrency to the CPU count ([backfill] workers, defaults to runtime.NumCPU()). The bench flag --workers defaults to 1, so set it explicitly to the box's core count to match the production default (--workers=8 on an 8-core box).
• Leave datastore prefetch and retry knobs at their defaults: --bsb-buffer-size=0, --bsb-num-workers=0, --retry-limit=0 all mean "backfill's own default." Do not tune them.

Hardware

Target minimum compute so the time is a conservative floor a small partner deployment can match: an x86 box at 8 vCPU / 32 GiB (m6id.2xlarge class). Workers follow the core count (8).

The constraint is disk. The expected cold-store footprint is 4–8 TB: the compressed input lake alone is 4.9 TB, and the cold store re-encodes the ledgers plus adds the txhash and events structures; the exact ratio between input and cold store is not yet measured. Before full provisioning, run a calibration step: ingest approximately 4 sample chunks spread across history (e.g., chunk 0, 300, 3,000, 6,345) on any small box. After each run, measure per-type sizes:

du -sh /mnt/data/cold/ledgers /mnt/data/cold/txhash /mnt/data/cold/events /mnt/data/cold/catalog

Integrate the sample bytes/chunk across the chunk range to fix the full-history footprint before provisioning. Single-chunk runs fit in 474 GB instance store; region does not affect byte counts.

Two hardware configurations to choose from:

• Recommended — storage-optimized NVMe instance, workers pinned to 8. After the calibration step, if footprint ≤ 7 TB, use an i4i.8xlarge-class instance (2×3.75 TB NVMe RAID0) and set --workers=8 to hold the compute profile at minimum. Instance-store NVMe keeps disk I/O honest, so the wall time is representative of real deployments using local fast storage. Trade-off: the box has more vCPU than the compute target, used only to support the attached NVMe.
• Alternative — 8-vCPU box + EBS data volume, exact-minimum compute. Keep the m6id.2xlarge and mount a provisioned-throughput volume (gp3 tuned high, or io2) sized after the calibration step for --cold-out-dir and --catalog-dir. This holds compute exactly at minimum. Trade-off: cold writes hit EBS, so provision throughput/IOPS appropriately — a throttled volume inflates the wall time. Report the exact volume specification (size, throughput, IOPS) in the results so the timing is interpretable.

The report must state which disk backed the run.

Wall-time estimate

Cold ingest is events-bound. A July 2026 bench campaign on m6id.2xlarge measured four real-pubnet chunks at 6.2 s (chunk 3000), 45.9 s (chunk 5000), 40.2 s (chunk 6100), and 61.7 s (chunk 6345), totaling 25.43 million events in 154.1 seconds. This yields a throughput of approximately 165,000 events/s. The model reproduces each measured chunk within 8% error.

Pubnet history holds approximately 20 billion events (calculated from measured events-per-ledger at four points across history). At 165,000 events/s, full-history backfill takes approximately 33 hours on 8-core minimum hardware. Download is not the constraint: 4.9 TB over 33 h ≈ 41 MB/s average, trivial in-region.

Plan for 1.5–3 days wall time. Uncertainty sources: (1) the July runs' worker setting is not recorded — if it was below 8, the real 8-worker run finishes faster than the estimate. (2) The events curve has limited sampling in the 30–50M ledger range where activity spikes; real history may carry more events. (3) Measured chunks exclude the approximately 6–7 cross-window txhash index rebuilds — unmeasured cost, add allowance. (4) Recent improvements to chunk-build performance shorten the wall time if merged before the run.

Measurements and report

Time. From driver.csv:

• backfill_wall — total backfill wall time. This is the headline number.
• ledgers_total, txhash_total, events_total — per-type ingest time (which data type dominates the clock).
• index_rebuild — cross-window txhash index build cost.
• peak_rss_bytes — peak process memory (Linux only), so partners can confirm 32 GiB is enough.

Disk. After the run, size each data-type subtree under --cold-out-dir:

du -sh /mnt/data/cold/ledgers /mnt/data/cold/txhash /mnt/data/cold/events /mnt/data/cold/catalog

Report:

• Absolute size per subtree (ledgers/, txhash/, events/, catalog/) and the total.
• The percentage split across ledgers / transaction hashes / events.
• Per-unit costs for extrapolation: bytes per chunk, bytes per ledger, bytes per million events.
• An extrapolation rule so a partner can size a shorter retention window from the full-history total.

State the box model, region, disk backing and spec (if EBS, include size/throughput/IOPS), exact --num-chunks and tip ledger, and total wall time alongside the numbers.

Risks

• Long single run. Full history is a 1.5–3 day run. Backfill uses a fresh scratch catalog per run and does not resume, so a mid-run failure loses progress. Run under tmux/screen, monitor free disk, and treat one clean pass as the deliverable.
• Calibration first. Before committing the full run, do the short calibration step above to validate the command, box, and disk-write rate and to produce the exact disk bracket. Measure both early and recent chunks — early-only samples underestimate event volume and final-chunk size.
• Final chunks largest. The tail of history carries the most events and largest ledgers per chunk. Monitor free space in the final half of the run.

Done when

A full-history backfill has run to completion on minimum hardware, and a published record reports the total backfill time, the peak memory, and the total disk footprint with a per-data-type percentage split and per-unit extrapolation constants. The record states the box model, region, disk backing (and volume spec if EBS), the exact chunk range run, and the corresponding tip ledger.

Related

• Full-history benchmark commands — ingest across all data types #771 — the bench-ingest cold command this run uses.
• Benchmarking infra: stellar-rpc-bench S3 bucket + benchmarking devbox #866 — results bucket for the CSVs and the report (GCS fallback OK).
• e2e benchmark campaigns workflow — dispatch, run on EC2, publish results #868 — campaign automation. This run is manual (single very long run); automation is optional here.
• Independent of Synthetic ledger datasets for Phases 1–3 ("golden packs") #867 — reads real pubnet from the public lake, not synthetic golden packs.