feat(debug-trace-server): age-based witness routing, configurable old-block budget, tip buffer

AGENTS.md

@@ -107,6 +107,8 @@ Two operating modes:
 - **Local cache mode** — With `data_dir`, enables chain sync to pre-fetch blocks into `ValidatorDB` for faster serving.
 
 The server includes an HTTP response cache (`quick_cache`) for pre-serialized JSON and a `DataProvider` with single-flight request coalescing.
+In local cache mode with a `--witness-generator-endpoint` plus at least one fallback `--witness-endpoint` (or, deprecated, the first of two or more `--witness-endpoint` values acting as the generator), request-serving witness fetches route by block age: blocks at least `--witness-local-window` blocks below the local tip skip the generator (which prunes beyond its `BACKUP` window) and fetch from the fallbacks.
+The background chain-sync prefetch always uses the full endpoint chain — it fetches at the sync frontier, which stays within the generator's retention unless `--blocks-to-keep` exceeds that retention during deep catch-up.
 
 ### Key Source Files
 

README.md

@@ -98,6 +98,17 @@ Two operating modes:
 - **Stateless mode** (no `--data-dir`): All data fetched from remote RPC on demand.
 - **Local cache mode** (with `--data-dir`): Enables chain sync to pre-fetch blocks for faster serving.
 
+**Witness endpoints:**
+Declare the internal witness generator via `--witness-generator-endpoint`; `--witness-endpoint` lists the durable fallbacks (e.g. an R2-backed witness service), tried in order.
+In local cache mode with a generator plus at least one fallback, requests for blocks at least `--witness-local-window` blocks below the local tip skip the generator and fetch from the fallbacks, because the generator only retains a recent window (its `BACKUP`, deployed at 4096) and probing it for pruned blocks is a guaranteed miss.
+The background chain-sync prefetch always uses the full endpoint chain.
+Deprecated: without `--witness-generator-endpoint`, the first of two or more `--witness-endpoint` values is treated as the generator (in local cache mode a startup warning nudges migration).
+
+**Witness routing and sync knobs** (each also settable via its `DEBUG_TRACE_SERVER_*` env var):
+- `--witness-local-window`: Block-age threshold for the historical witness route (default: 4096; should match the generator's `BACKUP`).
+- `--witness-old-block-timeout`: Witness-stage budget in seconds for blocks at or below the local tip (defaults to the full `--witness-timeout` budget, tracking it when raised; lower it to fail fast on pruned blocks).
+- `--tip-buffer`: Stay this many blocks behind the upstream head during chain sync so fetches don't race the witness generator (default: 2; must be smaller than `--blocks-to-keep`).
+
 ### Environment Variables
 
 Each command-line flag has an equivalent environment variable:

bin/debug-trace-server/src/chain_sync.rs

@@ -24,6 +24,12 @@ use crate::{metrics, response_cache::ResponseCache, server_db::BlockStore};
 /// Witnesses go through the zero-validation light decode (`get_witness_light`):
 /// the server never verifies the proof, so the full decode's per-point
 /// elliptic-curve work bought nothing.
+///
+/// Witness fetches deliberately use the full endpoint chain (no age-based routing): the sync
+/// frontier trails the remote head by only `tip_buffer`, inside the generator's retention —
+/// except during deep catch-up with `--blocks-to-keep` beyond that retention, where each
+/// pruned block burns one generator probe before failover (accepted; routing here would need
+/// a remote-head anchor instead of the local tip).
 pub struct TraceFetcher {
     pub rpc_client: Arc<RpcClient>,
 }

bin/debug-trace-server/src/metrics.rs

@@ -206,7 +206,10 @@ impl CacheMetrics {
     }
 }
 
-/// Tracks which source provided block data (cache/db/witness_generator).
+/// Tracks which source provided block data. Sources: `cache`, `db`, and the two RPC witness
+/// routes — `witness_generator` (full endpoint chain, generator first) and
+/// `witness_historical` (skip-generator chain for blocks at least the local window below the
+/// tip). The RPC path as a whole is the sum of the two witness labels.
 #[derive(Clone, Metrics)]
 #[metrics(scope = "debug_trace")]
 pub struct DataSourceMetrics {
@@ -421,6 +424,7 @@ fn pre_register_all_metrics() {
     let _ = DataSourceMetrics::new_for_source("cache");
     let _ = DataSourceMetrics::new_for_source("db");
     let _ = DataSourceMetrics::new_for_source("witness_generator");
+    let _ = DataSourceMetrics::new_for_source("witness_historical");
 
     // Data Fetch Layer: single-flight
     let _ = SingleFlightMetrics::new_for_type("new");
@@ -439,6 +443,7 @@ fn pre_register_all_metrics() {
 
     // Witness Layer
     let _ = WitnessSourceMetrics::new_for_source("witness_generator");
+    let _ = WitnessSourceMetrics::new_for_source("witness_historical");
 
     // Execution Layer (per method)
     let _ = EvmExecutionMetrics::new_for_method(METHOD_DEBUG_TRACE_BLOCK_BY_NUMBER);