feat(debug-trace-server): inbound admission control and response-size caps

AGENTS.md

@@ -109,7 +109,20 @@ Two operating modes:
 The server includes an HTTP response cache (`quick_cache`) for pre-serialized JSON and a `DataProvider` with single-flight request coalescing.
 Responses are serialized exactly once, straight from the trace output into raw JSON bytes (`RawJson`) spliced verbatim into the reply; the cache shares those bytes by `Arc`, so a hit never re-parses or re-serializes the JSON tree.
 Inbound JSON-RPC batch entries execute concurrently as independent runtime tasks through the regular per-request pipeline (synchronous trace CPU parallelizes too; spawned-entry CPU folds back into `x-execution-time-ns`/the CPU metric via a response extension) (`rpc_middleware.rs`, bounded by `--batch-item-concurrency`, default 16; 1 restores jsonrpsee's sequential behavior; batch shape observable via `debug_trace_batch_size`), so a batch answers near its slowest entry instead of the sum of its entries.
+Inbound admission control (`admission.rs`) bounds what clients may ask of the process — every other concurrency cap here is outbound, and unbounded inline EVM tracing otherwise starves chain sync, the accept loop and the metrics exporter along with the requests — shedding with `-32013 "Request queue is full"`, mega-reth's `ConcurrencyLimiter` contract byte for byte so existing client backoff applies unchanged.
+It is deliberately split in two: `AdmissionLayer` (installed *inside* `ConcurrentBatchLayer`, so it sees single calls and every batch entry, since that layer decomposes batches into per-entry `call`s rather than delegating to an inner `batch`) only ever runs a non-blocking CAS against `--admission-max-concurrent` + `--admission-max-queue`, while the execution permit is taken in the handler *after* `check_cache` misses.
+That placement is load-bearing, not stylistic: `CancelGuard` arms on a request's first poll and the handler records its arrival synchronously in that same poll, so a middleware gate that parked before the handler would record a cancellation with no matching arrival for every client that hung up while queued — negative, permanent identity drift, worst under exactly the overload the gate exists for; a layer that only CAS-es preserves the invariant by construction, and the permit wait then sits after the arrival is already booked.
+It also means a response-cache hit never takes a permit, the typed `RequestShape` is already in hand so the heavy-tracer sub-cap (`--admission-heavy-max-concurrent`, covering `prestateTracer`/JS/`muxTracer` and *every* struct-logger request including the bare default — the flags separating the two struct-logger shapes change output size, not kind — taken *before* the ordinary permit, so heavy requests never occupy execution permits while waiting for each other) costs no second parse of attacker-controlled JSON, and what the permits count is blocks actually being fetched and replayed.
+The permit wait is clamped to `deadline - witness_timeout` (`DataProvider::permit_cutoff`), so a request that queued away the budget its witness fetch still needs is refused now rather than started and timed out later — the difference between "may reject" and "times out"; the reserve falls back to half the budget when `--witness-timeout` does not fit inside `--block-fetch-timeout` (a legal pair, since the witness sub-deadline is a `min` against the outer one), because reserving all of it would leave a zero-length wait and silently reduce `--admission-max-queue` to a no-op.
+Every handler mints its deadline once and passes it to both the permit wait and the fetch (`get_block_data_by_hash`/`get_block_data_for_tx` take it as a parameter for exactly this reason), so the queue is carved out of `--block-fetch-timeout` rather than added on top of it.
+`debug_getCacheStatus` and its `timed_` alias are the sole exemption (matched on the `timed_`-stripped name, since the gateway adds that prefix by default and a bare-name comparison would never match production traffic); `assert_admission_covers_module` fails startup if a newly registered method escapes `metrics::GATED_METHODS`.
+The limiter uses a resizable FIFO `Semaphore` (shrink via `forget_permits` plus a debt counter settled on release) rather than the reference implementation's counter+`Notify`, which has a lost wakeup (its `Notified` is created *after* the capacity check) and never wakes parked waiters on a raise — both regression-tested here.
+Occupancy is counted, never derived as `limit - available_permits`: once a shrink leaves debt behind that difference reports the *new* limit, so the admin RPC would answer a retune by claiming it had already taken effect while every old holder was still resident (`occupancy_stays_truthful_while_a_shrink_is_outstanding`).
+`--max-response-size` is checked at this server's own serialization point (`RawJson::try_new` in `serialize_reply`/`compute_block_trace`, counted on `debug_trace_response_oversized_total`, classified `TraceError::Request` so an oversized ask never evicts a good block), and `--max-batch-response-size` separately caps the assembled batch — different bounds, because a batch retains every completed entry's body until it finishes, and that accumulation rather than any single response is what has previously exhausted the process; both `heavy_max_concurrent x max_response_size` and `max_concurrent x max_response_size` are logged at startup, and neither bounds a tracer's intermediate allocations — a per-transaction-count gate would, and is not implemented.
+`--admin-addr` (loopback enforced at startup, off unless set) serves `admin_getConcurrencyLimit`/`admin_setConcurrencyLimit` on a second listener with no RPC middleware — so a saturated public port cannot shed the call that relieves it — running on its own thread and runtime, since inline EVM tracing on the main runtime would otherwise starve it exactly when it is needed; `maxConcurrent = 0` is refused as unrecoverable.
 Every inbound request is accounted for exactly once, so "did any client see a timeout?" is a metric lookup rather than an inference: arrivals are counted by `debug_trace_request_shape_total` before the first await, and each request then lands in exactly one of `debug_trace_rpc_requests_total` (served), `debug_trace_rpc_errors_total{reason}` (failed), or `debug_trace_requests_cancelled_total` (client hung up — recorded from the drop of the request future in `rpc_middleware.rs`, the only layer that observes single calls and batch entries alike), so `shape = requests + errors + cancelled` holds per method.
+A shed adds the balanced pair `shape="shed"` / `reason="overloaded"`, recorded by the admission layer from *inside* the returned future — `record_rpc_error` sets the `ERROR_SELF_REPORTED` task-local that tells `settle_response` this non-framework `-32013` is already accounted for, which is what keeps `settle_response` unchanged; recorded from the layer's synchronous prefix instead, every shed would double-count its error and false-fire the `unattributed` drift alarm (`admission_shed_keeps_the_identity_closed` pins both).
+A permit refused after the handler already recorded its arrival contributes the outcome side alone.
 Requests the framework answers before any handler runs (unknown method, malformed top-level params, unparsable batch entries — recognized by their framework error codes) are folded in by the same middleware as the balanced pair `shape="rejected"` / `reason="rejected"`, an unrecorded error with a non-framework code lands error-side only on `reason="unattributed"` (a handler ran and recorded its arrival but bypassed the error funnel — a code-drift alarm that must stay at zero), and the opts-less `trace_*` methods record their `default` arrival at handler entry; the only deliberate approximation is a batch the server itself aborts over the response-size cap (its killed entries record no outcome rather than masquerade as client hangups) and batch entries that never started when the connection died (on neither side, so `cancelled` undercounts torn-down oversized batches).
 The identity is pinned end-to-end by `accounting_identity_holds_end_to_end` in `rpc_middleware.rs`, which drives every terminal path through a real server under a draining local metrics recorder.
 The `reason` label splits outcomes that share a JSON-RPC code — `deadline_witness` / `deadline_block` / `not_found` / `invalid_params` / `trace_failed` / `internal` — because a blown witness deadline and an unknown transaction both leave as `-32001` and only the first is an incident; the whole `(method, reason)` grid is pre-registered so an alert on `deadline_witness` reads zero instead of missing from boot.
@@ -163,13 +176,15 @@ The background chain-sync prefetch routes by freshness against the last observed
 | `bin/debug-trace-server/src/rpc_middleware.rs`                                                 | Concurrent execution of inbound JSON-RPC batch entries                               |
 | `bin/debug-trace-server/src/data_provider.rs`                                                  | Block data fetching with single-flight coalescing                                    |
 | `bin/debug-trace-server/src/block_data_cache.rs`                                               | Bounded in-memory `BlockData` cache keyed by block hash                              |
+| `bin/debug-trace-server/src/admission.rs`                                                      | Inbound admission gate: resizable FIFO limiter, guards, `RpcServiceT` layer          |
+| `bin/debug-trace-server/src/admin.rs`                                                          | Loopback-only `admin_*` listener for retuning the gate at runtime                    |
 | `bin/debug-trace-server/src/r2_witness.rs`                                                     | Direct-from-R2 witness source (light decode, deadline-aware)                         |
 | `bin/debug-trace-server/src/server_db.rs`                                                      | Defines + implements the bin-local `BlockStore` trait (backed by `stateless-db`)     |
 
 ## Test Organization
 
 Unit tests are embedded in source files alongside the code they test.
-Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, consistency, performance, timing_header, prune) and in `bin/stateless-validator/tests/integration.rs` (CLI parsing, mock-RPC pipeline, mainnet single-block validation).
+Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, compression, consistency, performance, timing_header) and in `bin/stateless-validator/tests/integration.rs` (CLI parsing, mock-RPC pipeline, mainnet single-block validation).
 Test data (block JSON files, contract bytecode, witness data) is stored in `test_data/`.
 
 ## Version Control

README.md

@@ -116,6 +116,45 @@ Each entry still goes through the regular per-request pipeline — response cach
 `--batch-item-concurrency` (default 16) bounds how many entries of one batch run at once, so a single huge batch cannot monopolize downstream resources against concurrently served requests; set it to 1 to restore sequential execution.
 The `debug_trace_batch_size` histogram records entries per inbound batch, and CPU burned inside spawned entries is folded back into `x-execution-time-ns` and the request CPU metric, so batch requests do not under-report their cost.
 
+**Admission control:**
+Every request is checked against a process-wide capacity budget before any work happens, so an offered load beyond what this server can serve is refused immediately instead of degrading everything already in flight.
+Without it the only backpressure is that all runtime workers are busy tracing, which starves chain sync, the accept loop and the metrics exporter along with the requests themselves; the outbound witness/data/R2 caps bound what we ask of others, never what clients ask of us.
+Refused requests are answered `-32013 "Request queue is full"`, matching mega-reth's `ConcurrencyLimiter` byte for byte so a client or gateway that already backs off for the node backs off for this server too.
+The gate has two stages.
+Arrival is a non-blocking check against `--admission-max-concurrent` + `--admission-max-queue`: past that total, the request is shed on the spot.
+An admitted request then waits for one of `--admission-max-concurrent` execution permits, taken only once the response cache has missed — a cache hit costs microseconds and is never queued behind a cold trace — and given up if the wait would leave too little of the request's budget for the witness fetch, which is what turns a full queue into a prompt rejection rather than a deadline error.
+The wait is carved *out of* the request's `--block-fetch-timeout` budget rather than added on top of it, so total client latency stays bounded by that flag however long the queue was; the derived maximum wait (`--block-fetch-timeout` less the witness reserve) is logged at startup as `max_permit_wait_ms`.
+Entries of a JSON-RPC batch admit individually, so one large batch is metered like the equivalent single calls rather than passing as one unit.
+`debug_getCacheStatus` (and its `timed_` alias) is the sole exemption: it does no I/O, and shedding the one endpoint that reports what the server is doing, precisely while it is shedding, would be self-defeating.
+Sizing is not a core count — roughly 2% of a trace request is CPU, the rest is waiting on the upstream node and R2 — so derive `--admission-max-concurrent` from measured service time (`throughput x mean_handler_seconds`), and read `--admission-max-queue` as latency: queue depth over service rate is how long a request waits, so at 1000 blocks/s a 1000-deep queue is one second.
+The defaults sit above the highest occupancy this server has been measured serving cleanly, so they bound a previously unbounded process without throttling a known-good workload; tighten them once `debug_trace_admission_in_flight` and `debug_trace_admission_permit_wait_seconds` show what production actually does.
+Memory-hungry tracers additionally pass `--admission-heavy-max-concurrent`, a much smaller budget: one such response has been measured near a gigabyte.
+That set is `prestateTracer`, JS tracers, `muxTracer`, and **every struct-logger request** — both the bare default (a `debug_trace*` call with no `tracer`) and one with non-default flags, since the flags change the size of that output rather than its kind, and a struct logger emits a record per executed opcode.
+Note the consequence: an opts-less `debug_traceBlockByNumber` is a heavy request, so a client that sends no tracer is limited to `--admission-heavy-max-concurrent` at a time; raise that flag if such traffic is a normal part of your workload.
+Startup logs both `heavy_max_concurrent x max_response_size` and `max_concurrent x max_response_size` — the first bounds the shapes that can actually reach the response cap, the second is the theoretical ceiling if every admitted request returned a maximal body — so they can be checked against the host's memory limit.
+Neither covers a tracer's *intermediate* allocations, which nothing here bounds; a per-transaction-count gate would be needed for that and is not implemented.
+Watch `debug_trace_rpc_errors_total{reason="overloaded"}` for shedding, `debug_trace_admission_in_flight` / `debug_trace_admission_executing` for occupancy (queue depth is their difference), and the `debug_trace_admission_max_*` gauges for what is actually in effect, since the limits are changeable at runtime.
+Disable the whole mechanism with `--admission-disabled`.
+
+**Response size limits:**
+`--max-response-size` (default 256MB) caps one serialized reply and `--max-batch-response-size` (default 1GB) caps a whole assembled batch response; both bodies are discarded rather than returned, and an over-limit single response is counted on `debug_trace_response_oversized_total{method}`.
+They bound different things: a batch retains every completed entry's body until the batch finishes, so its memory is the sum of its entries rather than the largest of them, and left unbounded that accumulation — not any single response — is what has previously exhausted this process.
+The single-response check runs where this server serializes the reply, so an over-limit body is dropped before it can be copied again into the JSON-RPC envelope; the framework's own cap is the backstop.
+
+**Runtime retuning:**
+`--admin-addr` (e.g. `127.0.0.1:8546`, off unless set) starts a second RPC listener serving `admin_getConcurrencyLimit` and `admin_setConcurrencyLimit`, which change the admission limits without a restart:
+
+```bash
+curl -s http://127.0.0.1:8546 -H 'content-type: application/json' \
+  -d '{"jsonrpc":"2.0","id":1,"method":"admin_setConcurrencyLimit","params":[256,4096,4]}'
+```
+
+Parameters are `[maxConcurrent, maxQueueSize, heavyMaxConcurrent]`, each optional — omitted fields are left alone — and the call returns the resulting limits plus current occupancy.
+Lowering a limit below current occupancy aborts nothing; it stops admitting until the excess drains.
+Zero execution permits is refused, since it would park every subsequent request with nothing left running to release one.
+The listener carries no admission or accounting middleware, so a saturated public port can never shed the call that relieves it, and it runs on its own thread and runtime, so inline EVM tracing on the main runtime cannot starve it — the moment it is needed most.
+It has no authentication and its setters can throttle request serving, so a non-loopback bind is refused at startup; reach it through a port-forward or a sidecar.
+
 **Response compression:**
 Responses negotiate gzip/zstd per request via the client's `Accept-Encoding` header; clients that do not send it keep receiving identity bodies, so nothing changes for consumers that have not opted in.
 Bodies under 4 KiB are always served identity: compressing them would cost a per-response encoder allocation and downgrade `Content-Length` to chunked framing for a few dozen saved bytes.
@@ -172,6 +211,16 @@ Any single witness-chain RPC attempt under a deadline is additionally capped at
 The round's last hop, and every hop of a single-provider chain, takes the remainder whole under the ceiling instead, so a stalled endpoint (or a saturated concurrency permit — waits are deadline-bounded too) can never consume the stage while a rotation is still worth reserving for, and a slow-but-honest transfer is never structurally condemned; the witness decode runs outside the attempt window, bounded by the request deadline alone.
 `--r2-max-concurrent-requests` caps in-flight GETs separately from `--witness-max-concurrent-requests` — the RPC cap sizes a shared gateway, R2 tolerates far more.
 
+**Admission and response-size knobs** (each also settable via its `DEBUG_TRACE_SERVER_*` env var):
+- `--admission-max-concurrent`: Requests that may fetch and replay a block at once (default: 640; must be at least 1).
+- `--admission-max-queue`: Requests that may wait for a permit on top of those executing (default: 8192; `0` means execute-or-shed).
+- `--admission-heavy-max-concurrent`: Concurrent memory-hungry tracer requests (default: 8; sized from memory, not throughput).
+- `--admission-disabled`: Kill switch restoring unbounded concurrent work.
+- `--max-response-size` / `--max-batch-response-size`: Caps on one reply body and on a whole assembled batch response (defaults: 256MB / 1GB).
+- `--admin-addr`: Loopback-only listener for `admin_*` retuning (off unless set).
+
+`--admission-max-concurrent` + `--admission-max-queue` must be at least `--batch-item-concurrency`, and `--max-batch-response-size` at least `--max-response-size`; both are checked at startup and named in the error.
+
 **Witness routing and sync knobs** (each also settable via its `DEBUG_TRACE_SERVER_*` env var):
 - `--witness-local-window`: Block-age threshold for the historical witness route (default: 4096; should match the generator's `BACKUP`).
 - `--witness-old-block-timeout`: Witness-stage budget in seconds for blocks at or below the local tip (defaults to the full `--witness-timeout` budget, tracking it when raised; lower it to fail fast on pruned blocks).