feat(debug-trace-server): inbound admission control and response-size caps

AGENTS.md

@@ -109,7 +109,20 @@ Two operating modes:
 The server includes an HTTP response cache (`quick_cache`) for pre-serialized JSON and a `DataProvider` with single-flight request coalescing.
 Responses are serialized exactly once, straight from the trace output into raw JSON bytes (`RawJson`) spliced verbatim into the reply; the cache shares those bytes by `Arc`, so a hit never re-parses or re-serializes the JSON tree.
 Inbound JSON-RPC batch entries execute concurrently as independent runtime tasks through the regular per-request pipeline (synchronous trace CPU parallelizes too; spawned-entry CPU folds back into `x-execution-time-ns`/the CPU metric via a response extension) (`rpc_middleware.rs`, bounded by `--batch-item-concurrency`, default 16; 1 restores jsonrpsee's sequential behavior; batch shape observable via `debug_trace_batch_size`), so a batch answers near its slowest entry instead of the sum of its entries.
+Inbound admission control (`admission.rs`) bounds what clients may ask of the process — every other concurrency cap here is outbound, and unbounded inline EVM tracing otherwise starves chain sync, the accept loop and the metrics exporter along with the requests — shedding with `-32013 "Request queue is full"`, mega-reth's `ConcurrencyLimiter` contract byte for byte so existing client backoff applies unchanged.
+It is deliberately split in two: `AdmissionLayer` (installed *inside* `ConcurrentBatchLayer`, so it sees single calls and every batch entry, since that layer decomposes batches into per-entry `call`s rather than delegating to an inner `batch`) only ever runs a non-blocking CAS against `--admission-max-concurrent` + `--admission-max-queue`, while the execution permit is taken in the handler *after* `check_cache` misses.
+That placement is load-bearing, not stylistic: `CancelGuard` arms on a request's first poll and the handler records its arrival synchronously in that same poll, so a middleware gate that parked before the handler would record a cancellation with no matching arrival for every client that hung up while queued — negative, permanent identity drift, worst under exactly the overload the gate exists for; a layer that only CAS-es preserves the invariant by construction, and the permit wait then sits after the arrival is already booked.
+It also means a response-cache hit never takes a permit, the typed `RequestShape` is already in hand so the heavy-tracer sub-cap (`--admission-heavy-max-concurrent`, covering `prestateTracer`/JS/`muxTracer` and *every* struct-logger request including the bare default — the flags separating the two struct-logger shapes change output size, not kind — taken *before* the ordinary permit, so heavy requests never occupy execution permits while waiting for each other, and bounded by their own share of the admitted budget — `heavy_max_concurrent x (1 + max_queue / max_concurrent)` — because the gate itself is class-blind, so without it a heavy flood fills the admitted budget while blocking on a sub-cap a fraction of its size and ordinary traffic is shed with most execution permits idle) costs no second parse of attacker-controlled JSON, and what the permits count is blocks actually being fetched and replayed.
+The permit wait is clamped to `deadline - witness_timeout` (`DataProvider::permit_cutoff`), so a request that queued away the budget its witness fetch still needs is refused now rather than started and timed out later — the difference between "may reject" and "times out"; the reserve falls back to half the budget when `--witness-timeout` does not fit inside `--block-fetch-timeout` (a legal pair, since the witness sub-deadline is a `min` against the outer one), because reserving all of it would leave a zero-length wait and silently reduce `--admission-max-queue` to a no-op.
+Every handler mints its deadline once and passes it to both the permit wait and the fetch (`get_block_data_by_hash`/`get_block_data_for_tx` take it as a parameter for exactly this reason), so the queue is carved out of `--block-fetch-timeout` rather than added on top of it.
+`debug_getCacheStatus` and its `timed_` alias are the sole exemption (matched on the `timed_`-stripped name, since the gateway adds that prefix by default and a bare-name comparison would never match production traffic); `assert_admission_covers_module` fails startup if a newly registered method escapes `metrics::GATED_METHODS`.
+The limiter uses a resizable FIFO `Semaphore` (shrink via `forget_permits` plus a debt counter settled on release) rather than the reference implementation's counter+`Notify`, which has a lost wakeup (its `Notified` is created *after* the capacity check) and never wakes parked waiters on a raise — both regression-tested here.
+Occupancy is counted, never derived as `limit - available_permits`: once a shrink leaves debt behind that difference reports the *new* limit, so the admin RPC would answer a retune by claiming it had already taken effect while every old holder was still resident (`occupancy_stays_truthful_while_a_shrink_is_outstanding`).
+`--max-response-size` is checked at this server's own serialization point (`RawJson::try_new` in `serialize_reply`/`compute_block_trace`, counted on `debug_trace_response_oversized_total`, classified `TraceError::Request` so an oversized ask never evicts a good block), and `--max-batch-response-size` separately caps the assembled batch — different bounds, because a batch retains every completed entry's body until it finishes, and that accumulation rather than any single response is what has previously exhausted the process; both `heavy_max_concurrent x max_response_size` and `max_concurrent x max_response_size` are logged at startup, and neither bounds a tracer's intermediate allocations — a per-transaction-count gate would, and is not implemented.
+`--admin-addr` (loopback enforced at startup, off unless set) serves `admin_getConcurrencyLimit`/`admin_setConcurrencyLimit` on a second listener with no RPC middleware — so a saturated public port cannot shed the call that relieves it — running on its own thread and runtime, since inline EVM tracing on the main runtime would otherwise starve it exactly when it is needed; `maxConcurrent = 0` is refused as unrecoverable.
 Every inbound request is accounted for exactly once, so "did any client see a timeout?" is a metric lookup rather than an inference: arrivals are counted by `debug_trace_request_shape_total` before the first await, and each request then lands in exactly one of `debug_trace_rpc_requests_total` (served), `debug_trace_rpc_errors_total{reason}` (failed), or `debug_trace_requests_cancelled_total` (client hung up — recorded from the drop of the request future in `rpc_middleware.rs`, the only layer that observes single calls and batch entries alike), so `shape = requests + errors + cancelled` holds per method.
+A shed adds the balanced pair `shape="shed"` / `reason="overloaded"`, recorded by the admission layer from *inside* the returned future — `record_rpc_error` sets the `ERROR_SELF_REPORTED` task-local that tells `settle_response` this non-framework `-32013` is already accounted for, which is what keeps `settle_response` unchanged; recorded from the layer's synchronous prefix instead, every shed would double-count its error and false-fire the `unattributed` drift alarm (`admission_shed_keeps_the_identity_closed` pins both).
+A permit refused after the handler already recorded its arrival contributes the outcome side alone.
 Requests the framework answers before any handler runs (unknown method, malformed top-level params, unparsable batch entries — recognized by their framework error codes) are folded in by the same middleware as the balanced pair `shape="rejected"` / `reason="rejected"`, an unrecorded error with a non-framework code lands error-side only on `reason="unattributed"` (a handler ran and recorded its arrival but bypassed the error funnel — a code-drift alarm that must stay at zero), and the opts-less `trace_*` methods record their `default` arrival at handler entry; the only deliberate approximation is a batch the server itself aborts over the response-size cap (its killed entries record no outcome rather than masquerade as client hangups) and batch entries that never started when the connection died (on neither side, so `cancelled` undercounts torn-down oversized batches).
 The identity is pinned end-to-end by `accounting_identity_holds_end_to_end` in `rpc_middleware.rs`, which drives every terminal path through a real server under a draining local metrics recorder.
 The `reason` label splits outcomes that share a JSON-RPC code — `deadline_witness` / `deadline_block` / `not_found` / `invalid_params` / `trace_failed` / `internal` — because a blown witness deadline and an unknown transaction both leave as `-32001` and only the first is an incident; the whole `(method, reason)` grid is pre-registered so an alert on `deadline_witness` reads zero instead of missing from boot.
@@ -163,13 +176,15 @@ The background chain-sync prefetch routes by freshness against the last observed
 | `bin/debug-trace-server/src/rpc_middleware.rs`                                                 | Concurrent execution of inbound JSON-RPC batch entries                               |
 | `bin/debug-trace-server/src/data_provider.rs`                                                  | Block data fetching with single-flight coalescing                                    |
 | `bin/debug-trace-server/src/block_data_cache.rs`                                               | Bounded in-memory `BlockData` cache keyed by block hash                              |
+| `bin/debug-trace-server/src/admission.rs`                                                      | Inbound admission gate: resizable FIFO limiter, guards, `RpcServiceT` layer          |
+| `bin/debug-trace-server/src/admin.rs`                                                          | Loopback-only `admin_*` listener for retuning the gate at runtime                    |
 | `bin/debug-trace-server/src/r2_witness.rs`                                                     | Direct-from-R2 witness source (light decode, deadline-aware)                         |
 | `bin/debug-trace-server/src/server_db.rs`                                                      | Defines + implements the bin-local `BlockStore` trait (backed by `stateless-db`)     |
 
 ## Test Organization
 
 Unit tests are embedded in source files alongside the code they test.
-Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, consistency, performance, timing_header, prune) and in `bin/stateless-validator/tests/integration.rs` (CLI parsing, mock-RPC pipeline, mainnet single-block validation).
+Integration tests live in `bin/debug-trace-server/tests/` (6 modules: cache_metrics, block_tag, compression, consistency, performance, timing_header) and in `bin/stateless-validator/tests/integration.rs` (CLI parsing, mock-RPC pipeline, mainnet single-block validation).
 Test data (block JSON files, contract bytecode, witness data) is stored in `test_data/`.
 
 ## Version Control