starknet_transaction_prover: track saturation and queue depth on the request path

[avi-starkware]

Adds the state /health will read in the next PR: a SaturationMonitor that
opens a window on the first busy-reject and closes it on any forward progress --
a request acquiring a worker slot, or a slot being released. Slot release has to
count, because once 503 pulls the load balancer's traffic away, a clear that
waited for a new request would latch 503 forever; a drop guard covers the
success, error and client-disconnect exits alike.

Both busy-reject arms go through record_busy_reject, which counts the outcome,
opens the window, warns and returns -32005 together, so a reject cannot count
without latching. Also exposes queue depth and queue-wait duration, bucketed at
the wait timeout.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

[reviewable-StarkWare]

This change is 

[avi-starkware]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• starknet_transaction_prover: log resolved config at startup (secrets redacted) #14684
• starknet_transaction_prover: origin-level breadcrumbs for validation errors #14172
• starknet_transaction_prover: /health returns 503 when service is saturated #14171
• starknet_transaction_prover: track saturation and queue depth on the request path #15027 👈 (View in Graphite)
• starknet_transaction_prover: bump prover_panics_total on panic hook fire #14170
• starknet_transaction_prover: HTTP request count + latency + in-flight metrics #14169
• starknet_transaction_prover: proving-job duration + outcome metrics #14168
• starknet_transaction_prover: Prometheus /metrics endpoint with build_info #14167
• starknet_transaction_prover: extract the TLS accept loop and test its drain #15049
• starknet_transaction_prover: graceful shutdown on SIGTERM/SIGINT #15026
• starknet_transaction_prover: global panic hook with structured log #14166
• starknet_transaction_prover: document existing logging, request ids and health probe in README #14203
• main-v0.14.4

This stack of pull requests is managed by Graphite. Learn more about stacking.

[cursor]

PR Summary

Medium Risk
Changes admission/queue behavior instrumentation and shared saturation state on the hot RPC path; reject counting and guard drop order affect operability signals ahead of health integration.

Overview
Adds SaturationMonitor state on the proving request path so a follow-up can distinguish sustained overload from one-off -32005 rejects. The window opens on the first busy-reject and clears on forward progress (worker slot acquired or released via SaturationClearGuard), with explicit ordering so slot release cannot leave saturation latched after traffic stops.

prove_transaction now routes both busy paths through record_busy_reject, which increments Prometheus outcome counters (rejected_queue_full, rejected_wait_timeout), opens saturation, logs, and returns -32005 together. While waiting for a worker, the service exposes prover_queue_waiting_requests (RAII gauge) and prover_queue_wait_duration_seconds on successful slot acquisition; proving duration histograms still exclude rejects. Metrics README and exporter pre-registration are updated accordingly.

Unit tests cover saturation timing, reject metrics, and guard/clear behavior without a full prove run.

^{Reviewed by Cursor Bugbot for commit 16eb35c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1d3075e. Configure here.}

[cursor]

Saturation latch on admission drop order

Medium Severity

_admission is declared before the clear guard, so it drops after mark_progress. A concurrent queue-full reject can reopen the saturation window in that gap with no SaturationClearGuard left to clear it. With max_queued_requests = 0 and max_concurrent_requests = 1, admission stays full until that late drop, so the window can latch open once load-balancer traffic drains — the failure mode this binding order was meant to prevent.

 

^{Reviewed by Cursor Bugbot for commit 1d3075e. Configure here.}