From d1cde13ab2193ea438653c98fbcb122bf5d09564 Mon Sep 17 00:00:00 2001 From: bdchatham Date: Thu, 9 Jul 2026 18:22:09 -0700 Subject: [PATCH 1/6] docs(runbooks): validate gas-vs-execution-time methodology end to end Replaces the eBPF-first draft with a validated RPC-native approach: debug_traceTransactionProfile gives clean per-tx gas + execution time in one call (r=0.9914 vs gas on real replayed pacific-1 txs, vs r=0.301 for whole-block latency). Documents the version-pinning requirement (app-hash mismatch on any binary divergence), the mock_chain_validation build mode for testing unreleased code against real transaction load, the Y=X-1-only limitation for Functionality 1, the debug_traceCall execution-time gap for Functionality 2, and the MaxTraceLookbackBlocks race condition on a fast-catching-up replayer. eBPF is demoted to a fallback tier, no longer the default path. --- .../benchmarking-evm-gas-vs-execution-time.md | 245 ++++++++++-------- 1 file changed, 133 insertions(+), 112 deletions(-) diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md index cbceee7..9f6f09b 100644 --- a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -1,10 +1,10 @@ # Benchmarking EVM Gas vs. Real Execution Time on Sei **Audience:** engineers (and their agents) on the gas-repricing project investigating whether Sei's EVM gas schedule tracks real compute cost. Covers both target functionalities: (1) replaying a real historical block against restored state and measuring gas + execution time per tx, and (2) executing arbitrary bytecode against a specific historical state and measuring the same. -**Scope:** standing up a replayer node (functionality 1) and an archive-backed environment (functionality 2) on harbor with confirmed prod-matching hardware, the privileged-attach gate for eBPF instrumentation, concrete probe design against real `seid`/go-ethereum symbols, and the confounds that will produce a false result if ignored. **No sei-chain source changes anywhere in this procedure** — everything here is external client traffic + kernel/process-level observation of a binary you don't touch. +**Scope:** standing up a replayer node, choosing the right binary version deliberately (§2), and using `debug_traceTransactionProfile` — a validated, existing RPC method — to get clean per-tx gas + execution-time pairs with a single call, no kernel instrumentation required for either functionality as specified. eBPF (§8) is a fallback for needs beyond what's documented here (e.g. true per-opcode dispatch timing), not the default path. **Not in scope:** *why* Sei's gas schedule has the specific values it does (a governance/design question — capture a resulting change proposal as a `/design` doc, not here). Running any of this against a shared/long-lived/production chain, or against the canonical shared archive node in a way that mutates its state. -> **Read this whole section before running anything.** Four things make the naive version of this experiment **silently wrong**: (1) at least one opcode's gas cost (`SSTORE`) is a **governance-mutable Sei chain param that already diverges from upstream go-ethereum** — using it as your "does gas match compute" baseline tests a value that was hand-tuned away from compute cost *by design*; (2) per-opcode timing via a uprobe **is not achievable** on this binary — the interpreter's opcode dispatch is an indirect call with no per-opcode symbol; (3) attaching any eBPF probe to a `seid` pod on harbor requires a **privileged kernel capability grant on a shared EKS cluster** — nothing in harbor's current namespace configuration technically blocks this, but that is an unhardened gap, not a green light; (4) **arbitrary-bytecode-against-historical-state (functionality 2) has an unverified RPC dependency** — confirm it before you build a campaign on top of an assumption. §1, §4, §5, and §6 exist to keep you out of these holes. +> **Read this whole section before running anything.** Six things make the naive version of this experiment **silently wrong**: (1) at least one opcode's gas cost (`SSTORE`) is a **governance-mutable Sei chain param that already diverges from upstream go-ethereum** — using it as your "does gas match compute" baseline tests a value that was hand-tuned away from compute cost *by design*; (2) **a replayer's binary version must match what actually produced the history you're replaying, or every block after the first will fail app-hash validation** — confirmed empirically this session (§2); (3) **`debug_traceTransactionProfile` and the rest of the `debug_trace*` family are capped to `MaxTraceLookbackBlocks` (default 10,000) blocks behind the node's current tip** — a fast-catching-up replayer ages a target height out of that window in minutes; trace it soon after it's produced (§4); (4) **only `Y=X-1` is supported for Functionality 1 by any tooling that exists today** — there is no code path for pinning a real historical block's transactions to an arbitrarily older base state (§4); (5) **Functionality 2's execution-time output has a real, unclosed gap** — `debug_traceCall` returns gas but no timing field for arbitrary bytecode against arbitrary `Y` (§5); (6) **attaching any eBPF probe to a shared harbor pod requires a privileged kernel capability grant** — nothing technically blocks it today, but that's an unhardened gap, not a green light (§8). --- @@ -14,18 +14,16 @@ |---|---| | `` | your harbor engineer alias; namespace is `eng-`. | | `` | the historical height whose post-block state you're restoring/executing against. | -| `` | the historical block (functionality 1 only) whose transactions you're re-executing and timing; `X > Y`. | +| `` | the historical block (Functionality 1 only) whose transactions you're re-executing and timing; in practice `X = Y+1` — see §4. | | `` | the `SeiNode` in `replayer` mode restoring to `` and replaying forward (§3). | -| `` | the archive-mode `SeiNode` or existing shared archive infra used for functionality 2 (§4). | -| `` | the single EVM opcode under test for a given functionality-2 run (§5). | -| `` | the PID of the `seid` process inside the target pod — resolve fresh each pod restart. | -| `` | your local `sei-chain` clone, used to resolve real symbol names before writing any probe (§7). | +| `` | the archive-mode `SeiNode` or existing shared archive infra used for deep historical `` reach (§5, §6). | +| `` | the sei-chain image the replayer runs — chosen deliberately per §2, never left at whatever default a template suggests. | --- ## 1. Mental model — what this can and cannot prove (read first) -The question: for a given opcode, does the gas charged scale with the real wall-clock cost of executing it? The honest answer requires knowing which parts of the gas schedule were **derived from measured compute** vs. **set by other means** (governance, upstream inheritance, a flat precompile fee). +The question: for a given opcode/tx, does the gas charged scale with the real wall-clock cost of executing it? The honest answer requires knowing which parts of the gas schedule were **derived from measured compute** vs. **set by other means** (governance, upstream inheritance, a flat precompile fee). **What's a fair target vs. a confound:** @@ -38,18 +36,66 @@ The question: for a given opcode, does the gas charged scale with the real wall- **Confounds that apply regardless of functionality:** - **Per-tx fixed overhead** (signature verification, the Cosmos ante handler, the EVM↔Cosmos StateDB bridge) is real wall-clock time attributable to *no* opcode — it dilutes a measurement unless your sample size / loop count makes it a small fraction of the total. -- **Optimistic concurrent execution (OCC) — this breaks naive per-tx correlation for functionality 1.** Sei applies a block's transactions through a **parallel scheduler**, not a serial loop: `DeliverTxBatch` → `tasks.NewScheduler(app.ConcurrencyWorkers(), …)` → `ProcessAll` spawns up to `ConcurrencyWorkers` goroutines (localnode default `concurrency-workers = 4`, `occ-enabled = true`; verified `app/abci.go:243`, `sei-cosmos/tasks/scheduler.go`), and re-executes any tx whose read/write set conflicts. **This same block-delivery path runs during replay** — a replayed historical block is applied by the OCC scheduler, not single-stepped. Consequences you must design around: - - **Functionality 2** (synthetic bytecode, §5) avoids all of this by construction — one tx per block means one task, no contention, no re-execution (§5). - - **Functionality 1** (replaying a real multi-tx block, §3) does **not** get clean per-tx timing for free: multiple `StateTransition.Execute` calls run concurrently on different OS threads, and a conflicting tx runs `Execute` more than once. An ordinal-by-call-count bracket (which is what §7's `tid`-keyed uprobe gives you) therefore maps to *neither* a specific tx *nor* a single execution — it silently produces **block-level aggregate timing, not per-tx timing**. See §3's correlation subsection for the fix (disable OCC for the measurement run) and its limits. - - **Determinism:** even setting correlation aside, OCC re-execution is not a deterministic function of the tx list alone (incarnation counts depend on scheduling), so repeated replays of the same `` can differ. If timing looks bimodal/inconsistent across repeated replays, suspect re-execution before suspecting your probe. +- **OCC (optimistic concurrent execution) does not break per-tx correlation the way it used to appear to.** Sei applies a block's transactions through a parallel scheduler (`app/abci.go`, `sei-cosmos/tasks/scheduler.go`), not a serial loop — but §4's measurement primitive (`debug_traceTransactionProfile`) isolates a single tx's execution by replaying the block *up to* that tx's index and then executing just it (`evmrpc/simulate.go`'s `replayTransactionTillIndex`), independent of how the block was originally scheduled. You do **not** need to disable OCC on the replayer to get clean per-tx numbers — that was true of a raw uprobe/uretprobe bracket around `StateTransition.Execute`, it is not true of this RPC. +- **Whole-block wall-clock time (log-line `latency_ms` on `"finalized block"`) is the wrong signal — verified empirically, not theoretically.** It's diluted by non-EVM per-block work (oracle vote tally, IBC, begin/end-blockers for every module) and, on a replayer still catching up, contention with block-fetching. A same-session sample: correlating gas against whole-block latency gave Pearson r=0.301; filtering to EVM-active blocks and using the OCC-scheduler's tx-execution-phase latency improved it to r=0.618, still with real outliers. Correlating gas against `debug_traceTransactionProfile`'s `executionNanos` on the same chain gave **r=0.9914** on a 9-tx sample. Use the RPC, not a log-derived proxy. --- -## 2. Hardware parity — already automatic, verify it rather than configure it +## 2. Version pinning — the replayer must match what actually produced the history, or every block after the first fails -**You do not need to request specific hardware.** `sei-k8s-controller` picks the Karpenter NodePool per node mode automatically: `internal/platform/platform.go`'s `NodepoolForMode()` — archive-mode nodes get `c.NodepoolArchive`, every other mode (including `replayer`) gets `c.NodepoolName` — and wires the matching toleration + required node affinity onto the pod spec (`internal/task/bootstrap_resources.go:203-227`, mirrored in `internal/noderesource/noderesource.go:454-479`). Both `clusters/harbor/sei-k8s-controller/controller-config.yaml` and `clusters/prod/sei-k8s-controller/controller-config.yaml` (in `sei-protocol/platform`) set the **identical** values: `nodepoolName: sei-node`, `nodepoolArchive: sei-archive`, `tolerationKey: sei.io/workload`. Harbor's `clusters/harbor/default/nodepool-{default,archive}.yaml` carry the **same** `r6i`/`r7i` instance-family, Nitro-hypervisor, on-demand requirements as `clusters/prod/default/nodepool-{default,archive}.yaml`. Net: a replayer or archive `SeiNode` in your `eng-` namespace lands on the same instance class as its prod counterpart, with zero engineer-side configuration. +**This is the load-bearing gotcha of the whole runbook, and it is not theoretical — it happened this session.** A replayer restores state via Tendermint state-sync (§3) and then blocksyncs forward, re-executing each real historical block against its own binary. If that binary's state-transition logic differs *at all* from what actually ran on the live chain — even one commit ahead of the released version — the resulting `AppHash` after the first replayed block will not match the real chain's recorded `AppHash` for the next block, and the node halts with `validateBlock(): wrong Block.Header.AppHash: expected X, got Y: app hash mismatch`, evicting every peer it asks for that block (they all report the same real historical data; the mismatch is your binary, not them). -**Verify it anyway before trusting a timing result** — a Karpenter requirements drift between clusters would silently invalidate every measurement: +**Confirm the live chain's actual version before choosing an image:** + +```sh +kubectl get seinode -n -o jsonpath='{.spec.image}{"\n"}' +``` + +Two deliberate choices, pick one per experiment — **do not default to "whatever image tag is lying around"**: + +### 2a. Pin to the released version matching the live chain — for a clean, zero-divergence baseline + +Use the exact tag the live chain runs (e.g. `ghcr.io/sei-protocol/sei:v6.5.2`). This gives you faithful replay of real history with **zero** validation errors — the right choice for "does the *currently released* gas schedule track *currently released* compute cost," which is almost certainly the baseline question the gas-repricing project needs before proposing any change. + +Verified this session: switching from an arbitrary dev-branch commit image to the release tag matching the live chain's actual version took the replayer from a permanent app-hash-mismatch eviction loop at the very first post-snapshot block to zero errors across tens of thousands of blocks. + +### 2b. `mock_chain_validation-` — for testing bleeding-edge/proposed code against real transaction load + +If you need to test code that **isn't released yet** (a proposed repricing change, a migration candidate) against real historical transaction patterns, sei-chain's CI builds a `mock_chain_validation` variant of essentially every commit. It's a Go build tag (`sei-tendermint/types/consensus_policy_mock_chain_validation.go`) that swallows the app-hash/results-hash/validator-set class of consensus errors (`ErrAppHash`, `ErrConsensusHash`, `ErrValidatorsHash`, `ErrUpgradeBeforeTrigger`, etc.) **while still fully verifying peer-delivered transaction data** (`ErrDataHash`/`ErrEvidenceHash` are never swallowed — a lying peer still can't inject fake transactions). It emits `sei_unsafe_validation_skipped_total` (a Prometheus counter labeled `validation_error`) every time it swallows one, so divergence is directly observable, not silent. + +**Find the image:** + +```sh +aws ecr describe-images --repository-name sei/sei-chain --region us-east-2 | \ + python3 -c " +import json,sys +d=json.load(sys.stdin) +imgs=[i for i in d['imageDetails'] for t in i.get('imageTags',[]) if 'mock_chain_validation' in t] +imgs.sort(key=lambda i: i['imagePushedAt'], reverse=True) +for i in imgs[:10]: print(i['imagePushedAt'], i.get('imageTags')) +" +``` + +On-demand builds are tagged `mock_chain_validation-`; nightly builds are `mock_chain_validation-nightly--`. Confirm the tag you pick actually contains the commit you care about (`git merge-base --is-ancestor ` in your `sei-chain` checkout) before using it. + +**Read the divergence rate before trusting anything measured under this mode:** + +```sh +kubectl port-forward -n eng- -0 :26660 +curl -s localhost:/metrics | grep sei_chain_sei_unsafe_validation_skipped_total +``` + +Verified this session: running `mock_chain_validation` built from `origin/main` (447 commits ahead of the last release) against real pacific-1 history produced **thousands of `app_hash` skips within the first few hundred blocks** — i.e. essentially every block's resulting state diverges from real history once you're running meaningfully different code. **This is expected, not a bug, and it compounds**: state at height `Y+N` reflects `N` blocks of the *new* binary's execution repeatedly applied to itself, not what actually existed historically at that height. That's fine and arguably desirable if your question is "does this new code's gas track this new code's own compute cost, exercised by realistic transaction load" — it is the wrong tool if your question requires reproducing exact historical state at a specific ``. Pick §2a for the latter. + +--- + +## 3. Hardware parity — already automatic, verify it rather than configure it + +**You do not need to request specific hardware.** `sei-k8s-controller` picks the Karpenter NodePool per node mode automatically: `internal/platform/platform.go`'s `NodepoolForMode()` — archive-mode nodes get `c.NodepoolArchive`, every other mode (including `replayer`) gets `c.NodepoolName` — and wires the matching toleration + required node affinity onto the pod spec (`internal/task/bootstrap_resources.go`, mirrored in `internal/noderesource/noderesource.go`). Both harbor's and prod's `sei-k8s-controller/controller-config.yaml` set the identical `nodepoolName`/`nodepoolArchive`/`tolerationKey` values, and both clusters' NodePool manifests carry the same `r6i`/`r7i` instance-family, Nitro-hypervisor, on-demand requirements. Net: a replayer or archive `SeiNode` in your `eng-` namespace lands on the same instance class as its prod counterpart, with zero engineer-side configuration. + +**Optionally**, for a throwaway single-tenant experiment node, apply the `sei.io/dedicated-node: "true"` annotation (see `sei-k8s-controller`'s dedicated-node feature) to guarantee the node isn't shared with another engineer's workload for the duration of the run — not required for correctness, but removes one source of noisy-neighbor variance from your timing numbers. + +**Verify hardware parity anyway before trusting a timing result:** ```sh NODE=$(kubectl -n eng- get pod -0 -o jsonpath='{.spec.nodeName}') @@ -60,55 +106,62 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive --- -## 3. Functionality 1 — replay block `` against state after ``, measure gas + time +## 4. Functionality 1 — replay block `` against state after ``, measure gas + time + +**Scope note, confirmed by reading the actual RPC/replayer code, not by inference: only `Y=X-1` is supported today.** Every path that replays a real historical block's transactions — this replayer, `debug_traceBlockByNumber`, `debug_traceTransactionProfile` — hardcodes the base state to exactly one block before the target (`evmrpc/simulate.go`'s `initializeBlock`: `prevBlockHeight := block.Number().Int64() - 1`). There is no hook anywhere for pinning execution to an arbitrarily older `Y`. If your actual need is the general case (e.g. isolating state-size effects by holding transactions fixed while varying the base height), that is new engineering, not a config change — flag it explicitly rather than silently reporting `Y=X-1` numbers as if they answered a different question. -**The state-restore-and-forward-replay mechanism is real, documented tooling** — this is the same machinery behind `validating-flatkv-memiavl-parity-via-sharded-replay.md`, used here for timing instead of storage-parity comparison. You do **not** need that runbook's flatKV-vs-memIAVL differential pair; stand up a single replayer engine. +### Steps -1. **Render a `replayer`-mode `SeiNode`** targeting ``: `spec.replayer.snapshot.s3.targetHeight: ` (per `validating-flatkv-memiavl-parity-via-sharded-replay.md:148`) — the seictl sidecar restores the highest snapshot `<= ` and syncs forward from there, peering a shared full-history archive over p2p as its block source. Render via the `/harbor-dev` skill's PR-based flow into `harbor-engineering-workspace`, same as any other `SeiNode`. -2. **Let it replay forward past ``.** There is no documented flag to halt exactly at a target height — the tooling (and its shadow-comparator sibling) is built for continuous mass replay (~40 blocks/s/node, paged in 100-block chunks for the parity use case). You don't need a halt, but note that you **cannot reactively bracket the `` window** — RPC height/logs report `` only *after* it commits (§3a). Instead, attach the probe *continuously across a replay span containing ``* and correlate afterward by height/tx; poll `.status`/RPC height or tail logs only to confirm `` has been applied and to bound the span. Letting the node continue replaying past `` is harmless. -3. **Gas is already there per-tx** — the replayed node has genuinely applied ``'s transactions; read `gasUsed` per tx the same way you would from any node (`seid query evm tx ` / `eth_getTransactionReceipt`), no comparator needed for a single-block read. -4. **Execution time is the actual gap.** Neither this replay mechanism nor its comparator captures timing anywhere — the comparator's three layers cover AppHash/`LastResultsHash`, per-tx `code`/`gasUsed`/`log`/`events`, and post-state storage/balance/code/nonce; there is no timing field in any of them. This is exactly what §7's eBPF instrumentation is for, bracketed around the height-`` transition you identified in step 2. +1. **Choose and pin the image deliberately (§2)** before rendering anything. +2. **Render a `replayer`-mode `SeiNode`** targeting ``: `spec.replayer.snapshot.s3.targetHeight: ` — the seictl sidecar restores the highest snapshot `<= ` and syncs forward from there, peering a shared full-history archive over p2p as its block source. Render via the `/harbor-dev` skill's PR-based flow into `harbor-engineering-workspace`, same as any other `SeiNode`. +3. **Let it replay forward past ``.** There is no documented flag to halt exactly at a target height. Poll `.status`/RPC height or tail logs to confirm `` has been applied; letting the node continue replaying past it is harmless. +4. **Get the tx hash(es) for ``:** `eth_getBlockByNumber(, true)` on the replayer's own EVM-RPC (port 8545) returns full transaction objects including `hash` and the block-level `gasUsed`. +5. **For each tx, call the one RPC that gives you both numbers:** -### 3a. Per-tx correlation inside a multi-tx block — the real constraint (read before trusting any per-tx number) + ```sh + curl -s -X POST -H "Content-Type: application/json" \ + --data '{"jsonrpc":"2.0","method":"debug_traceTransactionProfile","params":[""],"id":1}' \ + localhost:8545 + ``` -Bracketing `StateTransition.Execute` with a `tid`-keyed uprobe/uretprobe (§7) gives **block-level aggregate timing, not per-tx timing**, on any block with more than one tx — because the OCC scheduler runs several `Execute` calls concurrently on different threads and re-runs conflicting txs (§1). Two separate problems: (a) *which* invocation is *which* tx — `Execute()` takes **no arguments** (receiver-only; the tx lives in the `st.msg *Message` field), so there is no arg to read a hash from, and ordinal-by-call-count is ambiguous under concurrency + re-execution; (b) a `tid`-keyed entry/return pair can be split across threads by goroutine migration, worse under OCC's higher scheduling pressure. + The response's `result.trace.gas` is the real gas used. `result.profile.phases.executionNanos` is the isolated, clean per-tx EVM execution time — **use this field, not `result.profile.totalNanos`**, which also includes `replayHistoricalTxsNanos` (replaying the block up to this tx's index — real cost, but not *this tx's* cost) and `traceResultNanos` (tracing-instrumentation overhead itself). `result.profile.store.modules` additionally breaks down per-Cosmos-module store-access counts and timing if you need to attribute cost below the whole-tx level. -**Fix — serialize the measurement run by disabling OCC.** Render the replayer with `occ-enabled = false` and `concurrency-workers = 1` (app.toml keys; wired via `baseapp.SetOccEnabled`/`SetConcurrencyWorkers`, verified). The block-delivery path then takes its non-OCC serial branch (`!ctx.IsOCCEnabled()`) — one `Execute` at a time, in tx order, no conflict re-execution — so ordinal-by-call-count *does* map cleanly to the Nth tx in ``. **This does not corrupt the measurement you care about:** gas is charged per-tx independently of the parallel schedule, and a tx's own interpreter/state work is the same instruction stream serial or parallel; you are removing cross-tx cache/scheduler interference, not changing the tx's compute. Confirm the SeiNode/seictl spec actually exposes these app.toml keys before relying on it — **VERIFY**; if it doesn't, this is a `/harbor-dev` spec gap to close, not a workaround to skip. - - **If OCC cannot be disabled** on the replayer: state plainly that functionality 1 yields **block-level aggregate `Execute` time only** (sum over the block's txs + any re-executions), correlatable to the block's *total* `gasUsed`, not per-tx. Per-tx then requires a compiled CO-RE tool that reads `st.msg` from the receiver (pointer + resolved field offsets) and keys by goroutine id, not tid — fragile, version-pinned to the struct layout, and out of scope for a bpftrace one-liner. Don't claim per-tx numbers you can't attribute. - - **Determinism bonus:** serial application also removes the OCC re-execution non-determinism (§1), so repeated replays of `` are directly comparable. +6. **Do this promptly.** `debug_trace*` is capped to `MaxTraceLookbackBlocks` behind the node's current tip (default 10,000, `evmrpc/config/config.go`, configurable via `evm.max_trace_lookback_blocks`). A replayer still catching up can process 20-30 blocks/sec — 10,000 blocks ages out in under 10 minutes. If you collect a batch of tx hashes and then profile them later, some will fail with `"block number ... is beyond max lookback of 10000"`. Verified this session: exactly this happened mid-collection. Either profile each tx immediately after finding it, or wait until the node has fully caught up to live tip (stable height) before doing a bulk collection pass. -**Window detection is post-hoc, not reactive.** You cannot *react* to entering block ``: at ~40 blocks/s a block's application window is ~25 ms, and RPC height / logs only report `` *after* it has committed — by the time you observe it, the `Execute` calls are done. So the probe must be **attached and aggregating continuously across a replay span that contains ``**, then filtered afterward by height/tx. Under the disable-OCC serial path this is clean (aggregate per-tx in order, join to the tx list of `` after); with a fast continuous replay you also want to slow the arrival into `` (small `concurrency-workers`, or throttle the block source) so the window isn't lost in sampling noise. +7. **Sample size and interpretation.** A single tx pair doesn't tell you much; a same-session batch of 9 real txs (gas ranging ~31k to ~1.3M) gave Pearson r=0.9914 between gas and `executionNanos` — strong evidence the primitive is sound. Collect enough of a real range (different contracts, different gas magnitudes) before drawing a conclusion, and always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). --- -## 4. Functionality 2 — execute arbitrary bytecode against state after ``, measure gas + time +## 5. Functionality 2 — execute arbitrary bytecode against state after ``, measure gas + time -**This has an unverified dependency — confirm it before building anything on top of it.** Two paths, in preference order: +Two paths. Prefer Path B given the current state of the RPC surface — it fully reuses §4's validated primitive and closes the timing gap that Path A cannot. -### Path A (preferred) — read-only historical `eth_call` with a state override +### Path A — read-only `eth_call`/`debug_traceCall` (gas works, timing does not) -If Sei's EVM JSON-RPC supports `eth_call` with a historical `blockNumber`/`blockTag` **and** a state-override object (to simulate the bytecode as if already deployed, without a real mutating tx), this is the clean answer *for gas and correctness*: point it at an archive node's RPC, at height ``, no real mutating tx. **`operating-archive-node-byov.md` says nothing about this** — it's pure PV/EBS provisioning — so this capability is not documented anywhere in this repo's runbooks. **Confirm empirically before relying on it:** a trivial smoke test (a state-overridden `eth_call` for a known simple computation at a known historical height, checked against a hand-computed expected result). +`eth_call` and `debug_traceCall` both take a fully independent `blockNrOrHash` parameter — no `X` concept at all, so arbitrary bytecode/calldata against an arbitrary historical `Y` works structurally (`evmrpc/simulate.go`, `evmrpc/tracers.go`). But: -**Path A does not solve the *timing* half on the shared archive node.** Execution time here comes from §6/§7 eBPF instrumentation, and you may not attach a privileged probe to the **canonical shared archive node** — that is the §6 gate and the shared-blast-radius rule, and it is *not* your node to instrument. So Path A splits: -- **Gas / correctness:** the state-overridden `eth_call` against the shared archive is fine (read-only, no mutation). -- **Timing:** either (i) accept only a client-side RPC round-trip latency as a coarse, network-contaminated proxy (not eBPF-clean — state this limitation, don't dress it as compute time), or (ii) run the *same* state-overridden `eth_call` against **your own private archive/replay node** (restored to `` as in Path B step 1) which you *are* cleared to instrument. Option (ii) recovers eBPF-grade timing but forfeits Path A's "no new infrastructure" benefit — at which point Path A and Path B converge on "a private node you own," and the only difference is read-only `eth_call` vs. a real submitted tx. Choose Path A(ii) over Path B when the state-override RPC works, to avoid mutating even your private node. +- Plain `eth_call` returns **only return-data** — no gas, no time. +- `debug_traceCall` with `{"tracer":"callTracer"}` returns **real gas** in the JSON result — but **no execution-time field exists anywhere in this path**. The same `phaseDurations`/`ExecutionNanos` mechanism that powers §4's `debug_traceTransactionProfile` is not wired into `TraceCall` at all (confirmed by reading the call sites — only two places populate `phaseDurations`, and `TraceCall` isn't one of them). +- **This is a real, unclosed gap**, not a workaround-able limitation. Closing it is a small, well-scoped extension (thread the existing, already-working profiling mechanism into `TraceCall`) — but it's a sei-chain source change to a shared repo, out of scope for this runbook to make unilaterally. If Path A's read-only simulation (no real tx submitted) is a hard requirement, flag this gap to the team rather than fabricating a timing number. -### Path B (fallback) — private writable fork of state-at-`` +### Path B (preferred) — submit a real signed tx, then reuse §4's primitive -If Path A's RPC capability doesn't exist or doesn't support the injection you need, use the same state-restore mechanism as §3 but stop treating it as a passive replayer: restore a `SeiNode` to height `` (`spec.replayer.snapshot.s3.targetHeight: `, same as §3 step 1), then instead of continuing historical replay, submit your own synthetic signed EVM transactions against it (`seid tx evm deploy` / `call-contract`, §5's bytecode). This gives you real trie depth / cache-warmth from genuine historical state (unlike a from-scratch empty genesis chain) at the cost of standing up and mutating your own private copy — never do this against the canonical shared archive. +Restore a `SeiNode` to height `` exactly as in §4 step 2, then instead of continuing passive historical replay, submit your own synthetic signed EVM transaction against it (deploy + call, §6's differential-bytecode technique). Once it's mined, it's an ordinary historical tx — **`debug_traceTransactionProfile` on its hash gives you gas + `executionNanos` in one call**, identical to §4, with none of Path A's timing gap. This costs standing up and mutating your own private node (never the canonical shared archive) but fully closes both halves of Functionality 2's output contract with already-validated tooling. -Either path, **instrument the same node you're submitting to** — no cross-node clock correlation needed. +### Historical `` reach — pruning limits, verified live + +Your replayer/dedicated node prunes; the canonical shared archive doesn't. Checked directly this session: a replayer running Karpenter-default config had `pruning-keep-recent=86400`, `pruning-keep-every=500` (dense access to the last 86,400 blocks it has processed, sparse beyond that), while `p1-archive-canonical` ran `pruning=nothing` (full history to genesis). **For a genuinely old ``, point your tooling at the archive node's RPC, not a fresh replayer** — a replayer only has state back to wherever its own snapshot restore started forward from. Failure mode if you ask for a pruned height is loud, not silently wrong: `CacheMultiStoreWithVersion` returns `"failed to load state at height %d; %s (latest height: %d)"` — no risk of silently getting the wrong state. --- -## 5. Crafting opcode-isolated bytecode (Path B / any synthetic-tx run) +## 6. Crafting opcode-isolated bytecode (Path B / any synthetic-tx run) -**The differential-loop technique** isolates a single opcode's marginal cost without per-opcode instrumentation: write two bytecode variants identical except for the opcode under test. +**The differential-loop technique** isolates a single opcode's marginal cost without per-opcode instrumentation — still necessary even with §4/§5's primitive, because `debug_traceTransactionProfile`'s timing is per-tx/per-phase, not per-opcode; its `structLogs` give per-opcode *gas*, not per-opcode *time*. Write two bytecode variants identical except for the opcode under test. - **Variant A (opcode-under-test):** a loop body of `JUMPDEST`, the target `` (with operands pre-pushed), then loop-decrement/`JUMPI` back to `JUMPDEST`. - **Variant B (baseline):** identical scaffold, target opcode replaced with a cheap no-op pair (`PUSH1 0x00`/`POP`) of known cost. -`(wall-clock(A) − wall-clock(B)) ÷ iteration count ≈ per-opcode marginal wall-clock cost`; the same subtraction on `gasUsed` gives the marginal gas cost to compare against it. This cancels loop-overhead opcodes and per-tx fixed overhead from both sides symmetrically. +`(executionNanos(A) − executionNanos(B)) ÷ iteration count ≈ per-opcode marginal wall-clock cost`; the same subtraction on `gas` gives the marginal gas cost to compare against it. This cancels loop-overhead opcodes and per-tx fixed overhead from both sides symmetrically. ```sh seid tx evm deploy /path/to/-loop.bin \ @@ -119,76 +172,46 @@ seid tx evm call-contract \ --from= --value=0 --chain-id= --evm-rpc= ``` -Pick a large-but-fixed iteration count up front (large enough that fixed overhead is a small fraction of total gas/time; loop rather than unroll past a few hundred iterations — code-size cap). Run each variant multiple times and take the distribution, not a single sample — wall-clock has real variance from GC pauses, compaction, and scheduling noise that gas is blind to. - -**Submission protocol (Path B only):** submit exactly one transaction, wait for its block to finalize and its receipt to confirm, before submitting the next. This sidesteps the OCC re-execution confound (§1) by construction — nothing contends for the same slot if only one tx is ever in flight. `StateTransition.Execute` (§7) runs more than once per submitted tx — once during `eth_estimateGas`/`CheckTx` simulation, again at `FinalizeBlock` delivery — expect 2-3 invocations, and disambiguate by taking the one whose timestamp falls inside the actual block-production window. +Pick a large-but-fixed iteration count up front (large enough that fixed overhead is a small fraction of total gas/time; loop rather than unroll past a few hundred iterations — code-size cap). Run each variant multiple times and take the distribution, not a single sample — wall-clock has real variance from GC pauses, compaction, and scheduling noise that gas is blind to. Submit one tx at a time, wait for its receipt, before submitting the next — then call `debug_traceTransactionProfile` on each mined hash per §4/§5. --- -## 6. The privileged-attach gate — read before running `kubectl debug` - -Any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster** running other engineers' chains and `sei-k8s-controller` itself. This is root-on-node. The `/ebpf` skill's guardrail is unconditional here: **the gate is on the attach mechanism, not on how low-stakes the target is.** A throwaway replayer/archive-fork node does not waive it. +## 7. Reading results -**What's actually true about `eng-` today (verified against the platform repo, not assumed):** `eng-` namespaces carry no Pod Security Standard label (`clusters/harbor/engineers/base/namespace.yaml`), and harbor has no cluster-wide PSS default either (`clusters/harbor/admission/` contains only an unrelated tenant-hostname policy) — the effective PSS level is Kubernetes' own default, `privileged`, enforced nowhere. Compare **prod**'s `walle` namespace: explicit `restricted` PSS + a `ValidatingAdmissionPolicy` blocking ephemeral containers outright. Harbor's `eng-` has no equivalent. **This is an unhardened gap, not a sanctioned path**, and shared-node blast radius is real independent of PSS — engineer workloads bin-pack onto harbor's Karpenter pools alongside other tenants' processes. +For each opcode/tx under test: `gas` (Tier 0, free), `executionNanos` (§4/§5's primitive), and the live gas parameter value behind it if testing a governance-mutable opcode (§1). Report as a ratio (time-per-gas-unit) per opcode/tx, not a single aggregate. -**Do this, in order, before attaching anything:** - -1. **Get sign-off first** — surface what you're attaching, to what, on which pod, for how long, to the platform/security owner before running anything privileged. -2. **Pin to a dedicated node** — for the duration of the experiment, don't share the node with other tenants (§2's hardware-parity node is already tainted/dedicated by `sei-node`/`sei-archive`'s own scheduling — confirm nothing else landed there). -3. **Scope the capability grant to the minimum**, not `hostPID`: `kubectl debug --target=seid -it -0 --image= --profile=sysadmin` — `--target=` joins only that pod's PID namespace. Prefer `CAP_BPF`+`CAP_PERFMON` over the broader `sysadmin` profile where possible. -4. **If this becomes recurring**, that's the signal to open a platform PR for a dedicated profiling namespace with its own restricted-by-default PSS + a narrow, auditable exception (mirror `clusters/harbor/admission/tenant-hostname-policy.yaml`'s exception shape; `clusters/harbor/chaos-mesh/chaos-mesh.yaml` is harbor's existing precedent for an elevated-privilege, opt-in-gated workload) — **a platform decision, not something to self-approve.** +**Before concluding anything, re-check against §1:** a divergence on `SSTORE` confirms a known governance-set gap, not a finding; the real signal is `KECCAK256`/arithmetic. Precompile numbers belong in their own table, never blended with interpreted-opcode results. --- -## 7. Instrumentation — cheapest and least-privileged first +## 8. eBPF — only if §4/§7's primitive genuinely isn't enough + +**Don't reach for this by default — it is no longer the primary path for either functionality as specified.** `debug_traceTransactionProfile` already gives clean per-tx gas + execution time without any kernel-level instrumentation, validated this session (r=0.9914 gas-vs-`executionNanos` correlation on real replayed pacific-1 transactions). The only reason to reach for eBPF is a need genuinely beyond what §4-§7 answer — e.g. true per-opcode dispatch-level timing *within* a single tx's execution (the profiling RPC gives per-opcode *gas* via `structLogs` and per-*tx* timing, not per-opcode timing), or attributing time to something below the Go-function granularity the profiler's `phases`/`store.modules` breakdown already covers. -Don't reach for eBPF as step one. +If you do need it, the privileged-attach gate and symbol-resolution guidance below still apply. -**Tier 0 — free, no probe.** `gasUsed` from the receipt and `eth_estimateGas` are already there. Record the live gas parameters (e.g. `SeiSstoreSetGasEIP2200`, `x/evm/types/params.go:43`) and image digest before instrumenting anything — the comparison is meaningless without knowing what you're comparing against. +**The gate:** any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster**. This is root-on-node. `eng-` namespaces carry no Pod Security Standard label and harbor has no cluster-wide PSS default — the effective PSS level is Kubernetes' own default, `privileged`, enforced nowhere. **This is an unhardened gap, not a sanctioned path.** -**Tier 1 — Cosmos SDK's own pprof, no elevated privileges.** The rendered `config.toml`'s `pprof_laddr` gives `go tool pprof` access with zero Kubernetes privilege beyond what you already have. **Verify empirically whether this surfaces EVM-interpreter frames at all** on the running pod before relying on it — don't assume from a source read alone. If empty or unhelpful, move to Tier 2. +**Do this, in order, before attaching anything:** -**Tier 2 — eBPF (gated; §6 must be cleared first).** +1. Get sign-off first — surface what you're attaching, to what, on which pod, for how long, to the platform/security owner. +2. Pin to a dedicated node (§3's `sei.io/dedicated-node` annotation) for the duration. +3. Scope the capability grant to the minimum: `kubectl debug --target=seid -it -0 --image= --profile=sysadmin` — `--target=` joins only that pod's PID namespace. Prefer `CAP_BPF`+`CAP_PERFMON` over the broader `sysadmin` profile where possible. +4. If this becomes recurring, that's the signal to open a platform PR for a dedicated profiling namespace with its own restricted-by-default PSS + a narrow, auditable exception — a platform decision, not something to self-approve. -Resolve real symbols before writing any probe — `sei-chain`'s `go.mod` pulls go-ethereum via a `replace` directive, which **preserves the original import path in compiled symbol names**, so the binary's symbols are still `github.com/ethereum/go-ethereum/...`. Confirm on the actual binary: +Resolve real symbols before writing any probe — `sei-chain`'s `go.mod` pulls go-ethereum via a `replace` directive, which preserves the original import path in compiled symbol names: ```sh kubectl -n eng- debug --target=seid -it -0 --image= --profile=sysadmin -- \ go tool nm /proc//root/usr/bin/seid | grep -E 'StateTransition\)\.Execute|EVMInterpreter\)\.Run' ``` -The two real boundaries (confirm line numbers against `` before citing them to anyone): - -- **Per-tx bracket:** `github.com/ethereum/go-ethereum/core.(*StateTransition).Execute` (`core/state_transition.go`) — Sei's `x/evm/keeper/msg_server.go` (`applyEVMMessage`) calls this directly, bypassing the upstream `ApplyMessage` wrapper. -- **Per-opcode dispatch (not individually probeable):** `github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run` — indirect function-table dispatch, no stable per-opcode symbol. This is why §5 constructs isolation via bytecode rather than probing the loop. - -**Prefer `offcputime` and a PID-filtered on-CPU `profile` over raw entry/return timing** — the actual highest-value signal is whether `Execute`'s wall-clock time is CPU-bound (interpreter work, what gas is supposed to meter) or blocked on state I/O (which gas categorically does not meter). Both are map-aggregated, so overhead is event-rate-bound not drain-bound — but "near-zero" holds only for **functionality 2's low tx rate**. **On a functionality-1 replayer running ~40 blocks/s with heavy state I/O, the context-switch rate is high, and `offcputime` overhead scales with context-switch rate** (`/ebpf` skill's `pack-perf-methodology.md` §7 caveat) — it is bounded but no longer negligible, and it perturbs the very replay you're timing. Per guardrail 3 (which holds even off-prod, because a perturbing probe corrupts a *measurement*): **A/B the replay throughput (blocks/s) with the probe attached vs. detached before trusting any number**; if attaching the probe moves replay throughput, the timing is contaminated — narrow the window, drop the sampling rate, or serialize (§3a) so the rate is low. Note also that during replay there is **no `CheckTx`/`eth_estimateGas` path** (blocks arrive already-finalized over p2p), so — unlike §5's submission flow — `Execute` fires once per tx *per OCC incarnation*, not 2–3× per tx: - -``` -# on-CPU attribution, PID-filtered -bpftrace -e 'profile:hz:49 /pid == / { @[ustack] = count(); }' - -# off-CPU (blocked) time, PID-filtered -offcputime -p -f 30 -``` - -**Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only**, with two hazards to state plainly if you use it: (1) Go's stack-copying runtime and `uretprobe`'s return-address rewrite can interact badly — a known divergence, not hypothetical; (2) a goroutine can migrate OS threads mid-call on a blocking I/O read, breaking a `tid`-keyed entry/return pairing even at concurrency 1. - -``` -uprobe:/proc//root/usr/bin/seid:"github.com/ethereum/go-ethereum/core.(*StateTransition).Execute" -{ @start[tid] = nsecs; } -uretprobe:/proc//root/usr/bin/seid:"github.com/ethereum/go-ethereum/core.(*StateTransition).Execute" -/@start[tid]/ -{ @dur = hist(nsecs - @start[tid]); delete(@start[tid]); } -``` - ---- - -## 8. Reading results +- **Per-tx bracket:** `github.com/ethereum/go-ethereum/core.(*StateTransition).Execute` — Sei's `x/evm/keeper/msg_server.go` (`applyEVMMessage`) calls this directly, bypassing the upstream `ApplyMessage` wrapper. +- **Per-opcode dispatch (not individually probeable):** `github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run` — indirect function-table dispatch, no stable per-opcode symbol. This is why §6 constructs isolation via bytecode rather than probing the loop. -For each opcode/block under test: the differential (functionality 2) or bracketed (functionality 1) wall-clock time at the tier reached, the `gasUsed`, and the live gas parameter value behind it (Tier 0). Report as a ratio (time-per-gas-unit) per opcode/tx, not a single aggregate. +Prefer `offcputime` and a PID-filtered on-CPU `profile` over raw entry/return timing. On a replayer under heavy state I/O, `offcputime` overhead scales with context-switch rate and is no longer negligible — **A/B the replay throughput (blocks/s) with the probe attached vs. detached before trusting any number**. -**Before concluding anything, re-check against §1:** a divergence on `SSTORE` confirms a known governance-set gap, not a finding; the real signal is `KECCAK256`/arithmetic. Precompile numbers belong in their own table, never blended with interpreted-opcode results. For functionality 1, a bimodal or inconsistent time distribution across repeated replays of the same `` should raise the OCC-determinism assumption (§1) before any other explanation. +Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's stack-copying runtime and `uretprobe`'s return-address rewrite can interact badly, and a goroutine can migrate OS threads mid-call, breaking a `tid`-keyed entry/return pairing. --- @@ -196,41 +219,39 @@ For each opcode/block under test: the differential (functionality 2) or brackete | Symptom | Cause | Fix | |---|---|---| +| Replayer hits `wrong Block.Header.AppHash: expected X, got Y` and evicts every peer it asks | Binary version doesn't match what actually produced the history being replayed (§2) | Pin to the release tag matching the live chain (§2a), or deliberately use `mock_chain_validation` (§2b) if testing unreleased code and you accept compounding divergence from real history. | +| `debug_traceTransactionProfile`/`debug_traceCall` returns `"block number ... is beyond max lookback of 10000"` | Collected tx hashes, then profiled them later; the replayer's rapid catch-up aged the target height out of `MaxTraceLookbackBlocks` (§4 step 6) | Profile immediately after finding each tx, or wait for the node to fully catch up to live tip before a bulk pass. | +| Correlating gas against whole-block `latency_ms` gives a weak/inconsistent result | Whole-block wall time is diluted by non-EVM per-block work and (on a replayer) blocksync contention — verified this session (r=0.301) (§1) | Use `debug_traceTransactionProfile`'s `executionNanos`, not a log-derived block-level proxy (§4) — verified r=0.9914 on the same chain. | | `kubectl describe node` shows an instance type outside `r6i`/`r7i` | `controller-config.yaml` or NodePool requirements drifted between harbor and prod | Halt — the timing result won't reflect prod hardware. Diff `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml` and `clusters/{harbor,prod}/default/nodepool-*.yaml` before proceeding. | -| Can't get a clean read on block ``'s gas/results | Tried to force the replayer to halt exactly at `` instead of reading its result once the height passed | Let replay continue past ``; query ``'s results directly via RPC once the height has been applied (§3). | -| `eth_call` with a state override returns an error or silently ignores the override | Path A's RPC capability (§4) isn't actually supported on this `seid` version | Fall back to Path B — restore a private writable fork of state-at-`` and submit real signed txs. | -| Probe never fires / symbol not found | Assumed upstream go-ethereum's import path instead of resolving on the actual binary | Re-run the `go tool nm` resolution (§7) against `/proc//root/usr/bin/seid`. | -| Wall-clock measurements wildly inconsistent run-to-run | OCC re-execution (§1) — overlapping submissions (functionality 2) or parallel/re-executed scheduling on replay (functionality 1) | Functionality 2: strictly sequential one-tx-per-block. Functionality 1: render the replayer with `occ-enabled = false`, `concurrency-workers = 1` (§3a) for serial, deterministic, per-tx-correlatable application. | -| Functionality-1 per-tx timing doesn't line up with per-tx `gasUsed` | Bracketed a multi-tx block under OCC — got block-level aggregate, not per-tx (§1, §3a) | Disable OCC on the replayer (§3a) so ordinal-by-call-count maps to the Nth tx; if OCC can't be disabled, report block-level aggregate vs. block-total gas only. | -| Attaching the probe changes replay throughput (blocks/s) | High context-switch rate on a 40 blocks/s replayer makes `offcputime`/probe overhead non-negligible (§7, guardrail 3) | A/B throughput probe-on vs. probe-off; narrow the window, lower sampling rate, or serialize (§3a) until throughput is unperturbed. | -| Path A gives gas but no clean execution time | Tried to eBPF-attach the shared archive node (forbidden, §4/§6) | Run the state-overridden `eth_call` against your own private restore node (Path A(ii), §4), or fall back to Path B. | -| `seid` pod crashes after attaching a `uretprobe` | The Go-runtime/`uretprobe` interaction hazard (§7) | Expected risk on a throwaway node — redeploy and prefer `offcputime`/`profile` beyond a one-off rehearsal. | +| Path A (`debug_traceCall`) gives gas but no execution time | `TraceCall` was never wired to the `phaseDurations` profiling mechanism — a real, unclosed gap, not a config issue (§5) | Use Path B: submit a real signed tx against your own restored-to-`` node, then `debug_traceTransactionProfile` its mined hash. | +| Trying to replay block `X` against a `Y` that isn't `X-1` | No existing tooling supports the general case (§4) | This is new engineering, not a config change — confirm whether `Y=X-1` actually answers the real question before treating it as a blocker. | | "Gas doesn't match compute" conclusion on `SSTORE` | Tested a governance-mutable param as if compute-derived (§1) | Reframe as confirming the known divergence; re-run on `KECCAK256`/arithmetic for the real test. | -| Can't get sign-off / told to stop before attaching anything privileged | §6's gate working as intended | Platform team's call — escalate, don't self-approve a scoped-down version of the same attach. | +| Attaching an eBPF probe changes replay throughput (blocks/s) | High context-switch rate on a fast replayer makes `offcputime`/probe overhead non-negligible (§8) | A/B throughput probe-on vs. probe-off; narrow the window or lower sampling rate. Consider whether you need eBPF at all — §4's primitive may already answer the question. | +| Can't get sign-off / told to stop before attaching anything privileged | §8's gate working as intended | Platform team's call — escalate, don't self-approve a scoped-down version of the same attach. | --- ## 10. Pre-flight checklist -- [ ] Confirmed the target node landed on `r6i`/`r7i` via `sei-node`/`sei-archive` (§2) — don't trust the default without checking. -- [ ] Recorded live gas parameter values + image digest (§7 Tier 0). -- [ ] **Functionality 1:** replayer node restoring to `` and replaying forward (§3); confirmed per-tx `gasUsed` for `` is readable post-replay. -- [ ] **Functionality 1 (per-tx correlation):** if `` has >1 tx, replayer rendered with `occ-enabled = false` + `concurrency-workers = 1` (§3a) so serial ordinal correlation holds — or the report is explicitly scoped to block-level aggregate timing. Probe attached *continuously across a span containing ``* (window detection is post-hoc, not reactive — §3a), not started reactively on the height transition. -- [ ] **Functionality 2:** confirmed whether Path A's `eth_call` state-override capability actually exists on this `seid` version (§4) before committing to it; if not, Path B's private writable fork is staged instead. -- [ ] (Path B only) Differential bytecode pair built (§5), fixed high-enough iteration count, sequential one-tx-per-block submission harness logging `{submit_ts, tx_hash, block_height, gasUsed}`. -- [ ] Tier 0 and Tier 1 (pprof) tried and found insufficient before reaching for eBPF. -- [ ] **Sign-off obtained** for any privileged attach, capabilities scoped to `--target=seid` rather than `hostPID` (§6) — before running anything in §7 Tier 2. -- [ ] Symbols resolved against the actual running binary, not assumed from source alone (§7). +- [ ] Confirmed the target node landed on `r6i`/`r7i` via `sei-node`/`sei-archive` (§3) — don't trust the default without checking. +- [ ] **Chosen §2a (release-pinned) or §2b (`mock_chain_validation`) deliberately**, not defaulted to an arbitrary image tag. If §2b, confirmed `sei_unsafe_validation_skipped_total` is being watched, not ignored. +- [ ] **Functionality 1:** replayer restoring to `` and replaying forward (§4); confirmed `Y=X-1` actually answers the intended question, or explicitly flagged that it doesn't. +- [ ] **Functionality 1/2 measurement:** using `debug_traceTransactionProfile`'s `result.profile.phases.executionNanos` (not `totalNanos`, not a log-derived latency proxy) for timing, `result.trace.gas` for gas. +- [ ] **Traced promptly** relative to `MaxTraceLookbackBlocks` — not collecting a large batch of tx hashes to profile long after the fact on a still-catching-up node. +- [ ] **Functionality 2:** using Path B (real submitted tx + §4's primitive) unless a genuine read-only-simulation requirement forces Path A — in which case, the execution-time gap in `debug_traceCall` is flagged explicitly, not worked around with a fabricated number. +- [ ] Recorded live gas parameter values (e.g. `SeiSstoreSetGasEIP2200`) + image digest before drawing conclusions. +- [ ] Sample size covers a real range of gas magnitudes/contracts, not a single tx. +- [ ] Re-checked results against §1 (`SSTORE`/precompile confounds) before concluding anything about whether gas tracks compute. +- [ ] Confirmed eBPF is actually necessary (§8) before requesting sign-off for a privileged attach — §4-§7 already answer both functionalities as specified without it. --- ## References - Sei-EVM gas/precompile specifics: `/evm` skill — `kit-evm-parity-gas.md`, `kit-sei-precompiles.md`; `x/evm/types/params.go` in `sei-chain` for the governance-mutable param set. -- eBPF method, overhead-bounding, Go-uprobe/uretprobe divergences: `/ebpf` skill, `references/pack-perf-methodology.md`. +- Version-pinning / `mock_chain_validation`: `sei-chain` — `sei-tendermint/types/consensus_policy.go`, `consensus_policy_mock_chain_validation.go`, `consensus_policy_default.go`, `policy_metrics.go`; `.github/workflows/ecr.yml`, `.github/workflows/nightly-ecr.yml` for the image-tag scheme. +- Measurement primitive: `sei-chain` — `evmrpc/trace_profile.go` (`debug_traceTransactionProfile`, `profiledTraceTx`, `phaseDurations`), `evmrpc/simulate.go` (`initializeBlock`, `replayTransactionTillIndex`, `StateAtBlock`, plain `Call`), `evmrpc/tracers.go` (`TraceCall`), `evmrpc/block_trace_profiled.go` (`debug_traceBlockByNumber/Hash` — profiling exists but is passed `nil` here, not surfaced), `evmrpc/config/config.go` (`MaxTraceLookbackBlocks`). - Harbor chain spin-up mechanics: `/harbor-dev` skill, `references/ephemeral-chain-flow.md`, `references/seictl-cli.md`, `references/harbor-cluster.md`. -- Replay mechanism (functionality 1): `validating-flatkv-memiavl-parity-via-sharded-replay.md` (this directory) — the `replayer.snapshot.s3.targetHeight` restore, the shadow comparator's layer definitions (gas yes, timing no). -- Archive-node provisioning: `operating-archive-node-byov.md` (this directory) — infra only; does not document RPC query semantics, hence §4's Path A verification requirement. -- Hardware-parity mechanism: `sei-k8s-controller` — `internal/platform/platform.go` (`NodepoolForMode`), `internal/task/bootstrap_resources.go`, `internal/noderesource/noderesource.go`; `sei-protocol/platform` — `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml`, `clusters/{harbor,prod}/default/nodepool-{default,archive}.yaml`. +- Hardware-parity mechanism: `sei-k8s-controller` — `internal/platform/platform.go` (`NodepoolForMode`), `internal/task/bootstrap_resources.go`, `internal/noderesource/noderesource.go` (also home of the `sei.io/dedicated-node` annotation); `sei-protocol/platform` — `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml`, `clusters/{harbor,prod}/default/nodepool-{default,archive}.yaml`. - Harbor `eng-` Pod Security / admission posture: `sei-protocol/platform` — `clusters/harbor/engineers/base/namespace.yaml`, `clusters/prod/walle/{namespace,podcomposition-policy}.yaml` (the posture to eventually mirror), `clusters/harbor/chaos-mesh/chaos-mesh.yaml`, `clusters/harbor/admission/tenant-hostname-policy.yaml`. -- Execution entry points: `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here. +- Execution entry points (§8 only): `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here. From 065240e4ce8714cdf3fb1bdfeb04c8c15d95313c Mon Sep 17 00:00:00 2001 From: bdchatham Date: Thu, 9 Jul 2026 18:39:04 -0700 Subject: [PATCH 2/6] docs(runbooks): resolve xreview findings on the gas-vs-time runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four independent reviewers (prose-steward, kubernetes-specialist, systems-engineer as dissenter, security-specialist) found real gaps, all now fixed: - spec.peers was missing from the replayer render example (admission CEL rule requires it -- kubernetes-specialist) - the replayer is controller-flagged RPC-less (.status.endpoint absent); now documents port-forwarding 8545 directly (kubernetes-specialist) - "Tendermint state-sync" was the wrong term for the replayer's actual S3-snapshot restore mechanism (kubernetes-specialist) - the placeholder wired state-mutating §6 commands to shared archive infra; now scoped read-only with the boundary restated in §5/§6 (security-specialist) - "get sign-off first" now requires an affirmative response, not silence; added IMDS/node-credential blast radius, TTL/teardown, and fixed example commands contradicting their own minimize-caps advice (security-specialist) - §2b now states gas metering is untouched by mock_chain_validation, adds revert-rate monitoring alongside the skip counter, and restates the throwaway-node-only boundary (security-specialist, systems-engineer) - §5 no longer overstates debug_traceCall's historical reach -- it's capped by the same 10k-block lookback as the rest of debug_trace* (systems-engineer) - r=0.9914 now carries its n=9/one-session qualifier everywhere it's cited as justification; "447 commits" softened to a non-exact, rot-resistant phrasing (prose-steward, systems-engineer) - README.md's index entry updated to match the new methodology, was still describing the pre-rewrite eBPF-first draft (prose-steward) --- .agent/runbooks/README.md | 2 +- .../benchmarking-evm-gas-vs-execution-time.md | 62 ++++++++++++------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/.agent/runbooks/README.md b/.agent/runbooks/README.md index 673a442..cdbd1e9 100644 --- a/.agent/runbooks/README.md +++ b/.agent/runbooks/README.md @@ -32,7 +32,7 @@ Agents fetch this `README.md` to discover what's available, then `WebFetch` the | [`operating-archive-node-byov.md`](operating-archive-node-byov.md) | Required volume contents, PV/PVC spec, SeiNode/SeiNetwork spec, controller validation surface, and EBS-swap cutover sequence for archive nodes using the bring-your-own-volume (`dataVolume.import`) path. | Bringing up an archive node from a pre-populated EBS; swapping the underlying volume of an existing archive PV; debugging `ImportPVCReady=False`; receipt-store pruning concerns. | | [`migrating-validator-to-byo-secrets.md`](migrating-validator-to-byo-secrets.md) | Cutting a live validator from a legacy host onto the platform carrying its consensus identity via Secrets (`signingKey`/`nodeKey`): what migrates, SeiNetwork spec, controller validation surface, the stop-before-start double-sign discipline + layered equivocation defenses, cutover/rollback sequence, and dry-run gotchas. | Migrating an existing validator (e.g. arctic-1 node-19) off EC2 onto K8s; any cutover where a consensus key changes hosts; understanding the `replicas:1` CEL guard or the double-sign alerts. | | [`validating-flatkv-memiavl-parity-via-sharded-replay.md`](validating-flatkv-memiavl-parity-via-sharded-replay.md) | flatKV↔memIAVL storage-engine parity validation by differential historical replay: the flatKV+memIAVL replay-pair topology (same binary, same snapshot, blocks from a shared archive), the correctness gates (compare pair-not-archive; verify migration complete so flatKV reads are genuine; `historical_replay` build for pre-v6.5 txs), the seictl shadow comparator, result aggregation + Notion report, and the fan-out to 50+ shards. | Standing up a flatKV-vs-memIAVL correctness validation on harbor; driving a sharded replay campaign; debugging why a replay node is stuck or a comparison reads all-indeterminate/vacuous; understanding `migrate_evm` vs `memiavl_only`/`evm_migrated`/`flatkv_only` read routing. | -| [`benchmarking-evm-gas-vs-execution-time.md`](benchmarking-evm-gas-vs-execution-time.md) | Testing whether Sei's EVM gas schedule tracks real compute, for both (1) replaying a real historical block against restored state and (2) executing arbitrary bytecode against a specific historical state: why hardware parity with prod validators is automatic (`NodepoolForMode`, no engineer config needed), the replayer-node mechanism for functionality 1 and its real gap (gas is captured, execution time is not), the unverified `eth_call`-state-override dependency for functionality 2 and its private-writable-fork fallback, the confounds that invalidate a naive comparison (governance-mutable `SSTORE` gas, precompiles, OCC re-execution/determinism), the differential-loop bytecode-crafting technique for per-opcode isolation, the privileged-attach gate for eBPF on harbor's `eng-` namespaces (PSS posture, scoped `kubectl debug --target=`), and concrete probe design against real `seid`/go-ethereum symbols (why per-opcode uprobing doesn't work, why `offcputime`/`profile` beat raw uprobe timing). | Designing a gas-vs-compute benchmark; replaying a historical block for timing rather than storage-parity; deciding whether an opcode's gas cost is a fair compute proxy before trusting a result; attaching any eBPF probe to a harbor pod; resolving `seid`'s real symbol names before writing a probe. | +| [`benchmarking-evm-gas-vs-execution-time.md`](benchmarking-evm-gas-vs-execution-time.md) | Testing whether Sei's EVM gas schedule tracks real compute, for both (1) replaying a real historical block against restored state and (2) executing arbitrary bytecode against a specific historical state: the validated RPC-native measurement primitive (`debug_traceTransactionProfile` gives clean per-tx gas + execution time in one call, no eBPF required — replacing an earlier eBPF-first draft), the version-pinning discipline a replayer needs to avoid an app-hash-mismatch halt (including the `mock_chain_validation` build mode for testing unreleased code against real transaction load), why hardware parity with prod validators is automatic (`NodepoolForMode`, no engineer config needed), the scope limits confirmed by reading the actual code (only `Y=X-1` is supported for functionality 1; functionality 2's `debug_traceCall` path has a real execution-time gap), the confounds that invalidate a naive comparison (governance-mutable `SSTORE` gas, precompiles, whole-block-latency being the wrong timing proxy), the differential-loop bytecode-crafting technique for per-opcode isolation, and eBPF (§8) demoted to a fallback tier for needs beyond what's specified. | Designing a gas-vs-compute benchmark; replaying a historical block for timing; choosing between a release-pinned replayer and `mock_chain_validation`; deciding whether an opcode's gas cost is a fair compute proxy before trusting a result; understanding why `Y=X-1` is the supported case, not the general one; attaching any eBPF probe to a harbor pod only if the RPC primitive genuinely isn't enough. | ## Adding a new runbook diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md index 9f6f09b..83900c7 100644 --- a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -16,7 +16,7 @@ | `` | the historical height whose post-block state you're restoring/executing against. | | `` | the historical block (Functionality 1 only) whose transactions you're re-executing and timing; in practice `X = Y+1` — see §4. | | `` | the `SeiNode` in `replayer` mode restoring to `` and replaying forward (§3). | -| `` | the archive-mode `SeiNode` or existing shared archive infra used for deep historical `` reach (§5, §6). | +| `` | the archive-mode `SeiNode` or existing shared archive infra used for deep historical `` reach — **read-only queries only** (§5's Path A). Never a target for §6's `seid tx evm` submission commands, which mutate state — those go only to your own private node. | | `` | the sei-chain image the replayer runs — chosen deliberately per §2, never left at whatever default a template suggests. | --- @@ -36,14 +36,14 @@ The question: for a given opcode/tx, does the gas charged scale with the real wa **Confounds that apply regardless of functionality:** - **Per-tx fixed overhead** (signature verification, the Cosmos ante handler, the EVM↔Cosmos StateDB bridge) is real wall-clock time attributable to *no* opcode — it dilutes a measurement unless your sample size / loop count makes it a small fraction of the total. -- **OCC (optimistic concurrent execution) does not break per-tx correlation the way it used to appear to.** Sei applies a block's transactions through a parallel scheduler (`app/abci.go`, `sei-cosmos/tasks/scheduler.go`), not a serial loop — but §4's measurement primitive (`debug_traceTransactionProfile`) isolates a single tx's execution by replaying the block *up to* that tx's index and then executing just it (`evmrpc/simulate.go`'s `replayTransactionTillIndex`), independent of how the block was originally scheduled. You do **not** need to disable OCC on the replayer to get clean per-tx numbers — that was true of a raw uprobe/uretprobe bracket around `StateTransition.Execute`, it is not true of this RPC. +- **OCC (optimistic concurrent execution) does not break per-tx correlation for §4's RPC primitive.** Sei applies a block's transactions through a parallel scheduler (`app/abci.go`, `sei-cosmos/tasks/scheduler.go`), not a serial loop. A raw uprobe/uretprobe bracket around `StateTransition.Execute` (§8) *would* need OCC disabled to get clean per-tx numbers, since multiple `Execute` calls can run concurrently on different threads. `debug_traceTransactionProfile` sidesteps this entirely: it isolates a single tx's execution by replaying the block *up to* that tx's index and then executing just it (`evmrpc/simulate.go`'s `replayTransactionTillIndex`), independent of how the block was originally scheduled. You do **not** need to disable OCC on the replayer to get clean per-tx numbers via this RPC. - **Whole-block wall-clock time (log-line `latency_ms` on `"finalized block"`) is the wrong signal — verified empirically, not theoretically.** It's diluted by non-EVM per-block work (oracle vote tally, IBC, begin/end-blockers for every module) and, on a replayer still catching up, contention with block-fetching. A same-session sample: correlating gas against whole-block latency gave Pearson r=0.301; filtering to EVM-active blocks and using the OCC-scheduler's tx-execution-phase latency improved it to r=0.618, still with real outliers. Correlating gas against `debug_traceTransactionProfile`'s `executionNanos` on the same chain gave **r=0.9914** on a 9-tx sample. Use the RPC, not a log-derived proxy. --- ## 2. Version pinning — the replayer must match what actually produced the history, or every block after the first fails -**This is the load-bearing gotcha of the whole runbook, and it is not theoretical — it happened this session.** A replayer restores state via Tendermint state-sync (§3) and then blocksyncs forward, re-executing each real historical block against its own binary. If that binary's state-transition logic differs *at all* from what actually ran on the live chain — even one commit ahead of the released version — the resulting `AppHash` after the first replayed block will not match the real chain's recorded `AppHash` for the next block, and the node halts with `validateBlock(): wrong Block.Header.AppHash: expected X, got Y: app hash mismatch`, evicting every peer it asks for that block (they all report the same real historical data; the mismatch is your binary, not them). +**This is the load-bearing gotcha of the whole runbook, and it is not theoretical — it happened this session.** A replayer restores state via an S3 snapshot restore (§4 — not Tendermint's p2p state-sync variant; the `SeiNode` CRD supports both, but a `replayer` requires the `s3` source) and then blocksyncs forward, re-executing each real historical block against its own binary. If that binary's state-transition logic differs *at all* from what actually ran on the live chain — even one commit ahead of the released version — the resulting `AppHash` after the first replayed block will not match the real chain's recorded `AppHash` for the next block, and the node halts with `validateBlock(): wrong Block.Header.AppHash: expected X, got Y: app hash mismatch`, evicting every peer it asks for that block (they all report the same real historical data; the mismatch is your binary, not them). **Confirm the live chain's actual version before choosing an image:** @@ -61,8 +61,12 @@ Verified this session: switching from an arbitrary dev-branch commit image to th ### 2b. `mock_chain_validation-` — for testing bleeding-edge/proposed code against real transaction load +**Only ever the image of your own throwaway replayer in `eng-` — never merge this tag onto a shared, long-lived, or archive `SeiNode`.** Image selection for any `SeiNode` flows through the `/harbor-dev` skill's PR-based render into `harbor-engineering-workspace`, a repo shared across engineers; nothing stops you from pointing this tag at the wrong manifest, and a shared node running it would silently accept diverged state while still looking healthy. Confirm the target is your own disposable node before merging. + If you need to test code that **isn't released yet** (a proposed repricing change, a migration candidate) against real historical transaction patterns, sei-chain's CI builds a `mock_chain_validation` variant of essentially every commit. It's a Go build tag (`sei-tendermint/types/consensus_policy_mock_chain_validation.go`) that swallows the app-hash/results-hash/validator-set class of consensus errors (`ErrAppHash`, `ErrConsensusHash`, `ErrValidatorsHash`, `ErrUpgradeBeforeTrigger`, etc.) **while still fully verifying peer-delivered transaction data** (`ErrDataHash`/`ErrEvidenceHash` are never swallowed — a lying peer still can't inject fake transactions). It emits `sei_unsafe_validation_skipped_total` (a Prometheus counter labeled `validation_error`) every time it swallows one, so divergence is directly observable, not silent. +**Gas metering itself is untouched by this build tag** — the swallowed errors are all consensus-hash comparisons; `sei-cosmos/store/gaskv/store.go`'s `ConsumeGas` calls (the actual gas accounting) are identical to the default build. The tag only swaps which validation failures halt vs. continue, not how gas is charged — relevant reassurance given this whole runbook is a gas benchmark. + **Find the image:** ```sh @@ -85,13 +89,15 @@ kubectl port-forward -n eng- -0 :26660 curl -s localhost:/metrics | grep sei_chain_sei_unsafe_validation_skipped_total ``` -Verified this session: running `mock_chain_validation` built from `origin/main` (447 commits ahead of the last release) against real pacific-1 history produced **thousands of `app_hash` skips within the first few hundred blocks** — i.e. essentially every block's resulting state diverges from real history once you're running meaningfully different code. **This is expected, not a bug, and it compounds**: state at height `Y+N` reflects `N` blocks of the *new* binary's execution repeatedly applied to itself, not what actually existed historically at that height. That's fine and arguably desirable if your question is "does this new code's gas track this new code's own compute cost, exercised by realistic transaction load" — it is the wrong tool if your question requires reproducing exact historical state at a specific ``. Pick §2a for the latter. +Verified this session (illustrative, not a permanent number — re-check `origin/main`'s actual distance from the last release yourself): running `mock_chain_validation` built from `origin/main`, hundreds of commits ahead of the last release, against real pacific-1 history produced **thousands of `app_hash` skips within the first few hundred blocks** — i.e. essentially every block's resulting state diverges from real history once you're running meaningfully different code. **This is expected, not a bug, and it compounds**: state at height `Y+N` reflects `N` blocks of the *new* binary's execution repeatedly applied to itself, not what actually existed historically at that height. That's fine and arguably desirable if your question is "does this new code's gas track this new code's own compute cost, exercised by realistic transaction load" — it is the wrong tool if your question requires reproducing exact historical state at a specific ``. Pick §2a for the latter. + +**`sei_unsafe_validation_skipped_total` tells you divergence is happening — expected, by design — but not whether it has corrupted your measurement.** Watch tx success/revert rate alongside it. Once state has compounded far enough from reality, replayed transactions can start executing against genuinely unrealistic state (wrong balances, wrong nonces, wrong contract state) — early reverts and short-circuited paths, not real compute. That would corrupt even the "does new code's gas track new code's own compute" question this mode is otherwise good for, and the skip counter alone won't tell you it happened (it's *supposed* to be large under this mode). If the revert rate climbs, treat later measurements as suspect regardless of what the skip counter says. --- ## 3. Hardware parity — already automatic, verify it rather than configure it -**You do not need to request specific hardware.** `sei-k8s-controller` picks the Karpenter NodePool per node mode automatically: `internal/platform/platform.go`'s `NodepoolForMode()` — archive-mode nodes get `c.NodepoolArchive`, every other mode (including `replayer`) gets `c.NodepoolName` — and wires the matching toleration + required node affinity onto the pod spec (`internal/task/bootstrap_resources.go`, mirrored in `internal/noderesource/noderesource.go`). Both harbor's and prod's `sei-k8s-controller/controller-config.yaml` set the identical `nodepoolName`/`nodepoolArchive`/`tolerationKey` values, and both clusters' NodePool manifests carry the same `r6i`/`r7i` instance-family, Nitro-hypervisor, on-demand requirements. Net: a replayer or archive `SeiNode` in your `eng-` namespace lands on the same instance class as its prod counterpart, with zero engineer-side configuration. +**You do not need to request specific hardware.** `sei-k8s-controller` picks the Karpenter NodePool per node mode automatically: `internal/platform/platform.go`'s `NodepoolForMode()` — archive-mode nodes get `c.NodepoolArchive`, every other mode (including `replayer`) gets `c.NodepoolName` — and wires the matching toleration + required node affinity onto the pod spec (`internal/task/bootstrap_resources.go`, mirrored in `internal/noderesource/noderesource.go`). Harbor's and prod's `sei-k8s-controller/controller-config.yaml` are designed to carry identical `nodepoolName`/`nodepoolArchive`/`tolerationKey` values, and both clusters' NodePool manifests are designed to carry the same `r6i`/`r7i` instance-family, Nitro-hypervisor, on-demand requirements — by design, not independently re-verified here each time. Net: a replayer or archive `SeiNode` in your `eng-` namespace *should* land on the same instance class as its prod counterpart, with zero engineer-side configuration — confirm it with the step below rather than trusting the design intent, since drift between the two configs is exactly the failure mode that step catches. **Optionally**, for a throwaway single-tenant experiment node, apply the `sei.io/dedicated-node: "true"` annotation (see `sei-k8s-controller`'s dedicated-node feature) to guarantee the node isn't shared with another engineer's workload for the duration of the run — not required for correctness, but removes one source of noisy-neighbor variance from your timing numbers. @@ -113,10 +119,11 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive ### Steps 1. **Choose and pin the image deliberately (§2)** before rendering anything. -2. **Render a `replayer`-mode `SeiNode`** targeting ``: `spec.replayer.snapshot.s3.targetHeight: ` — the seictl sidecar restores the highest snapshot `<= ` and syncs forward from there, peering a shared full-history archive over p2p as its block source. Render via the `/harbor-dev` skill's PR-based flow into `harbor-engineering-workspace`, same as any other `SeiNode`. +2. **Render a `replayer`-mode `SeiNode`** targeting ``: `spec.replayer.snapshot.s3.targetHeight: ` **and `spec.peers`** — the CRD's admission rule rejects a replayer with no `peers` set (`api/v1alpha1/seinode_types.go`'s CEL rule: "peers is required when replayer mode is set"), so the render is incomplete without it. Point `peers` at a real historical block source (a shared full-history archive's p2p address, or a `label`/`selector` peer within your namespace) — the seictl sidecar restores the highest snapshot `<= ` and syncs forward from there. Render via the `/harbor-dev` skill's PR-based flow into `harbor-engineering-workspace`, same as any other `SeiNode`. 3. **Let it replay forward past ``.** There is no documented flag to halt exactly at a target height. Poll `.status`/RPC height or tail logs to confirm `` has been applied; letting the node continue replaying past it is harmless. -4. **Get the tx hash(es) for ``:** `eth_getBlockByNumber(, true)` on the replayer's own EVM-RPC (port 8545) returns full transaction objects including `hash` and the block-level `gasUsed`. -5. **For each tx, call the one RPC that gives you both numbers:** +4. **A replayer's EVM-RPC is reachable but not advertised — port-forward directly, don't look for it in `.status`.** The controller treats `replayer` mode as an RPC-less restore workload (`internal/controller/node/endpoints.go`'s `servesEVM()` returns `false` for it), so `.status.endpoint` is absent for a replayer regardless of whether the underlying `seid` process actually serves EVM traffic. It does — this session's measurements ran against it — but you must reach it via `kubectl port-forward -0 8545:8545`, not by reading it off the SeiNode's status. If a future controller change acts on the "RPC-less" intent by actually disabling EVM on replayers, this whole runbook's measurement path breaks; verify 8545 responds (`eth_blockNumber`) before building anything on top of it. +5. **Get the tx hash(es) for ``:** `eth_getBlockByNumber(, true)` on the replayer's own EVM-RPC (port 8545) returns full transaction objects including `hash` and the block-level `gasUsed`. +6. **For each tx, call the one RPC that gives you both numbers:** ```sh curl -s -X POST -H "Content-Type: application/json" \ @@ -126,9 +133,9 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive The response's `result.trace.gas` is the real gas used. `result.profile.phases.executionNanos` is the isolated, clean per-tx EVM execution time — **use this field, not `result.profile.totalNanos`**, which also includes `replayHistoricalTxsNanos` (replaying the block up to this tx's index — real cost, but not *this tx's* cost) and `traceResultNanos` (tracing-instrumentation overhead itself). `result.profile.store.modules` additionally breaks down per-Cosmos-module store-access counts and timing if you need to attribute cost below the whole-tx level. -6. **Do this promptly.** `debug_trace*` is capped to `MaxTraceLookbackBlocks` behind the node's current tip (default 10,000, `evmrpc/config/config.go`, configurable via `evm.max_trace_lookback_blocks`). A replayer still catching up can process 20-30 blocks/sec — 10,000 blocks ages out in under 10 minutes. If you collect a batch of tx hashes and then profile them later, some will fail with `"block number ... is beyond max lookback of 10000"`. Verified this session: exactly this happened mid-collection. Either profile each tx immediately after finding it, or wait until the node has fully caught up to live tip (stable height) before doing a bulk collection pass. +7. **Do this promptly.** `debug_trace*` is capped to `MaxTraceLookbackBlocks` behind the node's current tip (default 10,000, `evmrpc/config/config.go`, configurable via `evm.max_trace_lookback_blocks`). A replayer still catching up can process 20-30 blocks/sec — 10,000 blocks ages out in under 10 minutes. If you collect a batch of tx hashes and then profile them later, some will fail with `"block number ... is beyond max lookback of 10000"`. Verified this session: exactly this happened mid-collection. Either profile each tx immediately after finding it, or wait until the node has fully caught up to live tip (stable height) before doing a bulk collection pass. -7. **Sample size and interpretation.** A single tx pair doesn't tell you much; a same-session batch of 9 real txs (gas ranging ~31k to ~1.3M) gave Pearson r=0.9914 between gas and `executionNanos` — strong evidence the primitive is sound. Collect enough of a real range (different contracts, different gas magnitudes) before drawing a conclusion, and always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). +8. **Sample size and interpretation.** A single tx pair doesn't tell you much. **The following number is a same-session illustration of the primitive working, not a general finding about Sei's gas schedule — treat it as sizing the tool, not the research conclusion:** a 9-tx sample (gas ranging ~31k to ~1.3M) gave Pearson r=0.9914 between gas and `executionNanos`, against r=0.301 for the whole-block-latency proxy on the same chain. Neither number is independently reproducible from source — they're this session's measurements, not constants; re-derive your own before citing either. Collect enough of a real range (different contracts, different gas magnitudes) before drawing a conclusion, and always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). --- @@ -138,15 +145,17 @@ Two paths. Prefer Path B given the current state of the RPC surface — it fully ### Path A — read-only `eth_call`/`debug_traceCall` (gas works, timing does not) -`eth_call` and `debug_traceCall` both take a fully independent `blockNrOrHash` parameter — no `X` concept at all, so arbitrary bytecode/calldata against an arbitrary historical `Y` works structurally (`evmrpc/simulate.go`, `evmrpc/tracers.go`). But: +`eth_call` and `debug_traceCall` both take a fully independent `blockNrOrHash` parameter — no `X` concept at all, so arbitrary bytecode/calldata against an arbitrary historical `Y` works structurally for the *call* itself (`evmrpc/simulate.go`, `evmrpc/tracers.go`). But: -- Plain `eth_call` returns **only return-data** — no gas, no time. -- `debug_traceCall` with `{"tracer":"callTracer"}` returns **real gas** in the JSON result — but **no execution-time field exists anywhere in this path**. The same `phaseDurations`/`ExecutionNanos` mechanism that powers §4's `debug_traceTransactionProfile` is not wired into `TraceCall` at all (confirmed by reading the call sites — only two places populate `phaseDurations`, and `TraceCall` isn't one of them). +- Plain `eth_call` returns **only return-data** — no gas, no time. It is not subject to the lookback cap below, so it reaches genuinely arbitrary depth — at the cost of giving you neither number this runbook cares about. +- `debug_traceCall` with `{"tracer":"callTracer"}` returns **real gas** in the JSON result — but **no execution-time field exists anywhere in this path**, and **it is capped by the same `MaxTraceLookbackBlocks` as the rest of `debug_trace*`** (`evmrpc/tracers.go`'s `TraceCall` calls the same lookback guard). So Path A's gas-bearing option is bounded to recent history, not "arbitrary ``" — only the timing-blind, gas-blind plain `eth_call` reaches deep history. The `phaseDurations`/`ExecutionNanos` mechanism that powers §4's `debug_traceTransactionProfile` is simply not wired into `TraceCall` at all (confirmed by reading the call sites — one place populates `phaseDurations`, `TraceCall` isn't it). - **This is a real, unclosed gap**, not a workaround-able limitation. Closing it is a small, well-scoped extension (thread the existing, already-working profiling mechanism into `TraceCall`) — but it's a sei-chain source change to a shared repo, out of scope for this runbook to make unilaterally. If Path A's read-only simulation (no real tx submitted) is a hard requirement, flag this gap to the team rather than fabricating a timing number. ### Path B (preferred) — submit a real signed tx, then reuse §4's primitive -Restore a `SeiNode` to height `` exactly as in §4 step 2, then instead of continuing passive historical replay, submit your own synthetic signed EVM transaction against it (deploy + call, §6's differential-bytecode technique). Once it's mined, it's an ordinary historical tx — **`debug_traceTransactionProfile` on its hash gives you gas + `executionNanos` in one call**, identical to §4, with none of Path A's timing gap. This costs standing up and mutating your own private node (never the canonical shared archive) but fully closes both halves of Functionality 2's output contract with already-validated tooling. +Restore a `SeiNode` to height `` exactly as in §4 step 2, then instead of continuing passive historical replay, submit your own synthetic signed EVM transaction against it (deploy + call, §6's differential-bytecode technique). Once it's mined, it's an ordinary historical tx — **`debug_traceTransactionProfile` on its hash gives you gas + `executionNanos` in one call**, identical to §4, with none of Path A's timing gap. + +**This mutates state — target only your own private restored node, never the canonical shared archive or any other engineer's `SeiNode`.** §6's `seid tx evm deploy`/`call-contract` commands submit real signed transactions; point `--evm-rpc` at your own node's address, confirmed before every run, not at shared archive infra. This costs standing up and mutating your own private node but fully closes both halves of Functionality 2's output contract with already-validated tooling. ### Historical `` reach — pruning limits, verified live @@ -163,6 +172,8 @@ Your replayer/dedicated node prunes; the canonical shared archive doesn't. Check `(executionNanos(A) − executionNanos(B)) ÷ iteration count ≈ per-opcode marginal wall-clock cost`; the same subtraction on `gas` gives the marginal gas cost to compare against it. This cancels loop-overhead opcodes and per-tx fixed overhead from both sides symmetrically. +**`--evm-rpc` below must point at your own private node — never the canonical shared archive or a node you don't own.** These commands submit real signed, state-mutating transactions. + ```sh seid tx evm deploy /path/to/-loop.bin \ --from= --chain-id= --evm-rpc= \ @@ -178,7 +189,7 @@ Pick a large-but-fixed iteration count up front (large enough that fixed overhea ## 7. Reading results -For each opcode/tx under test: `gas` (Tier 0, free), `executionNanos` (§4/§5's primitive), and the live gas parameter value behind it if testing a governance-mutable opcode (§1). Report as a ratio (time-per-gas-unit) per opcode/tx, not a single aggregate. +For each opcode/tx under test: `gas` (free — already on the receipt, no extra instrumentation), `executionNanos` (§4/§5's primitive), and the live gas parameter value behind it if testing a governance-mutable opcode (§1). Report as a ratio (time-per-gas-unit) per opcode/tx, not a single aggregate. **Before concluding anything, re-check against §1:** a divergence on `SSTORE` confirms a known governance-set gap, not a finding; the real signal is `KECCAK256`/arithmetic. Precompile numbers belong in their own table, never blended with interpreted-opcode results. @@ -186,26 +197,29 @@ For each opcode/tx under test: `gas` (Tier 0, free), `executionNanos` (§4/§5's ## 8. eBPF — only if §4/§7's primitive genuinely isn't enough -**Don't reach for this by default — it is no longer the primary path for either functionality as specified.** `debug_traceTransactionProfile` already gives clean per-tx gas + execution time without any kernel-level instrumentation, validated this session (r=0.9914 gas-vs-`executionNanos` correlation on real replayed pacific-1 transactions). The only reason to reach for eBPF is a need genuinely beyond what §4-§7 answer — e.g. true per-opcode dispatch-level timing *within* a single tx's execution (the profiling RPC gives per-opcode *gas* via `structLogs` and per-*tx* timing, not per-opcode timing), or attributing time to something below the Go-function granularity the profiler's `phases`/`store.modules` breakdown already covers. +**Don't reach for this by default — it is no longer the primary path for either functionality as specified.** `debug_traceTransactionProfile` already gives clean per-tx gas + execution time without any kernel-level instrumentation, validated this session on a 9-tx sample (r=0.9914 gas-vs-`executionNanos` correlation on real replayed pacific-1 transactions; see §4 step 8 — this sizes the tool as sound, it is not a general finding about Sei's gas schedule). The only reason to reach for eBPF is a need genuinely beyond what §4-§7 answer — e.g. true per-opcode dispatch-level timing *within* a single tx's execution (the profiling RPC gives per-opcode *gas* via `structLogs` and per-*tx* timing, not per-opcode timing), or attributing time to something below the Go-function granularity the profiler's `phases`/`store.modules` breakdown already covers. If you do need it, the privileged-attach gate and symbol-resolution guidance below still apply. -**The gate:** any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster**. This is root-on-node. `eng-` namespaces carry no Pod Security Standard label and harbor has no cluster-wide PSS default — the effective PSS level is Kubernetes' own default, `privileged`, enforced nowhere. **This is an unhardened gap, not a sanctioned path.** +**The gate:** any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster**. This is root-on-node — and root-on-node on EKS means more than co-tenant pod access: it reaches the node's IAM role via IMDS (`169.254.169.254`), kubelet credentials, and any system DaemonSet's tokens/secrets scheduled on that node. Pinning to a dedicated node (§3) removes co-tenant *engineer* pods, not the node's AWS-credential surface. `eng-` namespaces carry no Pod Security Standard label and harbor has no cluster-wide PSS default — the effective PSS level is Kubernetes' own default, `privileged`, enforced nowhere. **This is an unhardened gap, not a sanctioned path.** **Do this, in order, before attaching anything:** -1. Get sign-off first — surface what you're attaching, to what, on which pod, for how long, to the platform/security owner. -2. Pin to a dedicated node (§3's `sei.io/dedicated-node` annotation) for the duration. -3. Scope the capability grant to the minimum: `kubectl debug --target=seid -it -0 --image= --profile=sysadmin` — `--target=` joins only that pod's PID namespace. Prefer `CAP_BPF`+`CAP_PERFMON` over the broader `sysadmin` profile where possible. -4. If this becomes recurring, that's the signal to open a platform PR for a dedicated profiling namespace with its own restricted-by-default PSS + a narrow, auditable exception — a platform decision, not something to self-approve. +1. **Get sign-off first, and require an affirmative response — silence is not consent.** Surface what you're attaching, to what, on which pod, for how long, and to whom (the harbor platform/security owner — confirm the current contact/channel with the team; don't proceed on an assumed default). Wait for an explicit "yes" from a human. An agent driving this runbook must not treat "I posted and got no objection" as approval. +2. Pin to a dedicated node (§3's `sei.io/dedicated-node` annotation) for the duration — this narrows co-tenant blast radius, not the node-credential exposure named above. +3. Scope the capability grant to the minimum — prefer `CAP_BPF`+`CAP_PERFMON` over the broader `sysadmin` profile wherever the tool supports it: `kubectl debug --target=seid -it -0 --image= --profile=` (fall back to `--profile=sysadmin` only if the narrower profile doesn't work for the specific probe you need, and say so explicitly when you do). `--target=` joins only that pod's PID namespace regardless of which profile you use. +4. **Set a hard time bound and tear down explicitly.** State the max duration in the sign-off ask (step 1), and when you're done, delete the debug container/pod and confirm it's gone — don't leave a root-on-node ephemeral container running on shared infrastructure with no revocation step. +5. If this becomes recurring, that's the signal to open a platform PR for a dedicated profiling namespace with its own restricted-by-default PSS + a narrow, auditable exception — a platform decision, not something to self-approve. Resolve real symbols before writing any probe — `sei-chain`'s `go.mod` pulls go-ethereum via a `replace` directive, which preserves the original import path in compiled symbol names: ```sh -kubectl -n eng- debug --target=seid -it -0 --image= --profile=sysadmin -- \ +kubectl -n eng- debug --target=seid -it -0 --image= --profile= -- \ go tool nm /proc//root/usr/bin/seid | grep -E 'StateTransition\)\.Execute|EVMInterpreter\)\.Run' ``` +(symbol resolution via `go tool nm` needs no elevated capability beyond joining the pod's PID namespace — `--profile=sysadmin` should not be needed here at all; reserve the broader profile for the actual probe attach in the step above, and only if the minimal profile doesn't cover it.) + - **Per-tx bracket:** `github.com/ethereum/go-ethereum/core.(*StateTransition).Execute` — Sei's `x/evm/keeper/msg_server.go` (`applyEVMMessage`) calls this directly, bypassing the upstream `ApplyMessage` wrapper. - **Per-opcode dispatch (not individually probeable):** `github.com/ethereum/go-ethereum/core/vm.(*EVMInterpreter).Run` — indirect function-table dispatch, no stable per-opcode symbol. This is why §6 constructs isolation via bytecode rather than probing the loop. @@ -220,8 +234,8 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta | Symptom | Cause | Fix | |---|---|---| | Replayer hits `wrong Block.Header.AppHash: expected X, got Y` and evicts every peer it asks | Binary version doesn't match what actually produced the history being replayed (§2) | Pin to the release tag matching the live chain (§2a), or deliberately use `mock_chain_validation` (§2b) if testing unreleased code and you accept compounding divergence from real history. | -| `debug_traceTransactionProfile`/`debug_traceCall` returns `"block number ... is beyond max lookback of 10000"` | Collected tx hashes, then profiled them later; the replayer's rapid catch-up aged the target height out of `MaxTraceLookbackBlocks` (§4 step 6) | Profile immediately after finding each tx, or wait for the node to fully catch up to live tip before a bulk pass. | -| Correlating gas against whole-block `latency_ms` gives a weak/inconsistent result | Whole-block wall time is diluted by non-EVM per-block work and (on a replayer) blocksync contention — verified this session (r=0.301) (§1) | Use `debug_traceTransactionProfile`'s `executionNanos`, not a log-derived block-level proxy (§4) — verified r=0.9914 on the same chain. | +| `debug_traceTransactionProfile`/`debug_traceCall` returns `"block number ... is beyond max lookback of 10000"` | Collected tx hashes, then profiled them later; the replayer's rapid catch-up aged the target height out of `MaxTraceLookbackBlocks` (§4 step 7) | Profile immediately after finding each tx, or wait for the node to fully catch up to live tip before a bulk pass. | +| Correlating gas against whole-block `latency_ms` gives a weak/inconsistent result | Whole-block wall time is diluted by non-EVM per-block work and (on a replayer) blocksync contention — verified this session (r=0.301) (§1) | Use `debug_traceTransactionProfile`'s `executionNanos`, not a log-derived block-level proxy (§4) — verified r=0.9914 on a 9-tx same-session sample, not a general constant. | | `kubectl describe node` shows an instance type outside `r6i`/`r7i` | `controller-config.yaml` or NodePool requirements drifted between harbor and prod | Halt — the timing result won't reflect prod hardware. Diff `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml` and `clusters/{harbor,prod}/default/nodepool-*.yaml` before proceeding. | | Path A (`debug_traceCall`) gives gas but no execution time | `TraceCall` was never wired to the `phaseDurations` profiling mechanism — a real, unclosed gap, not a config issue (§5) | Use Path B: submit a real signed tx against your own restored-to-`` node, then `debug_traceTransactionProfile` its mined hash. | | Trying to replay block `X` against a `Y` that isn't `X-1` | No existing tooling supports the general case (§4) | This is new engineering, not a config change — confirm whether `Y=X-1` actually answers the real question before treating it as a blocker. | From cfd5b7f3213c2850b161391a13f586116b30406e Mon Sep 17 00:00:00 2001 From: bdchatham Date: Thu, 9 Jul 2026 21:21:53 -0700 Subject: [PATCH 3/6] docs(runbooks): fold in StructLogger-contamination correction + open gaps Empirical follow-up this session found the default debug_traceTransactionProfile invocation attaches a full StructLogger inside the executionNanos timing bracket, inflating the gas-vs-time correlation (r=0.9914 -> r=0.8093 on a diverse sample once switched to callTracer). Also folds in a peer collaboration with systems-engineer that ruled out eBPF for the begin/end-blocker gap and per-opcode timing, endorsed perf-stat cycle counters as the one worthwhile (non-uprobe) kernel-level cross-validation, and flagged Y=X-1 as blocking the state-opcode cost-vs-state-size question specifically -- the biggest open gap, not closed by any instrumentation choice here. Adds min-of-N/forced-GC timing discipline, separates interpreter-compute from state-I/O in reporting, documents a real sampling mistake (near-zero-variance gas sample gave a meaningless r=0.11), and notes that release tags and backport branches can diverge on whether the profiling endpoint exists -- check the exact image, not the release family. --- .../benchmarking-evm-gas-vs-execution-time.md | 45 +++++++++++++------ 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md index 83900c7..fc153ec 100644 --- a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -37,7 +37,7 @@ The question: for a given opcode/tx, does the gas charged scale with the real wa - **Per-tx fixed overhead** (signature verification, the Cosmos ante handler, the EVM↔Cosmos StateDB bridge) is real wall-clock time attributable to *no* opcode — it dilutes a measurement unless your sample size / loop count makes it a small fraction of the total. - **OCC (optimistic concurrent execution) does not break per-tx correlation for §4's RPC primitive.** Sei applies a block's transactions through a parallel scheduler (`app/abci.go`, `sei-cosmos/tasks/scheduler.go`), not a serial loop. A raw uprobe/uretprobe bracket around `StateTransition.Execute` (§8) *would* need OCC disabled to get clean per-tx numbers, since multiple `Execute` calls can run concurrently on different threads. `debug_traceTransactionProfile` sidesteps this entirely: it isolates a single tx's execution by replaying the block *up to* that tx's index and then executing just it (`evmrpc/simulate.go`'s `replayTransactionTillIndex`), independent of how the block was originally scheduled. You do **not** need to disable OCC on the replayer to get clean per-tx numbers via this RPC. -- **Whole-block wall-clock time (log-line `latency_ms` on `"finalized block"`) is the wrong signal — verified empirically, not theoretically.** It's diluted by non-EVM per-block work (oracle vote tally, IBC, begin/end-blockers for every module) and, on a replayer still catching up, contention with block-fetching. A same-session sample: correlating gas against whole-block latency gave Pearson r=0.301; filtering to EVM-active blocks and using the OCC-scheduler's tx-execution-phase latency improved it to r=0.618, still with real outliers. Correlating gas against `debug_traceTransactionProfile`'s `executionNanos` on the same chain gave **r=0.9914** on a 9-tx sample. Use the RPC, not a log-derived proxy. +- **Whole-block wall-clock time (log-line `latency_ms` on `"finalized block"`) is the wrong signal — verified empirically, not theoretically.** It's diluted by non-EVM per-block work (oracle vote tally, IBC, begin/end-blockers for every module) and, on a replayer still catching up, contention with block-fetching. A same-session sample: correlating gas against whole-block latency gave Pearson r=0.301; filtering to EVM-active blocks and using the OCC-scheduler's tx-execution-phase latency improved it to r=0.618, still with real outliers. Correlating gas against `debug_traceTransactionProfile`'s `executionNanos` on the same chain gave r=0.9914 on an initial 9-tx sample — **but that invocation used the default StructLogger tracer, which adds opcode-scaled overhead into the timed window; switching to the lighter `callTracer` on a properly diverse sample gave r=0.8093** (§4 step 9 has the full story, including a sampling mistake worth learning from). Use the RPC with `callTracer`, not a log-derived proxy and not the default invocation. --- @@ -59,6 +59,8 @@ Use the exact tag the live chain runs (e.g. `ghcr.io/sei-protocol/sei:v6.5.2`). Verified this session: switching from an arbitrary dev-branch commit image to the release tag matching the live chain's actual version took the replayer from a permanent app-hash-mismatch eviction loop at the very first post-snapshot block to zero errors across tens of thousands of blocks. +**"The release tag" and "the exact commit someone else already validated" are not always the same thing — check, don't assume by release family.** Verified this session: a `release/v6.6` backport commit and the `v6.5.2` tag had genuinely diverged (380 commits ahead, 63 behind each other) despite both being "the v6.5-ish safe version" in casual conversation. Whether `debug_traceTransactionProfile` (§4) exists on a given image is commit-specific, not release-family-specific — no released *tag* contains it as of this writing, but a backport/hotfix branch someone else is already running might already have it (confirmed: one such commit had it, more completely than the version on `origin/main`, with an added lookback guard the original commit lacked). Before assuming you need `mock_chain_validation` (§2b) or a custom cherry-pick to get the profiling endpoint, check the *exact* image already in use for the exact height range you care about — `debug_traceTransactionProfile` against it directly is the fastest way to find out, faster than reasoning about it from tags. + ### 2b. `mock_chain_validation-` — for testing bleeding-edge/proposed code against real transaction load **Only ever the image of your own throwaway replayer in `eng-` — never merge this tag onto a shared, long-lived, or archive `SeiNode`.** Image selection for any `SeiNode` flows through the `/harbor-dev` skill's PR-based render into `harbor-engineering-workspace`, a repo shared across engineers; nothing stops you from pointing this tag at the wrong manifest, and a shared node running it would silently accept diverged state while still looking healthy. Confirm the target is your own disposable node before merging. @@ -123,19 +125,23 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive 3. **Let it replay forward past ``.** There is no documented flag to halt exactly at a target height. Poll `.status`/RPC height or tail logs to confirm `` has been applied; letting the node continue replaying past it is harmless. 4. **A replayer's EVM-RPC is reachable but not advertised — port-forward directly, don't look for it in `.status`.** The controller treats `replayer` mode as an RPC-less restore workload (`internal/controller/node/endpoints.go`'s `servesEVM()` returns `false` for it), so `.status.endpoint` is absent for a replayer regardless of whether the underlying `seid` process actually serves EVM traffic. It does — this session's measurements ran against it — but you must reach it via `kubectl port-forward -0 8545:8545`, not by reading it off the SeiNode's status. If a future controller change acts on the "RPC-less" intent by actually disabling EVM on replayers, this whole runbook's measurement path breaks; verify 8545 responds (`eth_blockNumber`) before building anything on top of it. 5. **Get the tx hash(es) for ``:** `eth_getBlockByNumber(, true)` on the replayer's own EVM-RPC (port 8545) returns full transaction objects including `hash` and the block-level `gasUsed`. -6. **For each tx, call the one RPC that gives you both numbers:** +6. **For each tx, call the RPC — but not with the default invocation.** No-config `debug_traceTransactionProfile` attaches a full opcode-level `StructLogger` (stack + storage capture per step) *inside* the `executionNanos` timing bracket (confirmed by reading `evmrpc/block_trace_profiled.go`'s `config==nil` path and the bracket around `core.ApplyTransactionWithEVM`). That overhead scales with opcode count — i.e. with gas — which artificially inflates any gas-vs-time correlation you measure. **Pass `{"tracer":"callTracer"}`** instead: it hooks call frames, not every opcode, and is far lighter (confirmed empirically this session: the same tx measured 65.1ms with the default StructLogger vs. 34.7ms with `callTracer` — nearly 2x contamination). It's still not zero-overhead — a genuinely tracer-free timing path would need a small sei-chain change (see §5) — but it's the best available today. ```sh curl -s -X POST -H "Content-Type: application/json" \ - --data '{"jsonrpc":"2.0","method":"debug_traceTransactionProfile","params":[""],"id":1}' \ + --data '{"jsonrpc":"2.0","method":"debug_traceTransactionProfile","params":["", {"tracer":"callTracer"}],"id":1}' \ localhost:8545 ``` - The response's `result.trace.gas` is the real gas used. `result.profile.phases.executionNanos` is the isolated, clean per-tx EVM execution time — **use this field, not `result.profile.totalNanos`**, which also includes `replayHistoricalTxsNanos` (replaying the block up to this tx's index — real cost, but not *this tx's* cost) and `traceResultNanos` (tracing-instrumentation overhead itself). `result.profile.store.modules` additionally breaks down per-Cosmos-module store-access counts and timing if you need to attribute cost below the whole-tx level. + The response's `result.trace.gas` (or `result.trace.gasUsed` depending on tracer) is the real gas used. `result.profile.phases.executionNanos` is the per-tx EVM execution time — **use this field, not `result.profile.totalNanos`**, which also includes `replayHistoricalTxsNanos` (replaying the block up to this tx's index — real cost, but not *this tx's* cost) and `traceResultNanos` (tracing-instrumentation overhead itself). **`executionNanos` itself still includes state-backend I/O time** — `historicalDbLookupNanos`/`profile.store` is subtracted from `totalNanos` to produce `otherNanos`, but *not* from `executionNanos` (confirmed by reading `trace_profile.go`). For a SLOAD/SSTORE-heavy tx this means cache-warmth (memIAVL/flatKV page-cache state, itself run-to-run variable) is baked into what you're calling "execution time." Report two numbers, not one: `executionNanos − storeNanos` (interpreter-local compute) and `storeNanos` (state-backend cost, from `result.profile.store`) — they're priced by different, separable parts of the gas schedule, and §1's table already treats them as separate targets. + +7. **Take min-of-N, not a single sample, per tx.** Wall-clock timing (`time.Since`) absorbs GC pauses, goroutine preemption, and OS thread migration — noise that only ever *adds* time, never subtracts it. Across several repeated profile calls on the same tx, the **minimum** is the closest estimate of true compute cost, and the spread across repeats tells you the noise magnitude directly. Force a GC before each run if your tooling allows it (or note that you didn't). This costs nothing and needs no elevated privilege — do it before reaching for anything in §8. Run on a dedicated, quiesced node (§3) with no concurrent RPC traffic during timing; that removes more noise than any instrumentation would merely reveal. + +8. **Do this promptly.** `debug_trace*` is capped to `MaxTraceLookbackBlocks` behind the node's current tip (default 10,000, `evmrpc/config/config.go`, configurable via `evm.max_trace_lookback_blocks`). A replayer still catching up can process 20-30 blocks/sec — 10,000 blocks ages out in under 10 minutes. If you collect a batch of tx hashes and then profile them later, some will fail with `"block number ... is beyond max lookback of 10000"`. Verified this session: exactly this happened mid-collection. Either profile each tx immediately after finding it, or wait until the node has fully caught up to live tip (stable height) before doing a bulk collection pass. -7. **Do this promptly.** `debug_trace*` is capped to `MaxTraceLookbackBlocks` behind the node's current tip (default 10,000, `evmrpc/config/config.go`, configurable via `evm.max_trace_lookback_blocks`). A replayer still catching up can process 20-30 blocks/sec — 10,000 blocks ages out in under 10 minutes. If you collect a batch of tx hashes and then profile them later, some will fail with `"block number ... is beyond max lookback of 10000"`. Verified this session: exactly this happened mid-collection. Either profile each tx immediately after finding it, or wait until the node has fully caught up to live tip (stable height) before doing a bulk collection pass. +9. **Sample size, and a real mistake to avoid.** A single tx pair doesn't tell you much, and **neither does a sample with no variance in gas** — collecting 25 txs from a narrow, bot-dominated block window this session produced near-identical gas (458,696 for 24 of 25 txs) and a meaningless r=0.11, not because the correlation is weak but because a nearly-constant independent variable can't test it at all. Deliberately collect a real range of gas magnitudes/contracts before computing anything. **The following numbers are this session's illustrations of the primitive working, not a general finding about Sei's gas schedule — treat them as sizing the tool, not the research conclusion, and re-derive your own:** on a properly diverse sample (gas ranging 0–1.3M), default-StructLogger gave Pearson r=0.9914 between gas and `executionNanos`; switching to `callTracer` on an equally diverse sample gave **r=0.8093** — still a strong, real correlation, but honestly lower once tracer contamination is reduced. A same-gas cluster (~130k-133k, n=8) still showed a genuine ~2x spread in measured `callTracer` time (29ms-59ms) — likely cache-warmth or scheduling noise per step 7, not a fluke. Always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). -8. **Sample size and interpretation.** A single tx pair doesn't tell you much. **The following number is a same-session illustration of the primitive working, not a general finding about Sei's gas schedule — treat it as sizing the tool, not the research conclusion:** a 9-tx sample (gas ranging ~31k to ~1.3M) gave Pearson r=0.9914 between gas and `executionNanos`, against r=0.301 for the whole-block-latency proxy on the same chain. Neither number is independently reproducible from source — they're this session's measurements, not constants; re-derive your own before citing either. Collect enough of a real range (different contracts, different gas magnitudes) before drawing a conclusion, and always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). +10. **The bigger limitation no measurement technique here closes: `Y=X-1` blocks the question with the most repricing leverage for state opcodes.** For `SLOAD`/`SSTORE`, the decision-relevant variable is how access cost scales with state size / tree depth as the chain grows — which requires holding bytecode fixed while sweeping the base height, i.e. the general `Y` for cycles/instructions on the differential-loop's bytecode runs is a hardware counter read, not a Go-symbol attach — none of the stack-copy/uretprobe/thread-migration hazards below apply to it. CPU cycles are immune to GC and scheduling jitter in a way wall-clock `time.Since` isn't, so `cycles(A) − cycles(B)) ÷ iterations` is a genuinely independent cross-check on the differential-loop's nanos-per-opcode numbers. Treat this as a **deferred, second-iteration cross-validation** — worth running only if you still see unexplained variance after the min-of-N discipline, or if a repricing recommendation needs to rest on absolute per-opcode precision rather than relative ordering. It still needs `CAP_PERFMON` and the same human sign-off as anything else in this section. -If you do need it, the privileged-attach gate and symbol-resolution guidance below still apply. +If you do need actual uprobes/`offcputime`/`profile`, the privileged-attach gate and symbol-resolution guidance below still apply. **The gate:** any eBPF uprobe, `offcputime`, or CPU `profile` run against `seid` requires a **privileged kernel capability grant** (`CAP_BPF`/`CAP_SYS_ADMIN`/`CAP_PERFMON`) on a pod in **harbor — a shared EKS cluster**. This is root-on-node — and root-on-node on EKS means more than co-tenant pod access: it reaches the node's IAM role via IMDS (`169.254.169.254`), kubelet credentials, and any system DaemonSet's tokens/secrets scheduled on that node. Pinning to a dedicated node (§3) removes co-tenant *engineer* pods, not the node's AWS-credential surface. `eng-` namespaces carry no Pod Security Standard label and harbor has no cluster-wide PSS default — the effective PSS level is Kubernetes' own default, `privileged`, enforced nowhere. **This is an unhardened gap, not a sanctioned path.** @@ -234,8 +246,10 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta | Symptom | Cause | Fix | |---|---|---| | Replayer hits `wrong Block.Header.AppHash: expected X, got Y` and evicts every peer it asks | Binary version doesn't match what actually produced the history being replayed (§2) | Pin to the release tag matching the live chain (§2a), or deliberately use `mock_chain_validation` (§2b) if testing unreleased code and you accept compounding divergence from real history. | -| `debug_traceTransactionProfile`/`debug_traceCall` returns `"block number ... is beyond max lookback of 10000"` | Collected tx hashes, then profiled them later; the replayer's rapid catch-up aged the target height out of `MaxTraceLookbackBlocks` (§4 step 7) | Profile immediately after finding each tx, or wait for the node to fully catch up to live tip before a bulk pass. | -| Correlating gas against whole-block `latency_ms` gives a weak/inconsistent result | Whole-block wall time is diluted by non-EVM per-block work and (on a replayer) blocksync contention — verified this session (r=0.301) (§1) | Use `debug_traceTransactionProfile`'s `executionNanos`, not a log-derived block-level proxy (§4) — verified r=0.9914 on a 9-tx same-session sample, not a general constant. | +| `debug_traceTransactionProfile`/`debug_traceCall` returns `"block number ... is beyond max lookback of 10000"` | Collected tx hashes, then profiled them later; the replayer's rapid catch-up aged the target height out of `MaxTraceLookbackBlocks` (§4 step 8) | Profile immediately after finding each tx, or wait for the node to fully catch up to live tip before a bulk pass. | +| Correlating gas against whole-block `latency_ms` gives a weak/inconsistent result | Whole-block wall time is diluted by non-EVM per-block work and (on a replayer) blocksync contention — verified this session (r=0.301) (§1) | Use `debug_traceTransactionProfile`'s `executionNanos` with `{"tracer":"callTracer"}`, not a log-derived block-level proxy and not the default StructLogger invocation (§4 step 6) — verified r=0.8093 on a diverse same-session sample, not a general constant. | +| Gas-vs-time correlation looks suspiciously perfect (near 0.99) | Default `debug_traceTransactionProfile` invocation attaches a full StructLogger inside the timed window; overhead scales with opcode count/gas (§4 step 6) | Re-run with `{"tracer":"callTracer"}` and compare — verified this session: ~2x overhead on one tx (65.1ms default vs. 34.7ms callTracer). | +| Gas-vs-time correlation looks near zero even though the tool is sound | Sample had little/no variance in gas (e.g. repeated bot/arbitrage calls in a narrow block window) — correlation is meaningless without variance in the independent variable, not evidence the relationship is weak (§4 step 9) | Widen the collection window or deliberately sample across gas magnitudes before computing a correlation. | | `kubectl describe node` shows an instance type outside `r6i`/`r7i` | `controller-config.yaml` or NodePool requirements drifted between harbor and prod | Halt — the timing result won't reflect prod hardware. Diff `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml` and `clusters/{harbor,prod}/default/nodepool-*.yaml` before proceeding. | | Path A (`debug_traceCall`) gives gas but no execution time | `TraceCall` was never wired to the `phaseDurations` profiling mechanism — a real, unclosed gap, not a config issue (§5) | Use Path B: submit a real signed tx against your own restored-to-`` node, then `debug_traceTransactionProfile` its mined hash. | | Trying to replay block `X` against a `Y` that isn't `X-1` | No existing tooling supports the general case (§4) | This is new engineering, not a config change — confirm whether `Y=X-1` actually answers the real question before treating it as a blocker. | @@ -250,13 +264,16 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta - [ ] Confirmed the target node landed on `r6i`/`r7i` via `sei-node`/`sei-archive` (§3) — don't trust the default without checking. - [ ] **Chosen §2a (release-pinned) or §2b (`mock_chain_validation`) deliberately**, not defaulted to an arbitrary image tag. If §2b, confirmed `sei_unsafe_validation_skipped_total` is being watched, not ignored. - [ ] **Functionality 1:** replayer restoring to `` and replaying forward (§4); confirmed `Y=X-1` actually answers the intended question, or explicitly flagged that it doesn't. -- [ ] **Functionality 1/2 measurement:** using `debug_traceTransactionProfile`'s `result.profile.phases.executionNanos` (not `totalNanos`, not a log-derived latency proxy) for timing, `result.trace.gas` for gas. +- [ ] **Functionality 1/2 measurement:** calling `debug_traceTransactionProfile` with `{"tracer":"callTracer"}` — **not the default invocation**, which attaches a full StructLogger and inflates `executionNanos` by roughly 2x (§4 step 6). Using `result.profile.phases.executionNanos` (not `totalNanos`, not a log-derived latency proxy) for timing, `result.trace.gas`/`gasUsed` for gas. +- [ ] **Reporting interpreter-compute and state-I/O separately** (`executionNanos − storeNanos` vs. `storeNanos` from `result.profile.store`), not one blended number (§4 step 6). +- [ ] **Timing runs use min-of-N** (forced GC before each, dedicated/quiesced node, §4 step 7), not a single sample per tx. - [ ] **Traced promptly** relative to `MaxTraceLookbackBlocks` — not collecting a large batch of tx hashes to profile long after the fact on a still-catching-up node. - [ ] **Functionality 2:** using Path B (real submitted tx + §4's primitive) unless a genuine read-only-simulation requirement forces Path A — in which case, the execution-time gap in `debug_traceCall` is flagged explicitly, not worked around with a fabricated number. - [ ] Recorded live gas parameter values (e.g. `SeiSstoreSetGasEIP2200`) + image digest before drawing conclusions. -- [ ] Sample size covers a real range of gas magnitudes/contracts, not a single tx. +- [ ] **Sample size covers a real range of gas magnitudes/contracts, and you've checked for variance before computing a correlation** — a narrow, bot-dominated window with near-constant gas will give a meaningless near-zero r regardless of the true relationship (§4 step 9). - [ ] Re-checked results against §1 (`SSTORE`/precompile confounds) before concluding anything about whether gas tracks compute. -- [ ] Confirmed eBPF is actually necessary (§8) before requesting sign-off for a privileged attach — §4-§7 already answer both functionalities as specified without it. +- [ ] **Named explicitly whether the question needs state-opcode cost-vs-state-size scaling** (§4 step 10) — if so, flagged that as an open engineering gap, not something a `Y=X-1` measurement answers. +- [ ] Confirmed eBPF uprobes/`offcputime` are actually necessary (§8) before requesting sign-off for a privileged attach — §4-§7 already answer both functionalities as specified without them; `perf stat` cycle counters (§8) are the one kernel-level tool worth considering, and only as a deferred cross-validation. --- @@ -264,7 +281,7 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta - Sei-EVM gas/precompile specifics: `/evm` skill — `kit-evm-parity-gas.md`, `kit-sei-precompiles.md`; `x/evm/types/params.go` in `sei-chain` for the governance-mutable param set. - Version-pinning / `mock_chain_validation`: `sei-chain` — `sei-tendermint/types/consensus_policy.go`, `consensus_policy_mock_chain_validation.go`, `consensus_policy_default.go`, `policy_metrics.go`; `.github/workflows/ecr.yml`, `.github/workflows/nightly-ecr.yml` for the image-tag scheme. -- Measurement primitive: `sei-chain` — `evmrpc/trace_profile.go` (`debug_traceTransactionProfile`, `profiledTraceTx`, `phaseDurations`), `evmrpc/simulate.go` (`initializeBlock`, `replayTransactionTillIndex`, `StateAtBlock`, plain `Call`), `evmrpc/tracers.go` (`TraceCall`), `evmrpc/block_trace_profiled.go` (`debug_traceBlockByNumber/Hash` — profiling exists but is passed `nil` here, not surfaced), `evmrpc/config/config.go` (`MaxTraceLookbackBlocks`). +- Measurement primitive: `sei-chain` — `evmrpc/trace_profile.go` (`debug_traceTransactionProfile`, `profiledTraceTx`, `phaseDurations`, the `historicalDbLookupNanos`→`otherNanos` subtraction that does *not* touch `executionNanos`), `evmrpc/simulate.go` (`initializeBlock`, `replayTransactionTillIndex`, `StateAtBlock`, plain `Call`), `evmrpc/tracers.go` (`TraceCall`), `evmrpc/block_trace_profiled.go` (`debug_traceBlockByNumber/Hash` — profiling exists but is passed `nil` here, not surfaced; also the `config==nil` → default `StructLogger` path and the `executionNanos` bracket around `core.ApplyTransactionWithEVM` — this is why the default invocation over-times relative to `callTracer`), `evmrpc/config/config.go` (`MaxTraceLookbackBlocks`). - Harbor chain spin-up mechanics: `/harbor-dev` skill, `references/ephemeral-chain-flow.md`, `references/seictl-cli.md`, `references/harbor-cluster.md`. - Hardware-parity mechanism: `sei-k8s-controller` — `internal/platform/platform.go` (`NodepoolForMode`), `internal/task/bootstrap_resources.go`, `internal/noderesource/noderesource.go` (also home of the `sei.io/dedicated-node` annotation); `sei-protocol/platform` — `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml`, `clusters/{harbor,prod}/default/nodepool-{default,archive}.yaml`. - Harbor `eng-` Pod Security / admission posture: `sei-protocol/platform` — `clusters/harbor/engineers/base/namespace.yaml`, `clusters/prod/walle/{namespace,podcomposition-policy}.yaml` (the posture to eventually mirror), `clusters/harbor/chaos-mesh/chaos-mesh.yaml`, `clusters/harbor/admission/tenant-hostname-policy.yaml`. From 8c01cb45daf12fdba1a676616bfd098921c1be10 Mon Sep 17 00:00:00 2001 From: bdchatham Date: Thu, 9 Jul 2026 21:34:14 -0700 Subject: [PATCH 4/6] docs(runbooks): resolve peer review findings from kubernetes-specialist + systems-engineer kubernetes-specialist (re-verified against current source): - sei.io/dedicated-node must be set at initial render, not toggled on a running replayer -- buildRunningPlan only rebuilds the pod on image drift or sidecar-reapproval, an annotation-only change triggers neither, and OnDelete means nothing else replaces the pod either. The prior guidance was silently ineffective if applied after the fact. - added the no-SeiNode-condition-on-stuck-Pending caveat (check the pod's own PodScheduled condition/events, not the SeiNode object). - confirmed the "highest snapshot <= targetHeight" claim directly against seictl's own source (resolveKeyForHeight), resolving an apparent conflict with the SeiNode CRD's own (incomplete) doc comment in favor of the runbook's original claim -- added the precise citation and a heads-up not to trust the CRD comment over seictl's actual logic. systems-engineer (second pass, fresh eyes on the prior fix): - caught a new, higher-consequence bug in the just-added executionNanos-minus-storeNanos guidance: the store trace spans the whole statedb lifetime including every replayed preceding tx, not just the target tx, so the subtraction silently goes negative past the first tx or two in a block. Replaced with an honest caveat and flagged the real fix (resetting the tracer at the execution boundary) as a sei-chain change, not a subtraction on today's fields. - caught a category error: a same-gas cluster's 2x timing spread was mischaracterized as noise addressable by min-of-N, which only reduces noise across repeats of one tx, not variance across different txs. Reframed as a real signal to check (repeat each tx individually) before dismissing it. - added statistical caveats to the r=0.8093 headline (single-sample, not min-of-N; Pearson r is a smoke test not a pricing-fidelity measure; r-squared/Spearman/residuals recommended over raw r). - corrected the callTracer overhead model (scales with call-frame/log count, not opcode count -- near-zero for the primary interpreter-local target), warned off noopTracer as a false lighter baseline, added the onlyTopCall differencing technique, and softened an overclaim about perf stat cycles being immune to GC. --- .../benchmarking-evm-gas-vs-execution-time.md | 31 +++++++++++++------ 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md index fc153ec..dfc080f 100644 --- a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -101,7 +101,9 @@ Verified this session (illustrative, not a permanent number — re-check `origin **You do not need to request specific hardware.** `sei-k8s-controller` picks the Karpenter NodePool per node mode automatically: `internal/platform/platform.go`'s `NodepoolForMode()` — archive-mode nodes get `c.NodepoolArchive`, every other mode (including `replayer`) gets `c.NodepoolName` — and wires the matching toleration + required node affinity onto the pod spec (`internal/task/bootstrap_resources.go`, mirrored in `internal/noderesource/noderesource.go`). Harbor's and prod's `sei-k8s-controller/controller-config.yaml` are designed to carry identical `nodepoolName`/`nodepoolArchive`/`tolerationKey` values, and both clusters' NodePool manifests are designed to carry the same `r6i`/`r7i` instance-family, Nitro-hypervisor, on-demand requirements — by design, not independently re-verified here each time. Net: a replayer or archive `SeiNode` in your `eng-` namespace *should* land on the same instance class as its prod counterpart, with zero engineer-side configuration — confirm it with the step below rather than trusting the design intent, since drift between the two configs is exactly the failure mode that step catches. -**Optionally**, for a throwaway single-tenant experiment node, apply the `sei.io/dedicated-node: "true"` annotation (see `sei-k8s-controller`'s dedicated-node feature) to guarantee the node isn't shared with another engineer's workload for the duration of the run — not required for correctness, but removes one source of noisy-neighbor variance from your timing numbers. +**Optionally**, for a throwaway single-tenant experiment node, set the `sei.io/dedicated-node: "true"` annotation (see `sei-k8s-controller`'s dedicated-node feature) — not required for correctness, but removes one source of noisy-neighbor variance from your timing numbers. **Set it in the initial render, not on an already-running replayer.** Verified against the reconcile logic: a Running replayer's plan only rebuilds the pod on image drift or a sidecar-reapproval need (`internal/planner/replay.go`'s `buildRunningPlan`); an annotation-only change matches neither, so nothing replaces the pod, and with the StatefulSet's `OnDelete` strategy nothing else will either. The SeiNode object updates but the pod keeps running on its original, possibly-shared node indefinitely — a false sense of isolation that directly undermines step 7's noise-reduction discipline below. If you need to add it after the fact, delete the pod yourself to force a rebuild (§2's image-swap steps already establish this pattern). + +**If a dedicated pod never schedules, nothing on the SeiNode object tells you.** This is documented behavior, not an oversight to work around: "a pod that can never be scheduled (no available single-tenant node) stays Pending with no SeiNode condition reflecting it" (`internal/noderesource/noderesource.go`). Check the pod's own `PodScheduled` condition and events directly if `.spec.nodeName` (the verify step below) comes back empty. **Verify hardware parity anyway before trusting a timing result:** @@ -121,11 +123,11 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive ### Steps 1. **Choose and pin the image deliberately (§2)** before rendering anything. -2. **Render a `replayer`-mode `SeiNode`** targeting ``: `spec.replayer.snapshot.s3.targetHeight: ` **and `spec.peers`** — the CRD's admission rule rejects a replayer with no `peers` set (`api/v1alpha1/seinode_types.go`'s CEL rule: "peers is required when replayer mode is set"), so the render is incomplete without it. Point `peers` at a real historical block source (a shared full-history archive's p2p address, or a `label`/`selector` peer within your namespace) — the seictl sidecar restores the highest snapshot `<= ` and syncs forward from there. Render via the `/harbor-dev` skill's PR-based flow into `harbor-engineering-workspace`, same as any other `SeiNode`. +2. **Render a `replayer`-mode `SeiNode`** targeting ``: `spec.replayer.snapshot.s3.targetHeight: ` **and `spec.peers`** — the CRD's admission rule rejects a replayer with no `peers` set (`api/v1alpha1/seinode_types.go`'s CEL rule: "peers is required when replayer mode is set"), so the render is incomplete without it. Point `peers` at a real historical block source (a shared full-history archive's p2p address, or a `label`/`selector` peer within your namespace) — the seictl sidecar restores the highest snapshot `<= ` and syncs forward from there (verified directly against seictl's own source, not just the sei-k8s-controller CRD: `sidecar/tasks/snapshot_restore.go` — "TargetHeight, when set, selects the highest available snapshot <= that height. When zero, the latest snapshot (from latest.txt) is used," enforced in `resolveKeyForHeight`'s height-capped search). **The `SeiNode` CRD's own doc comment on `TargetHeight` (`api/v1alpha1/common_types.go`) reads differently and only describes the zero-value/"restore latest, then sync forward" case — don't trust it over seictl's actual selection logic if the two seem to disagree.** Render via the `/harbor-dev` skill's PR-based flow into `harbor-engineering-workspace`, same as any other `SeiNode`. 3. **Let it replay forward past ``.** There is no documented flag to halt exactly at a target height. Poll `.status`/RPC height or tail logs to confirm `` has been applied; letting the node continue replaying past it is harmless. 4. **A replayer's EVM-RPC is reachable but not advertised — port-forward directly, don't look for it in `.status`.** The controller treats `replayer` mode as an RPC-less restore workload (`internal/controller/node/endpoints.go`'s `servesEVM()` returns `false` for it), so `.status.endpoint` is absent for a replayer regardless of whether the underlying `seid` process actually serves EVM traffic. It does — this session's measurements ran against it — but you must reach it via `kubectl port-forward -0 8545:8545`, not by reading it off the SeiNode's status. If a future controller change acts on the "RPC-less" intent by actually disabling EVM on replayers, this whole runbook's measurement path breaks; verify 8545 responds (`eth_blockNumber`) before building anything on top of it. 5. **Get the tx hash(es) for ``:** `eth_getBlockByNumber(, true)` on the replayer's own EVM-RPC (port 8545) returns full transaction objects including `hash` and the block-level `gasUsed`. -6. **For each tx, call the RPC — but not with the default invocation.** No-config `debug_traceTransactionProfile` attaches a full opcode-level `StructLogger` (stack + storage capture per step) *inside* the `executionNanos` timing bracket (confirmed by reading `evmrpc/block_trace_profiled.go`'s `config==nil` path and the bracket around `core.ApplyTransactionWithEVM`). That overhead scales with opcode count — i.e. with gas — which artificially inflates any gas-vs-time correlation you measure. **Pass `{"tracer":"callTracer"}`** instead: it hooks call frames, not every opcode, and is far lighter (confirmed empirically this session: the same tx measured 65.1ms with the default StructLogger vs. 34.7ms with `callTracer` — nearly 2x contamination). It's still not zero-overhead — a genuinely tracer-free timing path would need a small sei-chain change (see §5) — but it's the best available today. +6. **For each tx, call the RPC — but not with the default invocation.** No-config `debug_traceTransactionProfile` attaches a full opcode-level `StructLogger` (stack + storage capture per step) *inside* the `executionNanos` timing bracket (confirmed by reading `evmrpc/block_trace_profiled.go`'s `config==nil` path and the bracket around `core.ApplyTransactionWithEVM`). That overhead scales with opcode count — i.e. with gas — which artificially inflates any gas-vs-time correlation you measure. **Pass `{"tracer":"callTracer"}`** instead — confirmed empirically this session: the same tx measured 65.1ms with the default StructLogger vs. 34.7ms with `callTracer`, nearly 2x contamination. `callTracer` has no per-opcode hook at all (`OnTxStart`/`OnTxEnd`/`OnEnter`/`OnExit`/`OnLog` only, no `OnOpcode`) — its overhead scales with **call-frame and log count**, not opcode count, so for §1's primary target (interpreter-local arithmetic, `KECCAK256` — no nested calls, no logs) its residual is genuinely close to zero. Contamination concentrates on call-heavy, log-heavy, or state-heavy txs — the secondary/precompile categories §1 already treats separately. **Do not reach for `noopTracer` as an even-lighter baseline — it's actually heavier per-opcode than `callTracer`** (it does register `OnOpcode`); diffing against it would overstate `callTracer`'s contamination, not bound it. If you want to bound `callTracer`'s own nested-call-frame overhead specifically, diff it against `{"tracer":"callTracer","tracerConfig":{"onlyTopCall":true}}` (short-circuits nested `OnEnter`/`OnExit`) — same differencing technique that exposed the StructLogger gap, one level finer, no source change needed. The true zero-overhead floor (no tracer hooks at all, `executionNanos` still populated) needs a small sei-chain change — see §5 — but `callTracer` is the best available today without one. ```sh curl -s -X POST -H "Content-Type: application/json" \ @@ -133,13 +135,23 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive localhost:8545 ``` - The response's `result.trace.gas` (or `result.trace.gasUsed` depending on tracer) is the real gas used. `result.profile.phases.executionNanos` is the per-tx EVM execution time — **use this field, not `result.profile.totalNanos`**, which also includes `replayHistoricalTxsNanos` (replaying the block up to this tx's index — real cost, but not *this tx's* cost) and `traceResultNanos` (tracing-instrumentation overhead itself). **`executionNanos` itself still includes state-backend I/O time** — `historicalDbLookupNanos`/`profile.store` is subtracted from `totalNanos` to produce `otherNanos`, but *not* from `executionNanos` (confirmed by reading `trace_profile.go`). For a SLOAD/SSTORE-heavy tx this means cache-warmth (memIAVL/flatKV page-cache state, itself run-to-run variable) is baked into what you're calling "execution time." Report two numbers, not one: `executionNanos − storeNanos` (interpreter-local compute) and `storeNanos` (state-backend cost, from `result.profile.store`) — they're priced by different, separable parts of the gas schedule, and §1's table already treats them as separate targets. + The response's `result.trace.gas` (or `result.trace.gasUsed` depending on tracer) is the real gas used. `result.profile.phases.executionNanos` is the per-tx EVM execution time — **use this field, not `result.profile.totalNanos`**, which also includes `replayHistoricalTxsNanos` (replaying the block up to this tx's index — real cost, but not *this tx's* cost) and `traceResultNanos` (tracing-instrumentation overhead itself). **`executionNanos` itself still includes state-backend I/O time**, and for a SLOAD/SSTORE-heavy tx this means cache-warmth (memIAVL/flatKV page-cache state, itself run-to-run variable) is baked into what you're calling "execution time." + + **Do not subtract `result.profile.store`'s totals from `executionNanos` to isolate interpreter-only compute — that arithmetic is broken, caught in this session's own peer review before it shipped.** `executionNanos` brackets only the target tx (the window around `core.ApplyTransactionWithEVM`), but the store trace dump accumulates over the *entire* statedb lifetime — including every preceding tx's execution during `replayHistoricalTxsNanos`'s `ReplayTransactionTillIndex` pass, on the same tracer instance. For a tx at index k in its block, the store total includes k other transactions' state I/O, not just this one's; subtracting it from a single tx's `executionNanos` will silently go negative for anything past the first tx or two in a block. It's also incomplete even ignoring the windowing bug: the read-side lookup total (`historicalDbLookupNanos`) doesn't include `Set`/`Delete` (write) latency, which lives in a separate stats bucket — so it wouldn't capture SSTORE cost even if correctly scoped. Cleanly separating interpreter compute from state I/O per tx needs a sei-chain change to reset or snapshot the store tracer at the execution boundary — flag that as follow-on engineering rather than approximating it with this subtraction. Until then, report `executionNanos` as one number (compute + this tx's state I/O, not cleanly separable today) and treat `result.profile.store`'s counts as informative about which modules/keys were touched, not as a subtractable timing quantity. 7. **Take min-of-N, not a single sample, per tx.** Wall-clock timing (`time.Since`) absorbs GC pauses, goroutine preemption, and OS thread migration — noise that only ever *adds* time, never subtracts it. Across several repeated profile calls on the same tx, the **minimum** is the closest estimate of true compute cost, and the spread across repeats tells you the noise magnitude directly. Force a GC before each run if your tooling allows it (or note that you didn't). This costs nothing and needs no elevated privilege — do it before reaching for anything in §8. Run on a dedicated, quiesced node (§3) with no concurrent RPC traffic during timing; that removes more noise than any instrumentation would merely reveal. 8. **Do this promptly.** `debug_trace*` is capped to `MaxTraceLookbackBlocks` behind the node's current tip (default 10,000, `evmrpc/config/config.go`, configurable via `evm.max_trace_lookback_blocks`). A replayer still catching up can process 20-30 blocks/sec — 10,000 blocks ages out in under 10 minutes. If you collect a batch of tx hashes and then profile them later, some will fail with `"block number ... is beyond max lookback of 10000"`. Verified this session: exactly this happened mid-collection. Either profile each tx immediately after finding it, or wait until the node has fully caught up to live tip (stable height) before doing a bulk collection pass. -9. **Sample size, and a real mistake to avoid.** A single tx pair doesn't tell you much, and **neither does a sample with no variance in gas** — collecting 25 txs from a narrow, bot-dominated block window this session produced near-identical gas (458,696 for 24 of 25 txs) and a meaningless r=0.11, not because the correlation is weak but because a nearly-constant independent variable can't test it at all. Deliberately collect a real range of gas magnitudes/contracts before computing anything. **The following numbers are this session's illustrations of the primitive working, not a general finding about Sei's gas schedule — treat them as sizing the tool, not the research conclusion, and re-derive your own:** on a properly diverse sample (gas ranging 0–1.3M), default-StructLogger gave Pearson r=0.9914 between gas and `executionNanos`; switching to `callTracer` on an equally diverse sample gave **r=0.8093** — still a strong, real correlation, but honestly lower once tracer contamination is reduced. A same-gas cluster (~130k-133k, n=8) still showed a genuine ~2x spread in measured `callTracer` time (29ms-59ms) — likely cache-warmth or scheduling noise per step 7, not a fluke. Always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). +9. **Sample size, a real mistake to avoid, and what the correlation number can and can't tell you.** A single tx pair doesn't tell you much, and **neither does a sample with no variance in gas** — collecting 25 txs from a narrow, bot-dominated block window this session produced near-identical gas (458,696 for 24 of 25 txs) and a meaningless r=0.11, not because the correlation is weak but because a nearly-constant independent variable can't test it at all. Deliberately collect a real range of gas magnitudes/contracts before computing anything. + + **The following numbers are this session's illustrations of the primitive working, not a general finding about Sei's gas schedule — treat them as sizing the tool, not the research conclusion, and re-derive your own:** on a properly diverse sample (gas ranging 0–1.3M, n=25, single sample per tx — **not** the min-of-N discipline from step 7, so treat this as a lower bound on achievable precision), default-StructLogger gave Pearson r=0.9914 between gas and `executionNanos`; switching to `callTracer` on an equally diverse sample gave r=0.8093 — still a strong, real correlation, but honestly lower once tracer contamination is reduced. + + **Pearson r on a small, high-leverage sample is a smoke test that the tool produces signal, not a measure of pricing fidelity — don't let it become the headline finding.** A couple of extreme high-gas points and one `gas=0` point (an edge case worth checking directly — real EVM txs carry ≥21k intrinsic gas, so confirm what that tx actually is before including it) can dominate r on n=25; the question the gas-repricing project actually cares about is whether time-per-gas is *constant across opcode/contract mixes*, which is a residual-structure question, not an aggregate-association one. Report r² and the spread of individual (gas, time) points around a fit line, not just r; Spearman's rank correlation is more robust to the leverage/heteroscedasticity here than Pearson. Treat the differential-bytecode-loop (§6) as the real instrument for pricing-fidelity questions — the aggregate correlation in this step only establishes that the primitive measures something real. + + **A same-gas cluster (~130k-133k, n=8) showed a genuine ~2x spread in measured `callTracer` time (29ms-59ms) — do not dismiss this as noise addressable by step 7's min-of-N.** Min-of-N reduces measurement noise *across repeated calls of the same tx*; it says nothing about variance *across 8 different transactions* that happen to charge similar gas. Same gas does not mean same work — two txs charging ~130k can run entirely different opcode/state-access mixes, and a real spread here is arguably the exact phenomenon the gas-repricing project is hunting (equal gas, unequal real compute), not something to average away. **The correct check:** repeat each of the 8 txs individually with min-of-N. If the spread collapses within a single tx's repeats, it was measurement noise (most likely cache-warmth, since each target tx executes against post-replay state whose warmth depends on what its own block's preceding txs touched). If the spread persists across the 8 min-of-N values, that's a real, reportable gas-vs-compute divergence at fixed gas — treat it as a finding, not a fluke, and don't pre-judge it as noise before checking. + + Always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). 10. **The bigger limitation no measurement technique here closes: `Y=X-1` blocks the question with the most repricing leverage for state opcodes.** For `SLOAD`/`SSTORE`, the decision-relevant variable is how access cost scales with state size / tree depth as the chain grows — which requires holding bytecode fixed while sweeping the base height, i.e. the general `Y` for cycles/instructions on the differential-loop's bytecode runs is a hardware counter read, not a Go-symbol attach — none of the stack-copy/uretprobe/thread-migration hazards below apply to it. CPU cycles are immune to GC and scheduling jitter in a way wall-clock `time.Since` isn't, so `cycles(A) − cycles(B)) ÷ iterations` is a genuinely independent cross-check on the differential-loop's nanos-per-opcode numbers. Treat this as a **deferred, second-iteration cross-validation** — worth running only if you still see unexplained variance after the min-of-N discipline, or if a repricing recommendation needs to rest on absolute per-opcode precision rather than relative ordering. It still needs `CAP_PERFMON` and the same human sign-off as anything else in this section. +**The one kernel-level tool worth considering — and it isn't a uprobe.** `perf stat -p ` for cycles/instructions on the differential-loop's bytecode runs is a hardware counter read, not a Go-symbol attach — none of the stack-copy/uretprobe/thread-migration hazards below apply to it. It's immune to *off-CPU descheduling* jitter in a way wall-clock `time.Since` isn't — but it's process-wide, so it still counts GC worker threads' own cycles; that only cancels out in the `cycles(A) − cycles(B)` differential under an assumption of roughly equal GC load between the two variants, not as an absolute guarantee. With that caveat, `(cycles(A) − cycles(B)) ÷ iterations` is a genuinely independent cross-check on the differential-loop's nanos-per-opcode numbers. Treat this as a **deferred, second-iteration cross-validation** — worth running only if you still see unexplained variance after the min-of-N discipline, or if a repricing recommendation needs to rest on absolute per-opcode precision rather than relative ordering. It still needs `CAP_PERFMON` and the same human sign-off as anything else in this section. If you do need actual uprobes/`offcputime`/`profile`, the privileged-attach gate and symbol-resolution guidance below still apply. @@ -265,7 +277,7 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta - [ ] **Chosen §2a (release-pinned) or §2b (`mock_chain_validation`) deliberately**, not defaulted to an arbitrary image tag. If §2b, confirmed `sei_unsafe_validation_skipped_total` is being watched, not ignored. - [ ] **Functionality 1:** replayer restoring to `` and replaying forward (§4); confirmed `Y=X-1` actually answers the intended question, or explicitly flagged that it doesn't. - [ ] **Functionality 1/2 measurement:** calling `debug_traceTransactionProfile` with `{"tracer":"callTracer"}` — **not the default invocation**, which attaches a full StructLogger and inflates `executionNanos` by roughly 2x (§4 step 6). Using `result.profile.phases.executionNanos` (not `totalNanos`, not a log-derived latency proxy) for timing, `result.trace.gas`/`gasUsed` for gas. -- [ ] **Reporting interpreter-compute and state-I/O separately** (`executionNanos − storeNanos` vs. `storeNanos` from `result.profile.store`), not one blended number (§4 step 6). +- [ ] **Not subtracting `result.profile.store` totals from `executionNanos`** — that arithmetic is broken (different time windows; caught in peer review, §4 step 6). Cleanly separating interpreter compute from state I/O per tx needs a sei-chain change, not a subtraction on today's fields. - [ ] **Timing runs use min-of-N** (forced GC before each, dedicated/quiesced node, §4 step 7), not a single sample per tx. - [ ] **Traced promptly** relative to `MaxTraceLookbackBlocks` — not collecting a large batch of tx hashes to profile long after the fact on a still-catching-up node. - [ ] **Functionality 2:** using Path B (real submitted tx + §4's primitive) unless a genuine read-only-simulation requirement forces Path A — in which case, the execution-time gap in `debug_traceCall` is flagged explicitly, not worked around with a fabricated number. @@ -281,8 +293,9 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta - Sei-EVM gas/precompile specifics: `/evm` skill — `kit-evm-parity-gas.md`, `kit-sei-precompiles.md`; `x/evm/types/params.go` in `sei-chain` for the governance-mutable param set. - Version-pinning / `mock_chain_validation`: `sei-chain` — `sei-tendermint/types/consensus_policy.go`, `consensus_policy_mock_chain_validation.go`, `consensus_policy_default.go`, `policy_metrics.go`; `.github/workflows/ecr.yml`, `.github/workflows/nightly-ecr.yml` for the image-tag scheme. -- Measurement primitive: `sei-chain` — `evmrpc/trace_profile.go` (`debug_traceTransactionProfile`, `profiledTraceTx`, `phaseDurations`, the `historicalDbLookupNanos`→`otherNanos` subtraction that does *not* touch `executionNanos`), `evmrpc/simulate.go` (`initializeBlock`, `replayTransactionTillIndex`, `StateAtBlock`, plain `Call`), `evmrpc/tracers.go` (`TraceCall`), `evmrpc/block_trace_profiled.go` (`debug_traceBlockByNumber/Hash` — profiling exists but is passed `nil` here, not surfaced; also the `config==nil` → default `StructLogger` path and the `executionNanos` bracket around `core.ApplyTransactionWithEVM` — this is why the default invocation over-times relative to `callTracer`), `evmrpc/config/config.go` (`MaxTraceLookbackBlocks`). +- Measurement primitive: `sei-chain` — `evmrpc/trace_profile.go` (`debug_traceTransactionProfile`, `profiledTraceTx`, `phaseDurations`; the store trace dump spans the whole statedb lifetime including replayed preceding txs, not just the target tx — this is why `result.profile.store` can't be subtracted from `executionNanos`), `evmrpc/simulate.go` (`initializeBlock`, `replayTransactionTillIndex`, `StateAtBlock`, plain `Call`), `evmrpc/tracers.go` (`TraceCall`), `evmrpc/block_trace_profiled.go` (`debug_traceBlockByNumber/Hash` — profiling exists but is passed `nil` here, not surfaced; also the `config==nil` → default `StructLogger` path and the `executionNanos` bracket around `core.ApplyTransactionWithEVM` — this is why the default invocation over-times relative to `callTracer`), `evmrpc/config/config.go` (`MaxTraceLookbackBlocks`), `sei-cosmos/types/context.go` (`WithIsTracing` — where the store tracer gets minted once for the whole trace), `sei-cosmos/types/tracer.go` (`StoreTraceDump` — read/write stats tracked separately). +- Snapshot restore selection: `seictl` — `sidecar/tasks/snapshot_restore.go` (`resolveKeyForHeight` — the authoritative source for `TargetHeight`'s "highest snapshot <= height" semantics; the `SeiNode` CRD's own doc comment in `sei-k8s-controller`'s `api/v1alpha1/common_types.go` only describes the zero-value case and reads differently — trust seictl's source over it). - Harbor chain spin-up mechanics: `/harbor-dev` skill, `references/ephemeral-chain-flow.md`, `references/seictl-cli.md`, `references/harbor-cluster.md`. -- Hardware-parity mechanism: `sei-k8s-controller` — `internal/platform/platform.go` (`NodepoolForMode`), `internal/task/bootstrap_resources.go`, `internal/noderesource/noderesource.go` (also home of the `sei.io/dedicated-node` annotation); `sei-protocol/platform` — `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml`, `clusters/{harbor,prod}/default/nodepool-{default,archive}.yaml`. +- Hardware-parity mechanism: `sei-k8s-controller` — `internal/platform/platform.go` (`NodepoolForMode`), `internal/task/bootstrap_resources.go`, `internal/noderesource/noderesource.go` (also home of the `sei.io/dedicated-node` annotation and the documented no-SeiNode-condition-on-stuck-Pending behavior), `internal/planner/replay.go` (`buildRunningPlan` — why an annotation-only change on a Running replayer doesn't trigger a pod rebuild), `internal/planner/planner.go` (`imageDrifted` — image-only, not annotation-aware); `sei-protocol/platform` — `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml`, `clusters/{harbor,prod}/default/nodepool-{default,archive}.yaml`. - Harbor `eng-` Pod Security / admission posture: `sei-protocol/platform` — `clusters/harbor/engineers/base/namespace.yaml`, `clusters/prod/walle/{namespace,podcomposition-policy}.yaml` (the posture to eventually mirror), `clusters/harbor/chaos-mesh/chaos-mesh.yaml`, `clusters/harbor/admission/tenant-hostname-policy.yaml`. - Execution entry points (§8 only): `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here. From 9ffe2277b190f68e411c6fd7b646442c29c1cb2b Mon Sep 17 00:00:00 2001 From: bdchatham Date: Fri, 10 Jul 2026 16:21:09 -0700 Subject: [PATCH 5/6] docs(runbooks): fold in intrinsic-gas + per-block-timing findings, move OTel tracing to appendix Adds two validated findings to the main body: subtracting intrinsic gas before correlating (a client-side fix, no sei-chain change), and the full per-block FinalizeBlock timing answer via the existing "execution block time" log line, with the replayer-vs-validator and EVM-vs-native-traffic caveats. Moves the OpenTelemetry tracing investigation to Appendix A. It was built and tested with real data on two replayer nodes (concurrent and serial OCC execution against the identical historical chain) and found to have an irreducible, deterministic node-level confound (cross-node Jaccard 0.92 on the same outlier transactions rules out both OCC contention and random per-process noise) that biases the correlation rather than measuring it. Documented so it isn't re-proposed without this context. --- .../benchmarking-evm-gas-vs-execution-time.md | 82 +++++++++++++++---- 1 file changed, 68 insertions(+), 14 deletions(-) diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md index dfc080f..2cab968 100644 --- a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -1,10 +1,10 @@ # Benchmarking EVM Gas vs. Real Execution Time on Sei **Audience:** engineers (and their agents) on the gas-repricing project investigating whether Sei's EVM gas schedule tracks real compute cost. Covers both target functionalities: (1) replaying a real historical block against restored state and measuring gas + execution time per tx, and (2) executing arbitrary bytecode against a specific historical state and measuring the same. -**Scope:** standing up a replayer node, choosing the right binary version deliberately (§2), and using `debug_traceTransactionProfile` — a validated, existing RPC method — to get clean per-tx gas + execution-time pairs with a single call, no kernel instrumentation required for either functionality as specified. eBPF (§8) is a fallback for needs beyond what's documented here (e.g. true per-opcode dispatch timing), not the default path. +**Scope:** standing up a replayer node, choosing the right binary version deliberately (§2), and using `debug_traceTransactionProfile` — a validated, existing RPC method — to get clean per-tx gas + execution-time pairs with a single call, no kernel instrumentation required for either functionality as specified. Also covers a related, separate question that comes up alongside this work: reading whole-block execution time with no new RPC or instrumentation (§8). eBPF (§9) is a fallback for needs beyond what's documented here (e.g. true per-opcode dispatch timing), not the default path. A third avenue, OpenTelemetry tracing, was built and tested against real data and found not fit for this purpose — see Appendix A before re-proposing it. **Not in scope:** *why* Sei's gas schedule has the specific values it does (a governance/design question — capture a resulting change proposal as a `/design` doc, not here). Running any of this against a shared/long-lived/production chain, or against the canonical shared archive node in a way that mutates its state. -> **Read this whole section before running anything.** Six things make the naive version of this experiment **silently wrong**: (1) at least one opcode's gas cost (`SSTORE`) is a **governance-mutable Sei chain param that already diverges from upstream go-ethereum** — using it as your "does gas match compute" baseline tests a value that was hand-tuned away from compute cost *by design*; (2) **a replayer's binary version must match what actually produced the history you're replaying, or every block after the first will fail app-hash validation** — confirmed empirically this session (§2); (3) **`debug_traceTransactionProfile` and the rest of the `debug_trace*` family are capped to `MaxTraceLookbackBlocks` (default 10,000) blocks behind the node's current tip** — a fast-catching-up replayer ages a target height out of that window in minutes; trace it soon after it's produced (§4); (4) **only `Y=X-1` is supported for Functionality 1 by any tooling that exists today** — there is no code path for pinning a real historical block's transactions to an arbitrarily older base state (§4); (5) **Functionality 2's execution-time output has a real, unclosed gap** — `debug_traceCall` returns gas but no timing field for arbitrary bytecode against arbitrary `Y` (§5); (6) **attaching any eBPF probe to a shared harbor pod requires a privileged kernel capability grant** — nothing technically blocks it today, but that's an unhardened gap, not a green light (§8). +> **Read this whole section before running anything.** Six things make the naive version of this experiment **silently wrong**: (1) at least one opcode's gas cost (`SSTORE`) is a **governance-mutable Sei chain param that already diverges from upstream go-ethereum** — using it as your "does gas match compute" baseline tests a value that was hand-tuned away from compute cost *by design*; (2) **a replayer's binary version must match what actually produced the history you're replaying, or every block after the first will fail app-hash validation** — confirmed empirically this session (§2); (3) **`debug_traceTransactionProfile` and the rest of the `debug_trace*` family are capped to `MaxTraceLookbackBlocks` (default 10,000) blocks behind the node's current tip** — a fast-catching-up replayer ages a target height out of that window in minutes; trace it soon after it's produced (§4); (4) **only `Y=X-1` is supported for Functionality 1 by any tooling that exists today** — there is no code path for pinning a real historical block's transactions to an arbitrarily older base state (§4); (5) **Functionality 2's execution-time output has a real, unclosed gap** — `debug_traceCall` returns gas but no timing field for arbitrary bytecode against arbitrary `Y` (§5); (6) **attaching any eBPF probe to a shared harbor pod requires a privileged kernel capability grant** — nothing technically blocks it today, but that's an unhardened gap, not a green light (§9). --- @@ -36,7 +36,7 @@ The question: for a given opcode/tx, does the gas charged scale with the real wa **Confounds that apply regardless of functionality:** - **Per-tx fixed overhead** (signature verification, the Cosmos ante handler, the EVM↔Cosmos StateDB bridge) is real wall-clock time attributable to *no* opcode — it dilutes a measurement unless your sample size / loop count makes it a small fraction of the total. -- **OCC (optimistic concurrent execution) does not break per-tx correlation for §4's RPC primitive.** Sei applies a block's transactions through a parallel scheduler (`app/abci.go`, `sei-cosmos/tasks/scheduler.go`), not a serial loop. A raw uprobe/uretprobe bracket around `StateTransition.Execute` (§8) *would* need OCC disabled to get clean per-tx numbers, since multiple `Execute` calls can run concurrently on different threads. `debug_traceTransactionProfile` sidesteps this entirely: it isolates a single tx's execution by replaying the block *up to* that tx's index and then executing just it (`evmrpc/simulate.go`'s `replayTransactionTillIndex`), independent of how the block was originally scheduled. You do **not** need to disable OCC on the replayer to get clean per-tx numbers via this RPC. +- **OCC (optimistic concurrent execution) does not break per-tx correlation for §4's RPC primitive.** Sei applies a block's transactions through a parallel scheduler (`app/abci.go`, `sei-cosmos/tasks/scheduler.go`), not a serial loop. A raw uprobe/uretprobe bracket around `StateTransition.Execute` (§9) *would* need OCC disabled to get clean per-tx numbers, since multiple `Execute` calls can run concurrently on different threads. `debug_traceTransactionProfile` sidesteps this entirely: it isolates a single tx's execution by replaying the block *up to* that tx's index and then executing just it (`evmrpc/simulate.go`'s `replayTransactionTillIndex`), independent of how the block was originally scheduled. You do **not** need to disable OCC on the replayer to get clean per-tx numbers via this RPC. - **Whole-block wall-clock time (log-line `latency_ms` on `"finalized block"`) is the wrong signal — verified empirically, not theoretically.** It's diluted by non-EVM per-block work (oracle vote tally, IBC, begin/end-blockers for every module) and, on a replayer still catching up, contention with block-fetching. A same-session sample: correlating gas against whole-block latency gave Pearson r=0.301; filtering to EVM-active blocks and using the OCC-scheduler's tx-execution-phase latency improved it to r=0.618, still with real outliers. Correlating gas against `debug_traceTransactionProfile`'s `executionNanos` on the same chain gave r=0.9914 on an initial 9-tx sample — **but that invocation used the default StructLogger tracer, which adds opcode-scaled overhead into the timed window; switching to the lighter `callTracer` on a properly diverse sample gave r=0.8093** (§4 step 9 has the full story, including a sampling mistake worth learning from). Use the RPC with `callTracer`, not a log-derived proxy and not the default invocation. --- @@ -139,7 +139,7 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive **Do not subtract `result.profile.store`'s totals from `executionNanos` to isolate interpreter-only compute — that arithmetic is broken, caught in this session's own peer review before it shipped.** `executionNanos` brackets only the target tx (the window around `core.ApplyTransactionWithEVM`), but the store trace dump accumulates over the *entire* statedb lifetime — including every preceding tx's execution during `replayHistoricalTxsNanos`'s `ReplayTransactionTillIndex` pass, on the same tracer instance. For a tx at index k in its block, the store total includes k other transactions' state I/O, not just this one's; subtracting it from a single tx's `executionNanos` will silently go negative for anything past the first tx or two in a block. It's also incomplete even ignoring the windowing bug: the read-side lookup total (`historicalDbLookupNanos`) doesn't include `Set`/`Delete` (write) latency, which lives in a separate stats bucket — so it wouldn't capture SSTORE cost even if correctly scoped. Cleanly separating interpreter compute from state I/O per tx needs a sei-chain change to reset or snapshot the store tracer at the execution boundary — flag that as follow-on engineering rather than approximating it with this subtraction. Until then, report `executionNanos` as one number (compute + this tx's state I/O, not cleanly separable today) and treat `result.profile.store`'s counts as informative about which modules/keys were touched, not as a subtractable timing quantity. -7. **Take min-of-N, not a single sample, per tx.** Wall-clock timing (`time.Since`) absorbs GC pauses, goroutine preemption, and OS thread migration — noise that only ever *adds* time, never subtracts it. Across several repeated profile calls on the same tx, the **minimum** is the closest estimate of true compute cost, and the spread across repeats tells you the noise magnitude directly. Force a GC before each run if your tooling allows it (or note that you didn't). This costs nothing and needs no elevated privilege — do it before reaching for anything in §8. Run on a dedicated, quiesced node (§3) with no concurrent RPC traffic during timing; that removes more noise than any instrumentation would merely reveal. +7. **Take min-of-N, not a single sample, per tx.** Wall-clock timing (`time.Since`) absorbs GC pauses, goroutine preemption, and OS thread migration — noise that only ever *adds* time, never subtracts it. Across several repeated profile calls on the same tx, the **minimum** is the closest estimate of true compute cost, and the spread across repeats tells you the noise magnitude directly. Force a GC before each run if your tooling allows it (or note that you didn't). This costs nothing and needs no elevated privilege — do it before reaching for anything in §9. Run on a dedicated, quiesced node (§3) with no concurrent RPC traffic during timing; that removes more noise than any instrumentation would merely reveal. 8. **Do this promptly.** `debug_trace*` is capped to `MaxTraceLookbackBlocks` behind the node's current tip (default 10,000, `evmrpc/config/config.go`, configurable via `evm.max_trace_lookback_blocks`). A replayer still catching up can process 20-30 blocks/sec — 10,000 blocks ages out in under 10 minutes. If you collect a batch of tx hashes and then profile them later, some will fail with `"block number ... is beyond max lookback of 10000"`. Verified this session: exactly this happened mid-collection. Either profile each tx immediately after finding it, or wait until the node has fully caught up to live tip (stable height) before doing a bulk collection pass. @@ -153,7 +153,9 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive Always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). -10. **The bigger limitation no measurement technique here closes: `Y=X-1` blocks the question with the most repricing leverage for state opcodes.** For `SLOAD`/`SSTORE`, the decision-relevant variable is how access cost scales with state size / tree depth as the chain grows — which requires holding bytecode fixed while sweeping the base height, i.e. the general `Y` for cycles/instructions on the differential-loop's bytecode runs is a hardware counter read, not a Go-symbol attach — none of the stack-copy/uretprobe/thread-migration hazards below apply to it. It's immune to *off-CPU descheduling* jitter in a way wall-clock `time.Since` isn't — but it's process-wide, so it still counts GC worker threads' own cycles; that only cancels out in the `cycles(A) − cycles(B)` differential under an assumption of roughly equal GC load between the two variants, not as an absolute guarantee. With that caveat, `(cycles(A) − cycles(B)) ÷ iterations` is a genuinely independent cross-check on the differential-loop's nanos-per-opcode numbers. Treat this as a **deferred, second-iteration cross-validation** — worth running only if you still see unexplained variance after the min-of-N discipline, or if a repricing recommendation needs to rest on absolute per-opcode precision rather than relative ordering. It still needs `CAP_PERFMON` and the same human sign-off as anything else in this section. @@ -253,7 +274,7 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta --- -## 9. Failure modes (quick reference) +## 10. Failure modes (quick reference) | Symptom | Cause | Fix | |---|---|---| @@ -266,12 +287,14 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta | Path A (`debug_traceCall`) gives gas but no execution time | `TraceCall` was never wired to the `phaseDurations` profiling mechanism — a real, unclosed gap, not a config issue (§5) | Use Path B: submit a real signed tx against your own restored-to-`` node, then `debug_traceTransactionProfile` its mined hash. | | Trying to replay block `X` against a `Y` that isn't `X-1` | No existing tooling supports the general case (§4) | This is new engineering, not a config change — confirm whether `Y=X-1` actually answers the real question before treating it as a blocker. | | "Gas doesn't match compute" conclusion on `SSTORE` | Tested a governance-mutable param as if compute-derived (§1) | Reframe as confirming the known divergence; re-run on `KECCAK256`/arithmetic for the real test. | -| Attaching an eBPF probe changes replay throughput (blocks/s) | High context-switch rate on a fast replayer makes `offcputime`/probe overhead non-negligible (§8) | A/B throughput probe-on vs. probe-off; narrow the window or lower sampling rate. Consider whether you need eBPF at all — §4's primitive may already answer the question. | -| Can't get sign-off / told to stop before attaching anything privileged | §8's gate working as intended | Platform team's call — escalate, don't self-approve a scoped-down version of the same attach. | +| Attaching an eBPF probe changes replay throughput (blocks/s) | High context-switch rate on a fast replayer makes `offcputime`/probe overhead non-negligible (§9) | A/B throughput probe-on vs. probe-off; narrow the window or lower sampling rate. Consider whether you need eBPF at all — §4's primitive may already answer the question. | +| Can't get sign-off / told to stop before attaching anything privileged | §9's gate working as intended | Platform team's call — escalate, don't self-approve a scoped-down version of the same attach. | +| Correlating gas against a mixed-traffic block's `total_execution_ms` gives an inconsistent EVM-time proxy | The block isn't all-EVM — native Cosmos messages (IBC, staking, gov) are baked into the same aggregate (§8) | Confirm the sampled block is all-EVM before treating it as an EVM-time proxy, or use `run_msg_latency{type=".../evm.Msg/EVMTransaction"}` instead (§8). | +| Considering OpenTelemetry tracing spans instead of `debug_traceTransactionProfile` for per-tx timing | Looks like it would save a per-tx RPC round-trip | Don't, without reading Appendix A first — built and tested with real data, found an irreducible deterministic node-level confound in both concurrent and serial execution modes. | --- -## 10. Pre-flight checklist +## 11. Pre-flight checklist - [ ] Confirmed the target node landed on `r6i`/`r7i` via `sei-node`/`sei-archive` (§3) — don't trust the default without checking. - [ ] **Chosen §2a (release-pinned) or §2b (`mock_chain_validation`) deliberately**, not defaulted to an arbitrary image tag. If §2b, confirmed `sei_unsafe_validation_skipped_total` is being watched, not ignored. @@ -283,9 +306,37 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta - [ ] **Functionality 2:** using Path B (real submitted tx + §4's primitive) unless a genuine read-only-simulation requirement forces Path A — in which case, the execution-time gap in `debug_traceCall` is flagged explicitly, not worked around with a fabricated number. - [ ] Recorded live gas parameter values (e.g. `SeiSstoreSetGasEIP2200`) + image digest before drawing conclusions. - [ ] **Sample size covers a real range of gas magnitudes/contracts, and you've checked for variance before computing a correlation** — a narrow, bot-dominated window with near-constant gas will give a meaningless near-zero r regardless of the true relationship (§4 step 9). +- [ ] **Correlating `executionGas = gasUsed - IntrinsicGas(...)` against `executionNanos`**, not raw `gasUsed` (§4 step 10) — the intrinsic-gas floor moves gas without moving time and dilutes the signal, especially at the low-gas end. - [ ] Re-checked results against §1 (`SSTORE`/precompile confounds) before concluding anything about whether gas tracks compute. -- [ ] **Named explicitly whether the question needs state-opcode cost-vs-state-size scaling** (§4 step 10) — if so, flagged that as an open engineering gap, not something a `Y=X-1` measurement answers. -- [ ] Confirmed eBPF uprobes/`offcputime` are actually necessary (§8) before requesting sign-off for a privileged attach — §4-§7 already answer both functionalities as specified without them; `perf stat` cycle counters (§8) are the one kernel-level tool worth considering, and only as a deferred cross-validation. +- [ ] **Named explicitly whether the question needs state-opcode cost-vs-state-size scaling** (§4 step 11) — if so, flagged that as an open engineering gap, not something a `Y=X-1` measurement answers. +- [ ] **If reading whole-block execution time (§8), confirmed the sampled block is all-EVM** before treating `total_execution_ms` as an EVM-time proxy, and confirmed whether the node is a replayer or a live validator (the optimistic-processing caveat, §8). +- [ ] Confirmed eBPF uprobes/`offcputime` are actually necessary (§9) before requesting sign-off for a privileged attach — §4-§7 already answer both functionalities as specified without them; `perf stat` cycle counters (§9) are the one kernel-level tool worth considering, and only as a deferred cross-validation. + +--- + +## Appendix A: Invalidated avenues (read before re-proposing these) + +One path was investigated in depth for this project, built, run against real data, and found not fit for purpose. Documented here so the next engineer doesn't re-spend the time re-discovering why. + +### A.1 OpenTelemetry tracing for per-tx gas-vs-time correlation + +Sei ships real OTel tracing (Jaeger exporter, currently disabled by default), with a full span tree — `Block → BeginBlock/DeliverTxBatch(→SchedulerExecuteAll→SchedulerExecute→AnteHandler+RunMsgs)/MidBlock/EndBlock` — carrying `txHash`/`absoluteIndex`/`txIncarnation` on the OCC scheduler spans. Given that `debug_traceTransactionProfile` (§4) needs a per-tx RPC call each, tracing looked like an attractive way to get the same numbers as a byproduct of normal block processing, no extra calls needed. + +**Three independent specialist reviews (systems-engineer, observability-platform-engineer, opentelemetry-expert) first found real, structural problems, later confirmed against actual data:** a process-global mutex around every span-start call (`sei-cosmos/utils/tracing/tracer.go`) serializes span creation across OCC's concurrent workers; no trace backend exists anywhere in harbor; the collector endpoint is hardcoded to `localhost:14268`, forcing a sidecar-in-pod deployment shape; the enabling flag is CLI-argument-only, not reachable through the SeiNode CRD's config overrides; gas is not on any span, requiring a secondary height+txHash join regardless. + +**Built and tested it anyway, with real data, to settle the efficacy question rather than reason about it abstractly.** Stood up two tracing-enabled replayer nodes in `eng-fromtherain` (a wrapper image backgrounding an in-process Jaeger collector before exec'ing the real seid binary, so AppHash fidelity against pacific-1 was unaffected) — one running default concurrent OCC execution, one with `chain.concurrency_workers=1` forcing strict serial execution — both restored from the identical S3 snapshot, so both replayed the identical real historical chain. + +**The result was a real, informative negative:** + +| | r(gas, total span time) | r(gas, `RunMsgs` span time) | +|---|---|---| +| Concurrent OCC (n=190 EVM txs) | 0.736 | 0.686 | +| Strictly serial (same 190 txs) | 0.690 | 0.574 | +| Already-validated `debug_traceTransactionProfile` (§4) | — | 0.809 | + +Serializing execution did not close the gap to the validated method; it made the clean-signal correlation worse. Both modes showed a recurring outlier: certain transactions' `AnteHandler` span durations inflated 100-155x relative to their `RunMsgs` span, on the same transactions in both datasets (confirmed by joining on identical `txHash` across the two nodes). Because strict serial execution has zero cross-goroutine overlap (confirmed directly from span start-times — back-to-back, ~4µs gaps), this ruled out OCC/mutex contention as the cause. Comparing the same outlier transactions across the two independent nodes (separate pods, separate Jaeger processes, separate GC timing) showed the inflation lands on the *same transactions*, at matching magnitudes, on both (Jaccard 0.92) — which rules out random per-process contention too, including the in-process Jaeger collector stealing CPU from `seid`. The remaining explanation both datasets are consistent with: a deterministic, per-height node-level stall — most likely GC mark-assist or a memIAVL commit/snapshot-flush cycle — that inflates whatever span happens to be open at that moment, landing more often on longer (higher-gas) transactions and inflating the correlation on exactly the end that makes it look better. + +**Verdict: not fit for absolute per-tx gas-vs-execution-time correlation, in either execution mode.** The already-validated `debug_traceTransactionProfile` isolates a transaction's re-execution from the live block-delivery pipeline entirely — no concurrent scheduler, no GC/commit cycle landing mid-measurement — which is the structural reason it measures closer to true compute cost. OTel tracing remains genuinely useful for a narrower, different question: structural/relative phase attribution within a block or transaction (what fraction of the time went to ante-handling vs. message execution vs. begin/end-blockers), not for this correlation. --- @@ -294,8 +345,11 @@ Raw per-invocation duration via `uprobe`/`uretprobe` is rehearsal-only: Go's sta - Sei-EVM gas/precompile specifics: `/evm` skill — `kit-evm-parity-gas.md`, `kit-sei-precompiles.md`; `x/evm/types/params.go` in `sei-chain` for the governance-mutable param set. - Version-pinning / `mock_chain_validation`: `sei-chain` — `sei-tendermint/types/consensus_policy.go`, `consensus_policy_mock_chain_validation.go`, `consensus_policy_default.go`, `policy_metrics.go`; `.github/workflows/ecr.yml`, `.github/workflows/nightly-ecr.yml` for the image-tag scheme. - Measurement primitive: `sei-chain` — `evmrpc/trace_profile.go` (`debug_traceTransactionProfile`, `profiledTraceTx`, `phaseDurations`; the store trace dump spans the whole statedb lifetime including replayed preceding txs, not just the target tx — this is why `result.profile.store` can't be subtracted from `executionNanos`), `evmrpc/simulate.go` (`initializeBlock`, `replayTransactionTillIndex`, `StateAtBlock`, plain `Call`), `evmrpc/tracers.go` (`TraceCall`), `evmrpc/block_trace_profiled.go` (`debug_traceBlockByNumber/Hash` — profiling exists but is passed `nil` here, not surfaced; also the `config==nil` → default `StructLogger` path and the `executionNanos` bracket around `core.ApplyTransactionWithEVM` — this is why the default invocation over-times relative to `callTracer`), `evmrpc/config/config.go` (`MaxTraceLookbackBlocks`), `sei-cosmos/types/context.go` (`WithIsTracing` — where the store tracer gets minted once for the whole trace), `sei-cosmos/types/tracer.go` (`StoreTraceDump` — read/write stats tracked separately). +- Intrinsic gas (§4 step 10): go-ethereum fork's `core/state_transition.go` (`IntrinsicGas`); Sei's call sites — `app/app.go:2016`, `app/ante/evm_checktx.go:107`, `x/evm/ante/basic.go:51` — all identical, all with fork-activation flags hardcoded `true`. +- Per-block execution time (§8): `sei-cosmos/baseapp/abci.go` (the `"execution block time"` log line), `sei-cosmos/baseapp/metrics.go` (`run_msg_latency` and the rest of the OTel metric set), `app/app.go` (`FinalizeBlocker`'s optimistic-processing path), `x/evm/keeper/msg_server.go` (`EVMTransaction` — confirms the exact message-type URL to filter `run_msg_latency` by). +- OTel tracing investigation, invalidated (Appendix A): `sei-cosmos/utils/tracing/tracer.go` (the span-start mutex, the hardcoded collector endpoint), `sei-cosmos/tasks/scheduler.go` (the OCC scheduler spans), `sei-cosmos/server/start.go` (the CLI-only `--tracing` flag). - Snapshot restore selection: `seictl` — `sidecar/tasks/snapshot_restore.go` (`resolveKeyForHeight` — the authoritative source for `TargetHeight`'s "highest snapshot <= height" semantics; the `SeiNode` CRD's own doc comment in `sei-k8s-controller`'s `api/v1alpha1/common_types.go` only describes the zero-value case and reads differently — trust seictl's source over it). - Harbor chain spin-up mechanics: `/harbor-dev` skill, `references/ephemeral-chain-flow.md`, `references/seictl-cli.md`, `references/harbor-cluster.md`. - Hardware-parity mechanism: `sei-k8s-controller` — `internal/platform/platform.go` (`NodepoolForMode`), `internal/task/bootstrap_resources.go`, `internal/noderesource/noderesource.go` (also home of the `sei.io/dedicated-node` annotation and the documented no-SeiNode-condition-on-stuck-Pending behavior), `internal/planner/replay.go` (`buildRunningPlan` — why an annotation-only change on a Running replayer doesn't trigger a pod rebuild), `internal/planner/planner.go` (`imageDrifted` — image-only, not annotation-aware); `sei-protocol/platform` — `clusters/{harbor,prod}/sei-k8s-controller/controller-config.yaml`, `clusters/{harbor,prod}/default/nodepool-{default,archive}.yaml`. - Harbor `eng-` Pod Security / admission posture: `sei-protocol/platform` — `clusters/harbor/engineers/base/namespace.yaml`, `clusters/prod/walle/{namespace,podcomposition-policy}.yaml` (the posture to eventually mirror), `clusters/harbor/chaos-mesh/chaos-mesh.yaml`, `clusters/harbor/admission/tenant-hostname-policy.yaml`. -- Execution entry points (§8 only): `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here. +- Execution entry points (§9 only): `sei-chain` — `x/evm/keeper/msg_server.go` (`applyEVMMessage`); the vendored go-ethereum fork's `core/state_transition.go` (`StateTransition.Execute`) and `core/vm/interpreter.go` (`EVMInterpreter.Run`) — resolve the fork version via `go.mod`, don't hardcode it here. From 901cccf1b890b327ec321369c49418666e07117b Mon Sep 17 00:00:00 2001 From: bdchatham Date: Fri, 10 Jul 2026 16:43:49 -0700 Subject: [PATCH 6/6] docs(runbooks): fix xreview round 2 findings on the gas-vs-time runbook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four independent blinded reviewers (systems-engineer, kubernetes-specialist, opentelemetry-expert as dissenter, prose-steward) reviewed the restructuring in 9ffe227. Fixes: - run_msg_latency label value was wrong (cited the gRPC service/method path instead of the actual proto message-type name MsgTypeURL emits) - would have silently matched zero series if copy-pasted. - block_process_duration metric name was missing its rendered app_ prefix and _seconds unit suffix. - Appendix A's wrapper mechanism didn't state it only works by shadowing seid on $PATH, not via image ENTRYPOINT - the controller hardcodes the container Command/Args and never invokes the image's own entrypoint. - Appendix A's causal explanation for the AnteHandler-inflation outlier is corrected: two reviewers, independently and blinded from each other, converged on the same fix - the tracer's own in-process batch-export flush (size-triggered, deterministic by replay position) is a better-fitting candidate than GC/memIAVL, and "rules out the Jaeger collector stealing CPU" was overclaimed (only rules out random/wall-clock contention, not a deterministic in-process artifact both nodes share by construction). - The "still useful for phase attribution" carve-out is down-scoped: the same confound that corrupts the correlation also corrupts per-tx relative phase timing; only structural shape and aggregate/fleet-level timing survive. - Stale "eBPF (§8)" cross-reference in the adjacent runbooks/README.md index, missed by the in-document renumbering. Full ledger: .xreview/gas-vs-execution-time-runbook.md, Round 2. --- .agent/runbooks/README.md | 2 +- .../benchmarking-evm-gas-vs-execution-time.md | 22 ++++++++++++------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/.agent/runbooks/README.md b/.agent/runbooks/README.md index cdbd1e9..4e49e7e 100644 --- a/.agent/runbooks/README.md +++ b/.agent/runbooks/README.md @@ -32,7 +32,7 @@ Agents fetch this `README.md` to discover what's available, then `WebFetch` the | [`operating-archive-node-byov.md`](operating-archive-node-byov.md) | Required volume contents, PV/PVC spec, SeiNode/SeiNetwork spec, controller validation surface, and EBS-swap cutover sequence for archive nodes using the bring-your-own-volume (`dataVolume.import`) path. | Bringing up an archive node from a pre-populated EBS; swapping the underlying volume of an existing archive PV; debugging `ImportPVCReady=False`; receipt-store pruning concerns. | | [`migrating-validator-to-byo-secrets.md`](migrating-validator-to-byo-secrets.md) | Cutting a live validator from a legacy host onto the platform carrying its consensus identity via Secrets (`signingKey`/`nodeKey`): what migrates, SeiNetwork spec, controller validation surface, the stop-before-start double-sign discipline + layered equivocation defenses, cutover/rollback sequence, and dry-run gotchas. | Migrating an existing validator (e.g. arctic-1 node-19) off EC2 onto K8s; any cutover where a consensus key changes hosts; understanding the `replicas:1` CEL guard or the double-sign alerts. | | [`validating-flatkv-memiavl-parity-via-sharded-replay.md`](validating-flatkv-memiavl-parity-via-sharded-replay.md) | flatKV↔memIAVL storage-engine parity validation by differential historical replay: the flatKV+memIAVL replay-pair topology (same binary, same snapshot, blocks from a shared archive), the correctness gates (compare pair-not-archive; verify migration complete so flatKV reads are genuine; `historical_replay` build for pre-v6.5 txs), the seictl shadow comparator, result aggregation + Notion report, and the fan-out to 50+ shards. | Standing up a flatKV-vs-memIAVL correctness validation on harbor; driving a sharded replay campaign; debugging why a replay node is stuck or a comparison reads all-indeterminate/vacuous; understanding `migrate_evm` vs `memiavl_only`/`evm_migrated`/`flatkv_only` read routing. | -| [`benchmarking-evm-gas-vs-execution-time.md`](benchmarking-evm-gas-vs-execution-time.md) | Testing whether Sei's EVM gas schedule tracks real compute, for both (1) replaying a real historical block against restored state and (2) executing arbitrary bytecode against a specific historical state: the validated RPC-native measurement primitive (`debug_traceTransactionProfile` gives clean per-tx gas + execution time in one call, no eBPF required — replacing an earlier eBPF-first draft), the version-pinning discipline a replayer needs to avoid an app-hash-mismatch halt (including the `mock_chain_validation` build mode for testing unreleased code against real transaction load), why hardware parity with prod validators is automatic (`NodepoolForMode`, no engineer config needed), the scope limits confirmed by reading the actual code (only `Y=X-1` is supported for functionality 1; functionality 2's `debug_traceCall` path has a real execution-time gap), the confounds that invalidate a naive comparison (governance-mutable `SSTORE` gas, precompiles, whole-block-latency being the wrong timing proxy), the differential-loop bytecode-crafting technique for per-opcode isolation, and eBPF (§8) demoted to a fallback tier for needs beyond what's specified. | Designing a gas-vs-compute benchmark; replaying a historical block for timing; choosing between a release-pinned replayer and `mock_chain_validation`; deciding whether an opcode's gas cost is a fair compute proxy before trusting a result; understanding why `Y=X-1` is the supported case, not the general one; attaching any eBPF probe to a harbor pod only if the RPC primitive genuinely isn't enough. | +| [`benchmarking-evm-gas-vs-execution-time.md`](benchmarking-evm-gas-vs-execution-time.md) | Testing whether Sei's EVM gas schedule tracks real compute, for both (1) replaying a real historical block against restored state and (2) executing arbitrary bytecode against a specific historical state: the validated RPC-native measurement primitive (`debug_traceTransactionProfile` gives clean per-tx gas + execution time in one call, no eBPF required — replacing an earlier eBPF-first draft), the version-pinning discipline a replayer needs to avoid an app-hash-mismatch halt (including the `mock_chain_validation` build mode for testing unreleased code against real transaction load), why hardware parity with prod validators is automatic (`NodepoolForMode`, no engineer config needed), the scope limits confirmed by reading the actual code (only `Y=X-1` is supported for functionality 1; functionality 2's `debug_traceCall` path has a real execution-time gap), the confounds that invalidate a naive comparison (governance-mutable `SSTORE` gas, precompiles, whole-block-latency being the wrong timing proxy), the differential-loop bytecode-crafting technique for per-opcode isolation, subtracting intrinsic gas for a cleaner correlation, reading whole-block execution time from the existing log line with no new instrumentation, and eBPF (§9) demoted to a fallback tier for needs beyond what's specified. An appendix documents OpenTelemetry tracing as an investigated-and-invalidated avenue for this same measurement, with the real data that ruled it out. | Designing a gas-vs-compute benchmark; replaying a historical block for timing; choosing between a release-pinned replayer and `mock_chain_validation`; deciding whether an opcode's gas cost is a fair compute proxy before trusting a result; understanding why `Y=X-1` is the supported case, not the general one; attaching any eBPF probe to a harbor pod only if the RPC primitive genuinely isn't enough. | ## Adding a new runbook diff --git a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md index 2cab968..a4c9acd 100644 --- a/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md +++ b/.agent/runbooks/benchmarking-evm-gas-vs-execution-time.md @@ -101,7 +101,7 @@ Verified this session (illustrative, not a permanent number — re-check `origin **You do not need to request specific hardware.** `sei-k8s-controller` picks the Karpenter NodePool per node mode automatically: `internal/platform/platform.go`'s `NodepoolForMode()` — archive-mode nodes get `c.NodepoolArchive`, every other mode (including `replayer`) gets `c.NodepoolName` — and wires the matching toleration + required node affinity onto the pod spec (`internal/task/bootstrap_resources.go`, mirrored in `internal/noderesource/noderesource.go`). Harbor's and prod's `sei-k8s-controller/controller-config.yaml` are designed to carry identical `nodepoolName`/`nodepoolArchive`/`tolerationKey` values, and both clusters' NodePool manifests are designed to carry the same `r6i`/`r7i` instance-family, Nitro-hypervisor, on-demand requirements — by design, not independently re-verified here each time. Net: a replayer or archive `SeiNode` in your `eng-` namespace *should* land on the same instance class as its prod counterpart, with zero engineer-side configuration — confirm it with the step below rather than trusting the design intent, since drift between the two configs is exactly the failure mode that step catches. -**Optionally**, for a throwaway single-tenant experiment node, set the `sei.io/dedicated-node: "true"` annotation (see `sei-k8s-controller`'s dedicated-node feature) — not required for correctness, but removes one source of noisy-neighbor variance from your timing numbers. **Set it in the initial render, not on an already-running replayer.** Verified against the reconcile logic: a Running replayer's plan only rebuilds the pod on image drift or a sidecar-reapproval need (`internal/planner/replay.go`'s `buildRunningPlan`); an annotation-only change matches neither, so nothing replaces the pod, and with the StatefulSet's `OnDelete` strategy nothing else will either. The SeiNode object updates but the pod keeps running on its original, possibly-shared node indefinitely — a false sense of isolation that directly undermines step 7's noise-reduction discipline below. If you need to add it after the fact, delete the pod yourself to force a rebuild (§2's image-swap steps already establish this pattern). +**Optionally**, for a throwaway single-tenant experiment node, set the `sei.io/dedicated-node: "true"` annotation (see `sei-k8s-controller`'s dedicated-node feature) — not required for correctness, but removes one source of noisy-neighbor variance from your timing numbers. **Set it in the initial render, not on an already-running replayer.** Verified against the reconcile logic: a Running replayer's plan only rebuilds the pod on image drift or a sidecar-reapproval need (`internal/planner/replay.go`'s `buildRunningPlan`); an annotation-only change matches neither, so nothing replaces the pod, and with the StatefulSet's `OnDelete` strategy nothing else will either. The SeiNode object updates but the pod keeps running on its original, possibly-shared node indefinitely — a false sense of isolation that directly undermines §4 step 7's noise-reduction discipline below. If you need to add it after the fact, delete the pod yourself to force a rebuild (§2's image-swap steps already establish this pattern). **If a dedicated pod never schedules, nothing on the SeiNode object tells you.** This is documented behavior, not an oversight to work around: "a pod that can never be scheduled (no available single-tenant node) stays Pending with no SeiNode condition reflecting it" (`internal/noderesource/noderesource.go`). Check the pod's own `PodScheduled` condition and events directly if `.spec.nodeName` (the verify step below) comes back empty. @@ -153,7 +153,7 @@ Expect an `r6i.*`/`r7i.*` instance type and `nodepool=sei-node` (or `sei-archive Always re-check against §1's confounds (a divergence concentrated on `SSTORE`-heavy txs confirms the known governance gap, not a finding). -10. **Subtract intrinsic gas before correlating — a flat protocol charge is otherwise mixed into your independent variable.** `gasUsed` includes the intrinsic-gas floor (21000 base plus per-byte calldata costs, more for contract creation or an access list), deducted before the interpreter loop ever runs. It moves `gasUsed` without moving `executionNanos` at all, which is exactly why every minimum-gas transfer in a real sample clusters near zero on both axes regardless of what it actually touched. This is a client-side fix, not a sei-chain change: Sei calls the standard, unmodified go-ethereum `core.IntrinsicGas(data, accessList, authList, isContractCreation, true, true, true)` identically in three places (`app/app.go:2016`, `app/ante/evm_checktx.go:107`, `x/evm/ante/basic.go:51`), with the trailing fork-activation flags hardcoded `true`, not conditional on block height, so one formula covers all of Sei's history. Everything it needs — `input`, `to`, the access list — is already in `result.trace` or a plain `eth_getTransactionByHash`. Compute `executionGas = gasUsed - IntrinsicGas(...)` and correlate that against `executionNanos` instead of raw `gasUsed`. +10. **Subtract intrinsic gas before correlating — a flat protocol charge is otherwise mixed into your independent variable.** `gasUsed` includes the intrinsic-gas floor (21000 base plus per-byte calldata costs, more for contract creation or an access list), deducted before the interpreter loop ever runs. It moves `gasUsed` without moving `executionNanos` at all, which is exactly why every minimum-gas transfer in a real sample clusters near zero on both axes regardless of what it actually touched. This is a client-side fix, not a sei-chain change: Sei calls the standard, unmodified go-ethereum `core.IntrinsicGas(data, accessList, authList, isContractCreation, true, true, true)` identically in three places (`app/app.go:2016`, `app/ante/evm_checktx.go:107`, `x/evm/ante/basic.go:51`), with the trailing fork-activation flags hardcoded `true`, not conditional on block height, so one formula covers all of Sei's history. Everything it needs — `input`, `to`, the access list — is already in `result.trace` or a plain `eth_getTransactionByHash`. Compute `executionGas = gasUsed - IntrinsicGas(...)` and correlate that against `executionNanos` instead of raw `gasUsed`. One precision note: `gasUsed` on the receipt is net of any refund (SSTORE-clear, selfdestruct), while `IntrinsicGas` is charged up front and never refunded — `executionGas` is therefore execution gas net of refunds, not gross. For this step's purpose (removing the flat floor so low-gas transfers stop clustering at the origin) that's fine; keep it in mind if a specific transaction's refund is large relative to its execution gas. 11. **The bigger limitation no measurement technique here closes: `Y=X-1` blocks the question with the most repricing leverage for state opcodes.** For `SLOAD`/`SSTORE`, the decision-relevant variable is how access cost scales with state size / tree depth as the chain grows — which requires holding bytecode fixed while sweeping the base height, i.e. the general `Y` (`sei-k8s-controller`'s `internal/noderesource/noderesource.go`), bypassing any image-level `ENTRYPOINT` entirely. The wrapper has to replace the actual `seid` binary at the path the shell resolves it from (rename the real binary aside, put the wrapper script in its place) — an entrypoint-based wrapper would simply never run, `seid` would start without `--tracing`, and nothing would error. Exactly the kind of silent-failure trap this runbook exists to flag. **The result was a real, informative negative:** @@ -334,9 +336,13 @@ Sei ships real OTel tracing (Jaeger exporter, currently disabled by default), wi | Strictly serial (same 190 txs) | 0.690 | 0.574 | | Already-validated `debug_traceTransactionProfile` (§4) | — | 0.809 | -Serializing execution did not close the gap to the validated method; it made the clean-signal correlation worse. Both modes showed a recurring outlier: certain transactions' `AnteHandler` span durations inflated 100-155x relative to their `RunMsgs` span, on the same transactions in both datasets (confirmed by joining on identical `txHash` across the two nodes). Because strict serial execution has zero cross-goroutine overlap (confirmed directly from span start-times — back-to-back, ~4µs gaps), this ruled out OCC/mutex contention as the cause. Comparing the same outlier transactions across the two independent nodes (separate pods, separate Jaeger processes, separate GC timing) showed the inflation lands on the *same transactions*, at matching magnitudes, on both (Jaccard 0.92) — which rules out random per-process contention too, including the in-process Jaeger collector stealing CPU from `seid`. The remaining explanation both datasets are consistent with: a deterministic, per-height node-level stall — most likely GC mark-assist or a memIAVL commit/snapshot-flush cycle — that inflates whatever span happens to be open at that moment, landing more often on longer (higher-gas) transactions and inflating the correlation on exactly the end that makes it look better. +Serializing execution did not close the gap to the validated method; it made the clean-signal correlation worse. Both modes showed a recurring outlier: certain transactions' `AnteHandler` span durations inflated 100-155x relative to their `RunMsgs` span, on the same transactions in both datasets (confirmed by joining on identical `txHash` across the two nodes). Because strict serial execution has zero cross-goroutine overlap (confirmed directly from span start-times — back-to-back, ~4µs gaps), this ruled out OCC/mutex contention as the cause. + +**What the cross-node match actually rules out, and what it doesn't — this took a second review pass to get right.** Comparing the same outlier transactions across the two independent nodes (separate pods, separate Jaeger processes, independent GC pacing) showed the inflation lands on the *same transactions*, at matching magnitudes, on both (Jaccard 0.92). That's real evidence, but it only rules out effects that are **random or wall-clock-driven** — a noisy neighbor on the shared node, an independent GC pause whose timing depends on each process's own heap-growth pacing, a shared rate limit hit at a similar wall-clock moment. Two replayers restoring from the same snapshot drift out of height-lockstep almost immediately, so a wall-clock-correlated disturbance would land on *different* transactions on the two nodes (different heights at the same clock time) — the high Jaccard correctly excludes that whole class. It does **not** rule out an effect that's **locked to logical position in the replay** (this height, this cumulative span count) rather than to wall-clock time, because both nodes carry the identical instrumentation and replay the identical chain — a logical-position-triggered artifact fires at the same txHash on both nodes *by construction*. That's a different failure mode than "random per-process contention," and the first draft of this appendix conflated the two. + +**The best-fitting candidate on reflection is the tracer's own in-process batch-export flush, not a node-level stall.** Sei's Jaeger exporter batches spans (`sei-cosmos/utils/tracing/tracer.go`'s `WithBatcher`, default max batch size 512) and flushes on an async goroutine sharing the same pod CPU budget as `seid` — a size-triggered flush fires once a fixed number of spans have been produced, which is a function of replay position, not wall clock, so it lands at the same point in two nodes replaying the same chain. That also fits the specific shape of the symptom better than the alternatives: it explains why the inflation concentrates on `AnteHandler` specifically (the shortest span per tx — a fixed-duration stall shows up as a huge *ratio* on a sub-millisecond span, and barely moves the ratio on a 20ms `RunMsgs` span), and it survives serialization because the exporter runs independently of the OCC scheduler. GC mark-assist and memIAVL commit/snapshot-flush (the original candidates here) are weaker fits on reflection: GC pacing is per-process-independent and would be expected to degrade cross-node correlation well below 0.92, and commit/snapshot-flush both run in the inter-block gap after all tx execution for a height, not mid-block during an open `AnteHandler` span. **The disambiguating experiment, not yet run:** re-replay with the exporter's batch size set to 1 vs. a very large number (or pointed at a discard sink instead of a live collector); if the outlier set shifts with batch size, that confirms the exporter as the cause rather than the chain's own execution. -**Verdict: not fit for absolute per-tx gas-vs-execution-time correlation, in either execution mode.** The already-validated `debug_traceTransactionProfile` isolates a transaction's re-execution from the live block-delivery pipeline entirely — no concurrent scheduler, no GC/commit cycle landing mid-measurement — which is the structural reason it measures closer to true compute cost. OTel tracing remains genuinely useful for a narrower, different question: structural/relative phase attribution within a block or transaction (what fraction of the time went to ante-handling vs. message execution vs. begin/end-blockers), not for this correlation. +**Verdict: not fit for absolute per-tx gas-vs-execution-time correlation, in either execution mode — this conclusion holds regardless of which of the above turns out to be the exact trigger.** The already-validated `debug_traceTransactionProfile` isolates a transaction's re-execution from the live block-delivery pipeline entirely — no concurrent scheduler, no exporter or GC cycle sharing the same process, landing mid-measurement — which is the structural reason it measures closer to true compute cost. **The narrower fallback use — structural/relative phase attribution — needs its own caveat, not a blanket carve-out.** The tree *shape* (which phases exist, how they nest) is unaffected by any of this and remains genuinely useful. But per-transaction or per-block relative *timing* (what fraction of this tx's time went to ante-handling vs. message execution) is exactly what the 100-155x outliers already show failing — a stall that inflates whichever span happens to be open corrupts phase-timing attribution on precisely the transactions it hits. Treat relative phase timing as usable only in aggregate, across many blocks, after excluding flagged outliers — never at single-tx or single-block granularity. ---