Skip to content

docs: note public testnet RPC method differences across follow endpoints - #373

Open
kutluhaneth46 wants to merge 1 commit into
circlefin:mainfrom
kutluhaneth46:cursor/docs-rpc-endpoint-capabilities-88c1
Open

docs: note public testnet RPC method differences across follow endpoints#373
kutluhaneth46 wants to merge 1 commit into
circlefin:mainfrom
kutluhaneth46:cursor/docs-rpc-endpoint-capabilities-88c1

Conversation

@kutluhaneth46

Copy link
Copy Markdown
Contributor

Summary

  • Clarifies that the public testnet --follow.endpoint URLs in docs/running-an-arc-node.md are interchangeable for follow sync, but not for application JSON-RPC method support.
  • Adds a compact capability matrix for eth_getProof / eth_createAccessList and a note about unstable -32601 vs -32014 answers on some hosts.

Docs-only; does not change provider gateways. Distinct from #222 / #299 (rate limits / gas caps / fallback config).

Addresses #371.

Test plan

  • Diff review of the callout next to the follow-mode endpoint list
  • Maintainer confirm matrix still matches live fleet before merge

@osr21

osr21 commented Sep 9, 2026

Copy link
Copy Markdown

Thanks for turning this around so quickly, and for splitting follow-sync from application RPC — that was my main concern on #371 and this resolves it cleanly.

Discharging the open test-plan box

Your checklist has "maintainer confirm matrix still matches live fleet before merge" unchecked. Re-ran it just now, 2026-09-09 17:46 UTC, 5 samples per method per host:

rpc.testnet.arc.io        getProof: -32601 -32601 -32014 -32014 -32014 | createAccessList: -32601 ×3
rpc.drpc.testnet.arc.io   getProof: OK OK OK OK OK                     | createAccessList: OK ×3
rpc.quicknode...arc.io    getProof: -32601 -32014 -32601 -32601 -32014 | createAccessList: -32601 ×3
rpc.blockdaemon...arc.io  getProof: OK OK OK OK OK                     | createAccessList: -32003 ×3

Every row in your table matches. No drift since #371 was filed.

I also verified the "interchangeable for follow sync" claim rather than assuming it: the follow client's public-RPC surface is eth_getBlockByNumber, eth_call, eth_chainId, net_listening, and all four are served by all four hosts both individually and in a single batch. The engine_* family goes over the auth socket. Worth noting the follow client doesn't batch anyway — the serde_json::json!([...]) sites in crates/eth-engine/src/rpc/ethereum_rpc.rs are params arrays, not batch envelopes — so blockdaemon's batch filtering cannot reach it. The claim is safe as written.

One thing I'd change before merge: the probe advice is unsound

treat capability probes as sticky only after a stable -32601, not after a single sample

I think this bakes in a recipe that can't be followed, and it's my fault for not being clearer in #371. There is no "stable -32601" state to wait for. Same host, same request, three sittings:

when rpc.testnet.arc.io rpc.quicknode.testnet.arc.io
2026-09-08 (n=10) -32014, 2×-32601 -32601, 4×-32014
2026-09-09 14:55Z (n=5) -32601, 2×-32014 -32601, 3×-32014
2026-09-09 17:46Z (n=5) -32601, 3×-32014 -32601, 2×-32014

The mix reshuffles every time and neither code ever settles. A client told to wait for a stable -32601 before caching "unsupported" will keep probing indefinitely against a host that is permanently unable to serve the method — the exact opposite of the intended outcome.

The guidance the data actually supports is: don't branch on the code at all. Either code means "this host will not serve this method"; treat any error from a capability probe as unsupported and fail over to a host that does. Suggested replacement:

Some hosts return -32601 (method unsupported) for one call and -32014 (data unavailable) for the next, for the same request. Do not branch on the code — treat any error from a capability probe as "unsupported on this host" and fail over.

Smaller points

quicknode is missing from the table. rpc.quicknode.testnet.arc.io is a live public endpoint with the same divergence (data above: getProof unsupported, createAccessList -32601 3/3). Since the note is framed for application clients rather than follow operators, and says "measured against the public fleet", a reader may well be pointed at it. Either add the row or scope the sentence to "the follow endpoints listed above".

"rejects whole batches" reads stronger than the behaviour. Blockdaemon handles batches fine — the four-method follow batch above succeeded there. What it does is reject a batch containing a filtered method, discarding the unrelated calls beside it with a single top-level error ("id": null) instead of a per-item error inside the array. Maybe: "filtered (-32003); a batch containing it is rejected in full, losing unrelated calls".

The WS suffix deserves a clause here. The note now says the endpoints are interchangeable for follow sync, and sits directly above a list where blockdaemon uniquely needs wss=rpc.blockdaemon.testnet.arc.io/websocket while the other two take the bare host. A reader acting on "interchangeable" by swapping WS URLs pattern-wise will get it wrong. Half a sentence — "note the differing WebSocket paths above" — closes it.

Date-stamp the matrix. This is provider-side state that can change without any commit to this repo, and the unchecked box in your test plan is effectively an unbounded recurring ask on maintainers. An "observed YYYY-MM-DD" line plus the one-line repro makes it falsifiable by any reader instead:

curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"jsonrpc":"2.0","id":1,"method":"eth_getProof","params":["0x3600000000000000000000000000000000000000",["0x0"],"latest"]}' \
  https://rpc.testnet.arc.io

Nit: the hunk adds two consecutive blank lines before ### Verify operation.


Disclosure: I'm an external community contributor, not affiliated with Circle, with no write access to this repository. Advisory review only, not an approval — all measurements above are from my own probing of the public endpoints and should be confirmed independently.

kutluhaneth46 added a commit to kutluhaneth46/arc-node that referenced this pull request Sep 9, 2026
Address review on circlefin#373/circlefin#371: clarify follow sync is unaffected, fix
probe advice (do not wait for stable -32601), add quicknode + date stamp,
soften blockdaemon batch wording, note differing WS paths, and drop the
extra blank line.
@kutluhaneth46

Copy link
Copy Markdown
Contributor Author

Thanks @osr21 — excellent catch on the probe advice and the follow-vs-app split.

Updated the note to:

  • state plainly that follow sync is unaffected (and name the four methods)
  • replace the "stable `-32601`" guidance with "treat any probe error as unsupported and fail over"
  • add the quicknode row + observed 2026-09-09 date stamp + one-line repro
  • soften the blockdaemon batch wording
  • mention the differing WebSocket paths
  • drop the extra blank line

Matrix re-check appreciated — ready for another pass whenever you have a minute.

@osr21

osr21 commented Sep 9, 2026

Copy link
Copy Markdown

Revision looks good — the probe guidance, quicknode row, date stamp, batch wording, and WebSocket clause all read correctly now. Two things before you consider this final, and the first is me correcting my own review.

I got the batching claim wrong

In my last comment I wrote that "the follow client doesn't batch — the serde_json::json!([...]) sites in crates/eth-engine/src/rpc/ethereum_rpc.rs are params arrays, not batch envelopes."

That was the wrong file. EthereumRPC is the general execution-client wrapper. The actual follow client is crates/malachite-app/src/rpc_sync/client.rs, and fetch_blocks_batch builds two genuine JSON-RPC batches per height range and fires them in parallel via tokio::join!. So follow does batch. My conclusion held, but the reasoning behind it didn't, and the blockdaemon batch caveat is more load-bearing for follow than I implied, not less.

The four-method list is missing two methods

The note now says the follow client "only needs eth_getBlockByNumber, eth_call, eth_chainId, and net_listening". That's the list I gave you in #371 and it's incomplete. The follow path also uses:

  1. arc_getCertificate — batched one per height in fetch_blocks_batch, right alongside eth_getBlockByNumber. Non-standard, Arc-specific, and absolutely required for follow.
  2. eth_subscribe("newHeads") — over the wss= URL in rpc_sync/ws_subscription.rs (alloy's subscribe_blocks()), for peer height tracking, with unbounded retry/backoff.

I verified all three follow endpoints serve eth_subscribe("newHeads") over WSS. Your WebSocket clause is also empirically correct and worth keeping — the bare host genuinely fails:

wss://rpc.blockdaemon.testnet.arc.io/websocket   OK  subId=0x8407e45be509
wss://rpc.blockdaemon.testnet.arc.io  (bare)     connection failure

The bigger one: blockdaemon cannot serve follow catch-up below ~41.86M

Adding arc_getCertificate to the list matters because the hosts do not agree on it, which makes "those work on every public host below" true only for recent heights.

arc_getCertificate            h=1     h=1e3   h=1e6   h=1e7   h=3e7   h=5e7
rpc.testnet.arc.io            OK      OK      OK      OK      OK      OK
rpc.drpc.testnet.arc.io       OK      OK      OK      OK      OK      OK
rpc.blockdaemon...arc.io      MISS    MISS    MISS    MISS    MISS    OK

blockdaemon returns -32004 "Certificate not found" for old heights. This is not block pruning — eth_getBlockByNumber at height 1,000,000 returns a full block on that same host. It has the blocks and not the certificates.

Binary-searching the cutoff gives a single sharp boundary, not sporadic gaps:

MISS at 41,863,834   |   OK at 41,863,835      (head 61,306,927, measured 2026-09-09 23:3x UTC)

So roughly the last 19.44M blocks (~112 days at 500 ms), though one measurement can't distinguish a rolling window from a fixed restore point.

Why this is operationally sharp rather than cosmetic: in send_batch_request, any per-item JSON-RPC error maps to Err for the whole batch, and fetch_range propagates it. One missing certificate in a range fails the entire range fetch, not just that height. A node catching up from genesis — or from any height below the cutoff — against --follow.endpoint https://rpc.blockdaemon.testnet.arc.io will fail there, and that is exactly the new-operator path this document is for.

Suggested addition after the table:

Follow sync also uses arc_getCertificate (batched per height) and eth_subscribe("newHeads") over the wss= URL. rpc.blockdaemon.testnet.arc.io serves certificates only for recent history (observed: nothing below block ~41,863,835 on 2026-09-09) while still serving the blocks themselves, so use rpc.testnet.arc.io or rpc.drpc.testnet.arc.io when catching up from genesis or from deep history.

Repro:

curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"jsonrpc":"2.0","id":1,"method":"arc_getCertificate","params":[1000000]}' \
  https://rpc.blockdaemon.testnet.arc.io
# {"jsonrpc":"2.0","id":1,"error":{"code":-32004,"message":"Certificate not found"}}

Worth a maintainer's eye on whether that retention boundary is intended for a documented follow endpoint, since it may be a provider configuration issue rather than a docs issue. Either way the doc shouldn't imply the three are interchangeable for catch-up.

Minor, in support of your current wording

txpool_status / txpool_inspect also exist on EthereumRPC, but they have no production consumer — only eth-engine/tests/integration.rs and crates/test/checks/src/mempool.rs — and no public endpoint serves them (all three return -32601). Correctly excluded from the follow list; noting it so the omission doesn't look accidental to a later reader.


Disclosure: I'm an external community contributor, not affiliated with Circle, with no write access to this repository. Advisory review only, not an approval — all measurements are from my own probing of the public endpoints on 2026-09-09 and should be confirmed independently.

@kutluhaneth46

Copy link
Copy Markdown
Contributor Author

Thanks @osr21 — especially for correcting the batching path (`rpc_sync/client.rs`) and catching the incomplete method list.

Updated the note to:

  • list the real follow surface: `eth_getBlockByNumber`, `arc_getCertificate`, `eth_call`, `eth_chainId`, `net_listening`, `eth_subscribe("newHeads")`
  • warn that blockdaemon only has recent certificates (~41,863,835+ as of 2026-09-09), so genesis/deep catch-up should use `rpc.testnet.arc.io` or `rpc.drpc.testnet.arc.io`
  • keep the app-client `eth_getProof` / `eth_createAccessList` matrix as a separate concern

Ready for another pass whenever you have a minute.

kutluhaneth46 added a commit to kutluhaneth46/arc-node that referenced this pull request Sep 10, 2026
Address review on circlefin#373/circlefin#371: clarify follow sync is unaffected, fix
probe advice (do not wait for stable -32601), add quicknode + date stamp,
soften blockdaemon batch wording, note differing WS paths, and drop the
extra blank line.
@kutluhaneth46
kutluhaneth46 force-pushed the cursor/docs-rpc-endpoint-capabilities-88c1 branch from d72c8f6 to d8cabdd Compare September 10, 2026 11:54
@osr21

osr21 commented Sep 10, 2026

Copy link
Copy Markdown

Re-measured today against the live fleet. The note reads correctly now — the method list matches what rpc_sync/ actually calls, and the batch-failure sentence is right. And the question I left open last time is now settled, which changes one word in your wording.

The certificate floor is fixed, not a rolling window

Last time I said "one measurement can't distinguish a rolling window from a fixed restore point." A second measurement 23 hours later can:

                        2026-09-09 23:3x UTC     2026-09-10 22:47 UTC
blockdaemon head        61,306,927               61,469,538   (+162,611)
certificate floor       41,863,835               41,863,835   (+0)

The head advanced 162,611 blocks; the floor did not move by one. A rolling retention window would have carried the floor forward by the same amount. It is a fixed restore point.

The boundary is also exact and reproducible, not approximate — 5 samples each side, no flapping, and the OK side returns the same block_hash both days:

h=41,863,834  ->  -32004 "Certificate not found"   (5/5)
h=41,863,835  ->  OK, block_hash 0x466b5802…       (5/5)

Controls today, so this is not a general outage: rpc.testnet.arc.io and rpc.drpc.testnet.arc.io both serve arc_getCertificate at heights 1, 1,000,000 and 41,863,834. blockdaemon misses all three while still returning the block at 41,863,828 — certificates specifically, as before.

What that means for the wording

The note currently says blockdaemon "serves certificates only for recent history". That phrasing implies a sliding window, which has two consequences for a reader: they will assume the quoted number decays, and they will assume the doc rots. Neither is true. Concretely, a fixed floor means the certificate range widens over time as the head advances, but genesis catch-up on that host never becomes possible.

The floor has a wall-clock anchor, which is more durable than a block number:

block 41,863,835  ts=1778605724  ->  2026-05-12T17:08:44Z

So blockdaemon's certificate history begins 2026-05-12. Suggested tightening:

rpc.blockdaemon.testnet.arc.io has no certificates below block 41,863,835 (2026-05-12), while still serving the blocks themselves. This is a fixed floor, not a rolling window — it was unchanged across 2026-09-09 and 2026-09-10 while the head advanced ~162k blocks — so use rpc.testnet.arc.io or rpc.drpc.testnet.arc.io when catching up from genesis or from any height below it.

That also lets you drop the ~: the boundary is a single sharp height, not an estimate. Incidentally my "~112 days" figure last time was slightly off — measured block rate is ~516 ms, not 500 ms, so the floor is ~121 days back rather than 112. The date anchor avoids that arithmetic entirely.

Minor

  • The capability matrix lists rpc.quicknode.testnet.arc.io, which is not one of the three --follow.endpoint hosts in the command above it. That is correct for an app-client matrix, but a reader arriving from the follow list may pause on it — one clause noting the matrix covers the wider public fleet, not just the follow endpoints, would remove the ambiguity.
  • "Note the differing WebSocket paths in the command above" checks out — blockdaemon is the only one carrying /websocket in that block, and the bare host genuinely fails to connect.

Everything else I can verify holds. The retention boundary being a deliberate restore point rather than a transient gap is probably still worth a maintainer's eye, since it is a documented follow endpoint that cannot serve the from-genesis path this page describes.

Disclosure: I'm an external community contributor, not affiliated with Circle, with no write access to this repository. Advisory review only, not an approval. All measurements are my own probing of the public endpoints on 2026-09-09 and 2026-09-10 and should be confirmed independently before merge.

@kutluhaneth46

Copy link
Copy Markdown
Contributor Author

Thanks @osr21 — especially for settling the fixed-floor vs rolling-window question with the second measurement.

Updated the note to:

  • drop "recent history" / ~ and state the fixed floor at block 41,863,835 (2026-05-12)
  • note it was unchanged across 2026-09-09 and 2026-09-10 while the head advanced ~162k blocks
  • clarify that the capability matrix covers the wider public fleet, not only the three follow endpoints

Ready for another look whenever you have a minute.

@kutluhaneth46
kutluhaneth46 force-pushed the cursor/docs-rpc-endpoint-capabilities-88c1 branch from 43f96ae to ac565f2 Compare September 10, 2026 23:42
@kutluhaneth46 kutluhaneth46 reopened this Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants