SDK rpc proxy: a relayed node error does not trip the breaker

[feruzm]

Closes #1628.

In production a large share of the proxy's non-200 answers are hivemind application errors relayed as 502 with X-Ssr-Cache: RPCERROR (Tag <x> does not exist for crawler-made feed URLs, 5 of 30 real feed URLs replayed on one origin). callRPC counted each as a proxy miss, so three in a row opened the breaker for 10s and every eligible read in that window skipped a healthy proxy (thousands of skipped per replica per hour).

• New fallback reason rpcerror: a 502 tagged RPCERROR. The read still falls back to the node loop, so the caller gets the node's own answer as before; but the proxy reached a node and relayed it, so the relay resets the consecutive-miss count like a served call instead of advancing it. A bare 502/504, transport, parse and validator misses count as before.
• rpcProxyStats.fallbackByReason.rpcerror and the web's [rpc-proxy] line carry the new bucket, so real proxy failures and relayed node errors can be told apart in the logs.
• Specs: four relays in a row never open the breaker; a relay clears a count a real miss started; bare 502s still open it. Web spec updated for the line format.

The SDK source change reaches the web bundle through the dist rebuild, which is label-driven.

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of relayed RPC errors during server-side requests.
  • RPC error responses are tracked separately from proxy failures.
  • These responses no longer unnecessarily trigger proxy failover protection, improving request routing reliability.
  • Fallback status reports now include RPC error counts for clearer diagnostics.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

SDK RPC proxy: relayed node errors no longer trip the breaker

🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes

AI Description

• Adds rpcerror fallback reason for proxy 502 responses tagged X-Ssr-Cache: RPCERROR,
 indicating the proxy relayed a genuine node error.
• These relayed errors now reset the consecutive-miss counter (like a served call) instead of
 counting toward the breaker's failure threshold.
• Exposes rpcerror in rpcProxyStats.fallbackByReason and in the web [rpc-proxy] log line so
 real proxy failures are distinguishable from relayed node errors.
• Adds specs covering repeated relays, a relay resetting an in-progress miss streak, and confirming
 bare 502s still trip the breaker; updates web spec for the new log format.

Diagram

graph TD
  A["callRPC"] --> B{"Proxy response"}
  B -->|"502 + RPCERROR tag"| C["ProxyMiss: rpcerror"]
  B -->|"other error/timeout"| D["ProxyMiss: status/timeout/transport/parse/validate"]
  C --> E["Reset consecutive misses"]
  D --> F{"Threshold reached?"}
  F -->|"yes"| G["Open breaker (skip proxy)"]
  F -->|"no"| H["Increment counter"]
  C --> I["Fallback to node loop"] --> J["rpcProxyStats.fallbackByReason"] --> K["sdk-init.ts log line"]
  D --> I

Loading

High-Level Assessment

Detecting the relayed error via the X-Ssr-Cache: RPCERROR header and resetting (rather than ignoring or advancing) the miss counter is the simplest and lowest-risk fix given the existing ProxyMiss/reason infrastructure. Alternatives like inspecting the response body for hivemind error patterns or adding a separate breaker for 'soft' errors would add complexity without clear benefit, since the header already reliably signals a proxy-healthy relay.

Files changed (4) +67 / -9

Enhancement (1) +1 / -1

sdk-init.tsLog new rpcerror fallback count in proxy report line +1/-1

Log new rpcerror fallback count in proxy report line

• Updates the periodic '[rpc-proxy]' log line to include the new rpcerror fallback reason count alongside status, timeout, transport, validate and parse.

apps/web/src/core/sdk-init.ts

Bug fix (1) +17 / -4

call.tsDistinguish relayed node errors from real proxy misses +17/-4

Distinguish relayed node errors from real proxy misses

• Adds a new 'rpcerror' ProxyMissReason detected when the proxy responds with 502 and header X-Ssr-Cache: RPCERROR, indicating the proxy successfully relayed a node's own error. Instead of incrementing the consecutive-miss counter that can open the breaker, this reason resets it to zero, treating it like a healthy served call while the read still falls back to the node loop.

packages/sdk/src/hive-tx/helpers/call.ts

Tests (2) +49 / -4

server-rpc-proxy.spec.tsAdd specs for relayed-error breaker behavior +44/-0

Add specs for relayed-error breaker behavior

• Adds a test verifying that four consecutive relayed RPCERROR responses never open the breaker, that a relay clears an in-progress miss streak, and that bare 502 status misses still count toward and open the breaker.

packages/sdk/src/hive-tx/helpers/server-rpc-proxy.spec.ts

sdk-init-proxy.spec.tsUpdate web spec for new log line format with rpcerror +5/-4

Update web spec for new log line format with rpcerror

• Updates the mocked stats object and expected log strings to include the new rpcerror field in fallbackByReason and the rendered log line.

apps/web/src/specs/core/sdk-init-proxy.spec.ts

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 652abc26-5069-4fde-8105-5fd9fa8a4d97

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: cfbf9b04-492b-49d7-ac65-ad8b24ddb83e

📥 Commits

Reviewing files that changed from the base of the PR and between acf6685 and b7a7358.

📒 Files selected for processing (4)

• apps/web/src/core/sdk-init.ts
• apps/web/src/specs/core/sdk-init-proxy.spec.ts
• packages/sdk/src/hive-tx/helpers/call.ts
• packages/sdk/src/hive-tx/helpers/server-rpc-proxy.spec.ts

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The SDK now classifies tagged RPCERROR 502 responses separately, resets the proxy breaker for those responses, and reports their count. Tests cover breaker behavior and periodic statistics output.

Changes

RPCERROR proxy handling

Layer / File(s)	Summary
RPCERROR classification and breaker behavior packages/sdk/src/hive-tx/helpers/call.ts, packages/sdk/src/hive-tx/helpers/server-rpc-proxy.spec.ts	Tagged RPCERROR 502 responses use a separate fallback category and reset consecutive proxy misses. Untagged 502 responses continue to advance the breaker.
Fallback statistics reporting apps/web/src/core/sdk-init.ts, apps/web/src/specs/core/sdk-init-proxy.spec.ts	Periodic reports include the RPCERROR fallback count and updated fallback totals.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to b7a73

The change prevents relayed node errors from unnecessarily tripping the proxy breaker while preserving breaker behavior for actual proxy failures; no actionable merge-blocking risk remains at the current head beyond normal checks.

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant callRPC
  participant SSRProxy
  participant NodeLoop
  Caller->>callRPC: Request RPC data
  callRPC->>SSRProxy: Send proxy request
  SSRProxy-->>callRPC: 502 with X-Ssr-Cache: RPCERROR
  callRPC->>callRPC: Record rpcerror and reset breaker misses
  callRPC->>NodeLoop: Fall back to node requests
  NodeLoop-->>Caller: Return node response

Loading

Poem

A rabbit watched the proxy glow,
“RPCERRORs need not trip the flow.”
The counter hops into the chart,
The breaker gets a gentler start.
Thump, thump—clean fallbacks go!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: relayed node errors no longer trip the RPC proxy breaker.
Linked Issues check	✅ Passed	The changes meet issue #1628 by excluding tagged RPCERROR responses from breaker failures while preserving fallback behavior and genuine failure counting.
Out of Scope Changes check	✅ Passed	All code and test changes directly support RPCERROR classification, breaker behavior, fallback statistics, and log reporting.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 4 files.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/rpc-proxy-rpcerror-breaker

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. SDK dist not rebuilt ✓ Resolved 🐞 Bug ≡ Correctness

Description

@ecency/sdk exports only dist/* entrypoints, but the current packages/sdk/dist/* still
implements the old proxy miss behavior (all non-200s are status), so RPCERROR relays will continue
to advance/open the breaker in real consumers despite the src change.

Code

packages/sdk/src/hive-tx/helpers/call.ts[R138-139]

+      const relayed = res.status === 502 && (res.headers.get('x-ssr-cache') ?? '').toUpperCase() === 'RPCERROR'
+      throw new ProxyMiss(relayed ? 'rpcerror' : 'status', relayed ? 'proxy relayed a node error' : `proxy answered ${res.status}`)

Evidence

The PR changes add rpcerror handling in the TypeScript source, but the SDK’s package.json
declares that only dist is published/used via exports; the existing dist code still initializes
fallbackByReason without rpcerror and throws only "status" for non-200 proxy responses, so the
runtime behavior remains unchanged for consumers.

packages/sdk/src/hive-tx/helpers/call.ts[53-155]
packages/sdk/src/hive-tx/helpers/call.ts[1460-1485]
packages/sdk/package.json[17-38]
packages/sdk/dist/node/hive.mjs[1-8]
packages/sdk/dist/node/hive.cjs[1-1]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The repo’s SDK package (`@ecency/sdk`) is consumed via its `dist/` artifacts (per `package.json` exports/files), but the `dist/` outputs in this PR branch still contain the pre-change behavior (no `rpcerror` reason; non-200 always throws `status`; breaker logic unchanged). This means the intended production fix won’t ship to any consumer that imports `@ecency/sdk`.
## Issue Context
- `packages/sdk/src/hive-tx/helpers/call.ts` was updated to classify `502 + X-Ssr-Cache: RPCERROR` as `rpcerror` and to reset the breaker miss count.
- However, the compiled artifacts in `packages/sdk/dist/node/*` (and likely `dist/browser/*`) still show the old logic.
## Fix Focus Areas
- packages/sdk/package.json[17-38]
- packages/sdk/dist/node/hive.mjs[1-1]
- packages/sdk/dist/node/hive.cjs[1-1]
## What to do
1. Rebuild the SDK (`pnpm --filter @ecency/sdk build` or equivalent for your workspace).
2. Commit the updated `packages/sdk/dist/**` outputs.
3. Verify `dist` contains:
 - `fallbackByReason` initialized with `rpcerror`
 - proxy non-200 path classifies relayed RPCERROR and the breaker logic resets consecutive misses on that reason.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 84 rules

✅ Skills: 6 invoked

  add-feature

  add-query

  add-sdk-mutation

  add-test

  code-review

  debug

Review mode: ⚖️ Balanced: This changes runtime SDK breaker classification and fallback state across proxy response handling, with web reporting and tests; it is meaningful behavioral logic but not broad or defect-dense enough to justify redundant extended review.

Tip of the day

💡 Did you know, you can tweak Display preferences with a live preview to see your comment before it ships

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

1. SDK dist not rebuilt 🐞 Bug ≡ Correctness

Description

@ecency/sdk exports only dist/* entrypoints, but the current packages/sdk/dist/* still
implements the old proxy miss behavior (all non-200s are status), so RPCERROR relays will continue
to advance/open the breaker in real consumers despite the src change.

Code

packages/sdk/src/hive-tx/helpers/call.ts[R138-139]

+      const relayed = res.status === 502 && (res.headers.get('x-ssr-cache') ?? '').toUpperCase() === 'RPCERROR'
+      throw new ProxyMiss(relayed ? 'rpcerror' : 'status', relayed ? 'proxy relayed a node error' : `proxy answered ${res.status}`)

Evidence

The PR changes add rpcerror handling in the TypeScript source, but the SDK’s package.json
declares that only dist is published/used via exports; the existing dist code still initializes
fallbackByReason without rpcerror and throws only "status" for non-200 proxy responses, so the
runtime behavior remains unchanged for consumers.

packages/sdk/src/hive-tx/helpers/call.ts[53-155]
packages/sdk/src/hive-tx/helpers/call.ts[1460-1485]
packages/sdk/package.json[17-38]
packages/sdk/dist/node/hive.mjs[1-8]
packages/sdk/dist/node/hive.cjs[1-1]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The repo’s SDK package (`@ecency/sdk`) is consumed via its `dist/` artifacts (per `package.json` exports/files), but the `dist/` outputs in this PR branch still contain the pre-change behavior (no `rpcerror` reason; non-200 always throws `status`; breaker logic unchanged). This means the intended production fix won’t ship to any consumer that imports `@ecency/sdk`.

## Issue Context
- `packages/sdk/src/hive-tx/helpers/call.ts` was updated to classify `502 + X-Ssr-Cache: RPCERROR` as `rpcerror` and to reset the breaker miss count.
- However, the compiled artifacts in `packages/sdk/dist/node/*` (and likely `dist/browser/*`) still show the old logic.

## Fix Focus Areas
- packages/sdk/package.json[17-38]
- packages/sdk/dist/node/hive.mjs[1-1]
- packages/sdk/dist/node/hive.cjs[1-1]

## What to do
1. Rebuild the SDK (`pnpm --filter @ecency/sdk build` or equivalent for your workspace).
2. Commit the updated `packages/sdk/dist/**` outputs.
3. Verify `dist` contains:
  - `fallbackByReason` initialized with `rpcerror`
  - proxy non-200 path classifies relayed RPCERROR and the breaker logic resets consecutive misses on that reason.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Tickets: 🎫 SDK rpc proxy: an RPCERROR relay should not trip the breaker

Review mode: ⚖️ Balanced: This is a behavioral change to SDK proxy failure handling and breaker state, with observable stats and web integration; it is localized but carries real runtime risk, so a complete single-pass review is appropriate.

Tip of the day

💡 Did you know, you can tweak Display preferences with a live preview to see your comment before it ships

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗