An abandoned request still crashes the server: #2317 did not cover every WorkerDbAbortError path

[ateles-agent]

Summary

The hosted instance crash-looped on 2026-09-23 with the same unhandled WorkerDbAbortError that #2316 describes, on a build that contains #2317, the fix for #2316. So #2317 closed one path and at least one more remains.

Evidence

• Deployed image: neotoma-markmhendrickson v53, built from a80340c66 (main, 2026-09-22). git merge-base --is-ancestor ba314ad07 a80340c66 → true, so fix(db): an abandoned request must not kill the server (#2316) #2317 is in the build.
• Crash, from the app's own output:

[neotoma] exit code=1 uptime=21.6s
file:///app/dist/repositories/worker/worker_file_database.js:545
    this.abandon(new WorkerDbAbortError("DB request aborted by caller"));
WorkerDbAbortError: DB request aborted by caller
    at EventTarget.onAbort (worker_file_database.js:545:34)
    at AbortController.abort (node:internal/abort_controller:392:5)
    at ServerResponse.onClose (middleware/db_abort_context.js:32:28)
    at Socket.onServerResponseClose (node:_http_server:277:5)
Node.js v20.20.2

• Immediately before each exit the log shows DB reader pool recovered after 1714ms of saturation, and requests taking ~2.8s.
• Fly event log: repeated exit_code=134 roughly 20–30s after each start, until Fly stopped restarting. Subsequent manual starts were refused with resource_exhausted: rate limit exceeded.

Mechanism (as far as the logs show)

Load saturates the reader pool; callers time out and close their connections; db_abort_context aborts the in-flight DB request; the abort surfaces as an exception nothing catches, and the process exits. On restart the same backlog of retrying clients arrives at once, so the loop reproduces within seconds. A client disconnect is an ordinary event and must not be able to end the process.

What to check

• Every path by which abandon(new WorkerDbAbortError(...)) can reject a promise that no caller awaits — the fix(db): an abandoned request must not kill the server (#2316) #2317 fix covered one; the stack above is the onAbort listener path in worker_file_database.
• Whether the rejection escapes because the request that owned it has already finished or detached, leaving the promise orphaned.
• A regression test that reproduces a mid-query client disconnect under reader-pool saturation, and fails on a80340c66. A test that passes on the crashing build is not coverage for this.

Related: #2316 (same symptom, closed), #2317 (its fix), #2322 (reader-pool exhaustion), #2339 (interrupting abandoned reads).

Swarm specification

This section is maintained by the Ateles swarm. Each lens agent owns exactly one subsection below; the human-written description above these markers is never modified.

Product / Scope (PM)

The plan_contribution entity stored cleanly with unknown_fields_count: 0. Since there's no issue entity for neotoma#2483 in Neotoma yet (only derived task entities), I can't correct() a gate_status.pm field on an issue entity that doesn't exist here — that gate state lives on the GitHub issue itself, which the dispatcher/Anthus manages outside this MCP's entity graph. The plan_contribution sign-off record is the artifact I can write; I'll note this in my verdict rather than fabricate a correct() call against a nonexistent entity.

[pavo] acceptance_criteria: PM scoping signed off. Problem: #2317 closed one WorkerDbAbortError/abandon() escape path; this issue proves at least one more exists (2026-09-23 hosted crash-loop on a build containing #2317, exact stack in worker_file_database.js:545 onAbort). Scope: full audit of abandon(new WorkerDbAbortError(...)) call sites, root-cause fix for the orphaned-promise rejection, regression test reproducing a mid-query disconnect under reader-pool saturation that is red on a80340c66. Out of scope: reader-pool saturation itself (#2322), read cancellation (#2339); containment PR #2484 explicitly does not satisfy closure here. Acceptance checklist and priority/sequencing posted in the spec section. Localized bug fix, no interface-surface change — bug fast path to Eng/impl applies, no forced arch gate. plan_contribution sign-off stored as ent_b8d16387086f36d3c907eeb3.

Engineering

Prior art check: Containment PR #2484 already exists against this issue (qa: signed_off, pr_review: changes_requested per gate_status) but is explicitly recorded as not satisfying closure — do not treat it as the fix. Its diff should be read as a first data point (what it changed and why reviewers requested changes) but this plan targets the root cause, not the containment patch.

Root-cause hypothesis. #2317 added guardAbandonedRead() — a synchronous no-op .catch() — at the outermost public boundary (WorkerStatement.get/.all in src/repositories/worker/worker_file_database.ts). That should mark the entire derived promise chain as handled, including the promise WorkerConnection.request() constructs and rejects via onAbort (~line 617 TS / dist line 545). Since the crash still reproduces on a build containing that fix, the guard is not covering this call. Two concrete candidates, in priority order:

1. PR Rust panic in neon binding aborts the process when a client disconnects mid-statement (#2316 fix incomplete) #2324 ("retire an abandoned reader instead of terminating it mid-native-call", merged 2026-09-08, one day after fix(db): an abandoned request must not kill the server (#2316) #2317) reworked reader retirement/orphan-draining. This is the most likely place a second, unguarded promise was introduced — e.g. a retirement/drain path that re-issues or re-wraps the read outside the WorkerStatement.get/.all boundary, or a case where failInFlight/abandon now rejects an entry that isn't the one guardAbandonedRead attached to.
2. A caller that bypasses WorkerStatement entirely — some other code path obtaining a prepared statement or connection handle and calling into routeRead/dispatchRead/WorkerConnection.request directly, skipping the guarded boundary.

Files/modules to touch:

• src/repositories/worker/worker_file_database.ts — primary fix location. Specifically: WorkerConnection.request() (the onAbort and timeout branches, ~line 600-626), dispatchRead/routeRead (async boundary layers), WorkerStatement.get/.all (existing guardAbandonedRead call site), and whatever retirement/drain logic PR Rust panic in neon binding aborts the process when a client disconnects mid-statement (#2316 fix incomplete) #2324 added (grep for retire, drain, orphan in this file).
• src/middleware/db_abort_context.ts — read-only reference during investigation (unchanged since Two slow reads exhaust the 2-worker reader pool and hang every query; no statement timeout reclaims a reader #2217, not expected to need changes, but confirm the AbortController it creates isn't being reused/shared across requests in a way that causes cross-request rejection).
• Test file(s) alongside worker_file_database.ts (repo's existing convention, e.g. worker_file_database.test.ts or equivalent under a __tests__/test dir — confirm exact path in step 1 of build steps below) — new regression test.
• No schema, API, or entity-contract changes. This is a pure internal promise-lifecycle bug fix; no data/contract changes.

The concrete change:

• Wherever the audit (build step 1 below) finds an unguarded promise construction or rejection path reachable from a client-disconnect-during-saturation scenario, apply the SAME pattern fix(db): an abandoned request must not kill the server (#2316) #2317 established: attach a synchronous no-op .catch() at every point a WorkerDbAbortError/WorkerDbTimeoutError-rejecting promise is constructed or re-derived, not just at the outermost caller-facing boundary. If PR Rust panic in neon binding aborts the process when a client disconnects mid-statement (#2316 fix incomplete) #2324's retirement path re-issues a promise for an abandoned/retired reader, that re-issued promise needs its own guard — it is a new construction site, not a derivation of the one guardAbandonedRead already wraps.
• Do not weaken or remove the existing guardAbandonedRead wrapping on WorkerStatement.get/.all — keep it, and add guards at any additional construction site found, rather than restructuring the existing fix.
• Keep the fix scoped to reads (matching fix(db): an abandoned request must not kill the server (#2316) #2317's deliberate scoping) unless the audit shows the write/transaction path is also affected by Rust panic in neon binding aborts the process when a client disconnects mid-statement (#2316 fix incomplete) #2324's retirement rework.

Build-step checklist:

1. grep -rn "WorkerDbAbortError\|\.abandon(\|new WorkerDbTimeoutError" src/ repo-wide (the fix(db): an abandoned request must not kill the server (#2316) #2317 investigation only checked within worker_file_database.ts) — enumerate every construction and rejection site, not just the one in the reported stack.
2. Fetch and read PR Rust panic in neon binding aborts the process when a client disconnects mid-statement (#2316 fix incomplete) #2324's diff in full (gh pr diff 2324 or equivalent) against worker_file_database.ts — confirm or rule out whether its retirement/drain rework introduced a second unguarded promise or bypassed the guardAbandonedRead boundary.
3. Also fetch containment PR Contain abandoned-request aborts and capture diagnostics (#2483) #2484's diff — read what it changed and why Vanellus requested changes, so the root-cause fix here doesn't collide with or duplicate an in-flight patch on the same file.
4. Confirm every caller of WorkerFileDatabase.prepare(...).get(/.all( across the repo goes through WorkerStatement (candidate Feature unit FU-113 execution #2 above) — grep for direct WorkerConnection/routeRead/dispatchRead usage outside this file.
5. Write a regression test that: opens a read under simulated reader-pool saturation (reuse or extend existing saturation-simulation helpers used by Reader-pool exhaustion takes the instance down: auth consumes a pooled reader before it can reject an invalid token, and /ready failing under load kills a merely-slow machine #2322's tests if present), triggers a mid-query client disconnect (fire the same AbortController.abort() path db_abort_context.ts uses on res 'close'), and asserts the process does NOT crash / no unhandled rejection is emitted (e.g. via a process.on('unhandledRejection') listener in the test harness, or Node's --unhandled-rejections=throw semantics reproduced in-test). Confirm this test is RED on a80340c66 before writing the fix — a test that passes on the crashing build is not coverage (explicit PM/issue requirement).
6. Apply the guard(s) at the construction site(s) identified in steps 1-4. Re-run the new test — confirm GREEN.
7. Run the full existing worker_file_database test suite plus any Reader-pool exhaustion takes the instance down: auth consumes a pooled reader before it can reject an invalid token, and /ready failing under load kills a merely-slow machine #2322/Interrupt an abandoned libsql read instead of waiting it out (follow-up to #2324) #2339-related suites to confirm no regression in reader-pool or read-cancellation behavior (both explicitly out of scope for this fix — do not modify that logic, only confirm no interaction regression).
8. Regenerate any generated/derived files whose source was touched (test catalog via npm run generate:test-catalog if a new test file was added or moved) as the last step before opening the PR.
9. PR description states the design basis (no docs/foundation/ document governs internal worker promise lifecycle; state "Design basis: no design applies — internal bug fix, no interface/contract change") and explicitly notes the relationship to containment PR Contain abandoned-request aborts and capture diagnostics (#2483) #2484 (does this fix supersede it, or apply on top of it — resolve by reading Contain abandoned-request aborts and capture diagnostics (#2483) #2484's diff in step 3 before writing this line).

QA / Test Plan

Prior-art check (performed, not assumed): Searched gh issue/pr list for WorkerDbAbortError/abandon/saturation coverage before writing anything.

• Contain abandoned-request aborts and capture diagnostics (#2483) #2484's own regression suite (tests/integration/unhandled_abandoned_abort_containment.test.ts) is NOT coverage for this issue and must not be cited as satisfying it. Read its diff: it proves only that installAbandonedAbortContainment()'s process.on("unhandledRejection") listener is wired at the real boot call site (a load-bearing-call-site test) and that it suppresses WorkerDbAbortError specifically. It asserts the symptom is contained, not that the leak is closed — the PM/Eng framing above (containment ≠ closure) holds up under reading the actual diff.
• No reusable reader-pool-saturation helper exists in the repo. Reader-pool exhaustion takes the instance down: auth consumes a pooled reader before it can reject an invalid token, and /ready failing under load kills a merely-slow machine #2322 diagnosed saturation from production logs, not a test fixture; fix(db): retire an abandoned reader instead of terminating it mid-native-call (#2324) #2338's PR touches only the existing db_abort_client_disconnect_survival.test.ts. The build-step note below corrects Eng step 5 accordingly: there is no existing helper to "reuse or extend" — one must be built inline in the new test file, modeled on fix(db): an abandoned request must not kill the server (#2316) #2317's own child-process harness.
• Root cause traced by reading PR fix(db): retire an abandoned reader instead of terminating it mid-native-call (#2324) #2338's diff directly, not just the Eng plan's hypothesis. WorkerFileDatabase.routeRead() (pre-Rust panic in neon binding aborts the process when a client disconnects mid-statement (#2316 fix incomplete) #2324: called this.readerConnection().request(...) directly, which is what guardAbandonedRead at the WorkerStatement.get/.all boundary was proven to cover) now calls a new intermediate async dispatchRead() (added by fix(db): retire an abandoned reader instead of terminating it mid-native-call (#2324) #2338, worker_file_database.ts) that retries up to 4 times on WorkerDbRetiredError via await this.readerConnection().request(...) inside a try/catch loop. Per fix(db): an abandoned request must not kill the server (#2316) #2317's own postmortem, an async function positioned between the guarded boundary and WorkerConnection.request() derives a fresh promise at each await — exactly the failure mode fix(db): an abandoned request must not kill the server (#2316) #2317 already diagnosed and fixed once (by moving the guard outward from WorkerConnection.request() to WorkerStatement.get/.all). dispatchRead reintroduces an intermediate async layer inside that boundary, so a WorkerDbAbortError rejecting on the awaited request() call on attempt 2-4 of the retry loop — reachable only under reader-pool saturation, where retirements are frequent enough to exhaust the 4-attempt bound or race a genuine abort against a retry — has a materially different timing/derivation path than the one case fix(db): an abandoned request must not kill the server (#2316) #2317's tests exercise (single non-retried request() call, no saturation). This is the leading construction site to test and, if red, to fix.

Regression test — the core deliverable, must be RED on a80340c66

New file: tests/integration/db_abort_reader_pool_saturation_survival.test.ts (new file, sibling to and explicitly modeled on tests/integration/db_abort_client_disconnect_survival.test.ts from #2317 — same child-process-liveness pattern, for the identical reason #2317's own PR description gives: vitest installs its own unhandledRejection handler, so no in-process assertion can distinguish "crashed" from "survived" — only a child-process exit-code check can. Do not attempt an in-process version of this test; it will pass on the crashing build and give false coverage.)

Saturation helper (new, since none exists — corrects Eng build-step 5's assumption): a small in-file helper that opens N concurrent slow reads (N ≥ configured reader-pool size) to hold every reader busy, confirmed via the DB reader pool recovered after Nms of saturation log line (or the underlying readerPoolStats()/saturation signal worker_file_database.ts already emits — reuse that signal rather than inventing a new one) before firing the disconnect under test. This is the one precondition #2317's own tests never construct.

Cases (fail-then-pass table, same convention #2317 used):

Case	on a80340c66 (must be ✗)	with fix (must be ✓)
Mid-query client disconnect while the reader pool is saturated (N concurrent slow reads holding every reader, then one caller's res emits 'close' mid-read)	✗	✓
Disconnect landing during a dispatchRead retry attempt (2nd–4th), i.e. abort races a WorkerDbRetiredError-triggered re-dispatch	✗	✓
Disconnect landing on the 4th (final, non-retried) dispatchRead attempt, where lastError is thrown directly instead of retried	✗	✓
Reconnect loop: 5× abandoned SSE streams under sustained saturation (mirrors #2317's own reconnect-loop case, which is what reproduced the loop in production, not just a single crash)	✗	✓
Control: mid-query disconnect without saturation (single reader, no contention) — must already pass on a80340c66 per #2317's existing suite	✓ (already covered)	✓

Each case asserts, per #2317's established assertion shape: (1) child process is still alive after the disconnect, (2) child still serves a fresh DB read afterward, (3) no unhandled-rejection crash trace in child stderr. Do not assert on #2484's containment log line ("unhandled WorkerDbAbortError contained") as a substitute pass condition — if the root-cause fix lands, that log line should stop appearing for this scenario entirely (the rejection becomes handled at its construction site, never reaching the process-wide listener); a test that passes because #2484's global handler caught it is testing containment, not closure, and must not be conflated with this suite's purpose.

Edge cases for the fix's new branches (once the construction site is found and guarded)

• Guard applied to the dispatchRead retry loop's per-attempt promise must not swallow a genuine WorkerDbRetiredError on attempts 1-3 (retry must still occur) — only the caller-abandonment case is guarded, not the retry's own control flow.
• A write reaching a retired connection (if dispatchRead's pattern is ever mirrored for writes) must NOT be guarded — fix(db): an abandoned request must not kill the server (#2316) #2317 and this plan both keep the write path loud deliberately; add an explicit negative-path test asserting a write-side unhandled rejection still crashes the process, so a future change to dispatchRead-adjacent write logic can't silently widen the guard's scope.
• If the fix touches retire()'s Promise.reject(new WorkerDbRetiredError(...)) (line ~327 of the fix(db): retire an abandoned reader instead of terminating it mid-native-call (#2324) #2338 diff) directly: confirm a caller still actively awaiting (not abandoned) receives the real WorkerDbRetiredError/WorkerDbAbortError and can still map it to a response — same "callers that await are unaffected" property fix(db): an abandoned request must not kill the server (#2316) #2317 verified explicitly.
• Exhausting all 4 dispatchRead retry attempts under sustained saturation (pool never recovers within the retry budget) — assert the final thrown lastError is still guarded if the caller has abandoned, and still surfaces normally if the caller is still listening.

No new endpoints / no contract surface

This is a pure internal promise-lifecycle fix (per Eng scope) — no MCP tool, recipe, schema, or CLI surface changes. No agentic_eval fixture or contract test applies; the regression test above (Vitest integration, child-process harness) is the correct and sufficient artifact per this repo's own established convention for this exact bug class (#2317's precedent), not the Neotoma agentic-eval substrate, which covers agent-facing MCP/store-retrieve behavior rather than internal worker-process crash survival.

Definition-of-done checklist

• tests/integration/db_abort_reader_pool_saturation_survival.test.ts added, using the child-process liveness pattern (not in-process) per fix(db): an abandoned request must not kill the server (#2316) #2317's precedent
• Saturation helper built inline (none existed to reuse — corrects Eng step 5)
• All 4 saturation-scenario cases confirmed RED on a80340c66 before the fix is written (screenshot/log of the failing run retained in the PR description, matching fix(db): an abandoned request must not kill the server (#2316) #2317's own fail-then-pass table convention)
• Control case (no saturation) confirmed already green on a80340c66, proving the new test isolates the saturation precondition specifically
• Construction-site audit (Eng step 1) run repo-wide, not scoped to worker_file_database.ts alone
• dispatchRead (added by fix(db): retire an abandoned reader instead of terminating it mid-native-call (#2324) #2338) specifically inspected as the leading candidate — confirmed above by reading the fix(db): retire an abandoned reader instead of terminating it mid-native-call (#2324) #2338 diff directly, not left as an open hypothesis
• Fix applied; all 4 saturation cases + reconnect-loop case turn GREEN
• Negative-path write test added and passing (write-side abandonment still crashes)
• Full existing worker_file_database, Reader-pool exhaustion takes the instance down: auth consumes a pooled reader before it can reject an invalid token, and /ready failing under load kills a merely-slow machine #2322, and Interrupt an abandoned libsql read instead of waiting it out (follow-up to #2324) #2339-related suites re-run with no regression
• npm run generate:test-catalog re-run (new test file added)
• PR explicitly states whether this fix supersedes Contain abandoned-request aborts and capture diagnostics (#2483) #2484 or layers on top of it, resolved from Contain abandoned-request aborts and capture diagnostics (#2483) #2484's diff (this review found Contain abandoned-request aborts and capture diagnostics (#2483) #2484 to be a process-wide containment net that should very likely remain as defense-in-depth even after root-cause closure — recommend "layers on top of," not "supersedes," since Contain abandoned-request aborts and capture diagnostics (#2483) #2484 also protects against other, still-undiscovered WorkerDbAbortError leaks; Eng/PM should confirm this framing in the PR body)

QA gate status: Cannot sign off yet — no eval/test artifact is committed against this repo checkout in this session (spec-authoring turn only, no code changes made). This test plan is the committed deliverable for the QA lens at this stage of the ORDERED ADDITIVE SPEC pipeline; the actual fixture/spec file lands when Eng implements against this plan, and QA re-verifies red→green at that point before signing the qa gate.

[ateles-agent]

Another reproduction on the operator instance, 2026-09-23 ~10:36–10:45Z.

Trigger: a read-only audit client ran 10 concurrent requests (GET /entities/<id> plus POST /list_observations) over roughly 500 entities, with client-side timeouts and retries. Some of those requests were abandoned mid-statement.

Effect:

• The machine exited with exit_code=134, oom_killed=false (SIGABRT) about 30s after each start, twice (10:38:34Z and 10:40:52Z).
• The Fly proxy's autostart then flooded Starting machine, and the machines API returned rate limit exceeded. A manual fly machine start was refused until about 10:45Z.
• Just before the abort, the log showed DB reclaimed 1 in-flight statement(s): DB request aborted by caller repeated every 1–3s, plus Failed to get raw_fragments … DB request aborted before dispatch.

Recovery: stopping the audit client, then running fly machine start once the rate limit window cleared. The machine has been healthy since.

This matches the #2317-uncovered path described above. The client-side mitigation (concurrency ≤2 and no abandon-on-timeout for bulk reads) is being applied on our side, but the server should not abort on client disconnect.

🤖 Ateles

[ateles-agent]

Investigation: not reproduced locally

A worker attempted to reproduce this against the crashing build (a80340c66, unpatched, run with --unhandled-rejections=throw to match production). Five harness shapes, all survived:

1. One reader, concurrent reads with one disconnecting caller — bystander requests are rejected by abandon(), but each rejection is caught by its own guardAbandonedRead.
2. Two readers, abort and timeout racing, 300 batched iterations.
3. Streaming-handler shape (headers written, fire-and-forget .all()), abort and timeout racing, 400 iterations.
4. One reader, 14 concurrent disconnecting clients per iteration for 90s — reproduces the exact production log signature (DB reclaimed 14 in-flight statement(s), DB reader pool recovered after 2200ms of saturation) without the process exiting.
5. Handler shape with several awaits and a dynamic import before the read, disconnecting both after first byte and before any response.

Static trace: every read path reaches WorkerStatement and guardAbandonedRead; no fire-and-forget read outside that pattern was found. The dispatchRead() retry loop added by #2338 sits between routeRead and request() and is the leading suspect — #2317's own note warned that an async hop derives a fresh promise — but no harness made it leak.

One unrelated gap found: WorkerConnection.terminate() awaits worker.terminate() without a catch, unlike the two guarded call sites in abandon(). Only reachable from close(), which the request path never calls.

Next step

Since the failure is real in production but not reproducible synthetically, the follow-up is containment plus diagnosis rather than a speculative source fix:

• a handler scoped to WorkerDbAbortError only, that logs a structured event (full stack, cause, in-flight counts) and does not exit — any other unhandled rejection still exits as today;
• --stack-trace-limit raised so the next occurrence carries the async frames beyond onAbort;
• the unguarded terminate() fixed.

The logged events then identify the leaking path, and the source fix follows from them.

[neotoma-agent]

🤖 Lanius — Ateles swarm, issue triage
COMMENT

Issue triaged. Pavo has been assigned as PM owner for Phase 1 scoping.

Triage & Gate-Status Board

Workflow: neotoma|bug (fast path: label:bug → skip ux)

Gate	Owner	Status
pm	Pavo	pending
ux	Accipiter	not_required (bug fast path)
arch	Waxwing	pending
impl	Gryllus	pending
pr_review	Vanellus	pending
qa	Phoenicurus	pending
legal	Buteo	not_required

Owner: current phase owner is Pavo (Phase 1, pm gate). Next: Waxwing (arch) once Pavo signs off — arch is not skipped here despite the bug label, since this is a production crash fix, not a schema/API/tool/contract change per the interface-surface predicate.

Labels applied: lanius-triage (pre-existing on repo, applied to this issue).

• Issue entity initialized (gate_status, current_owner, owner_history)
• Linked to workflow_definition:neotoma|feature (bug fast-path variant applied)
• Related issues on record: An abandoned request crashes the server: unhandled WorkerDbAbortError from db_abort_context, then a neon SIGABRT #2316, fix(db): an abandoned request must not kill the server (#2316) #2317, Reader-pool exhaustion takes the instance down: auth consumes a pooled reader before it can reject an invalid token, and /ready failing under load kills a merely-slow machine #2322, Interrupt an abandoned libsql read instead of waiting it out (follow-up to #2324) #2339

📎 Neotoma: issue ent_a92adbfc579e0b8755a97721