Repository navigation
perf: cut Railway memory (web fetch deadline leak, heap cap, pool idle timeout, proxy GOMEMLIMIT) - #196
Merged
Conversation
sqlx keeps idle pool connections for 10-30 minutes by default, and each one holds 2-6 MB in the Postgres server process (more as its prepared-statement cache fills). Close idle connections after 60s and recycle each connection after 10 minutes. The pool size (SEC-09) and acquire timeout are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each multi-MB food-photo request allocates ~30x its size in garbage while the body is translated, and Go's default GC keeps that resident long after the request. A 64MiB soft limit cut retained memory after photo traffic from ~100 MB to ~77 MB locally (10 MB images: 180 MB to 80 MB) at negligible latency cost, and keeps large requests from being OOM-killed under a hard limit. Operators can still override it with the GOMEMLIMIT variable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backendFetch combined the caller's signal with AbortSignal.timeout(10s). Once fetch attaches its abort listener, Node keeps a pending timeout signal alive until the timer fires, and through it the finished request, response and gzip buffers. Under load that held ~1,600 completed requests at once, roughly doubling the buffer peak and leaving glibc's arena at that high-water mark (settled memory ~165 MB vs ~107 MB locally). Use a plain AbortController plus a timer that is cleared once the body ends, errors or is cancelled. The deadline still covers the body, still rejects with a TimeoutError, and still forwards the caller's abort reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Railway gives the web service an 8 GB limit, so V8 sizes its heap for that and keeps a traffic peak's heap resident until the next deploy (production sits flat at ~145 MB overnight). With the backend fetch fix the live heap after load is ~45-75 MB; capping old space at 192 MB took local settled memory from ~108 MB to ~80 MB and the peak from ~280 MB to ~130 MB, at +1 ms p50 / +3 ms p95 under a 4-way concurrent burst. Set on the command line rather than NODE_OPTIONS so the startup migration child is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AbortSignal.timeout never kept the process alive; the replacement timer should not either. Also pin that cancelling the response body (the barcode disconnect path) releases the deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cuts steady-state memory on Railway without user-visible changes. Railway bills memory per GB-minute, and the web service sat flat at ~145 MB (prod) / ~175 MB (staging) all night after each traffic burst.
backendFetchclears its 10 s deadline once the response settles, instead ofAbortSignal.timeout--max-old-space-size=192on the web start command (bothrailway.tomlcopies)idle_timeout60 s +max_lifetime10 min (pool size unchanged, SEC-09)GOMEMLIMIT=64MiBdefault in the proxy entrypointRoot cause (web): once
fetchattaches its abort listener, Node 22 keeps a pendingAbortSignal.timeoutalive until the timer fires, and through it the finished request, response and gzip buffers. A heap snapshot after load held ~1,600 finishedServerResponses and 1,456 live 10 s timers. That doubled the Buffer peak, and glibc's malloc arena stays at its high-water mark. The new code uses anAbortControllerplus a timer cleared when the body ends, errors or is cancelled. The deadline still covers the body, still rejects withTimeoutError, and still forwards the caller's abort reason. The returnedResponseno longer carriesurl/redirected; no caller reads them.Measured locally in Docker with Node 22.23.4 (Railway's version), the staging code, a Rust backend and Postgres 18, driving 1,380 authenticated page + RSC requests per round at 4-way concurrency.
Outside this diff (Railway settings, applied separately)
max_connections500 → 100 (the Railway image appends 500); applied on staging, production follows on promotion.shared_buffers128MB → 32MB, alongside it.Verification
backend-client.test.ts(deadline released after body read and for body-less responses, stalled body still times out, caller abort reason forwarded, status/headers/body pass through). The two release tests failed before the fix.postgres_pool_releases_idle_connections_quickly(Rust).backendFetchcaller (68 tests incl.backend-proxy-cancellation), Railway config/startup tests, web + db typecheck, eslint: pass.staging; CI is authoritative for those.Versioning
Performance fix, no feature: version unchanged (3.6.0).
🤖 Generated with Claude Code