Skip to content

release: promote Railway memory reduction to staging - #197

Merged
lnieuwenhuis merged 7 commits into
stagingfrom
dev
Oct 23, 2026
Merged

lnieuwenhuis merged 7 commits into
stagingfrom
dev

Conversation

@lnieuwenhuis

Copy link
Copy Markdown
Owner

Summary

Promotes dev to staging: the Railway memory reduction from #196 (plus the #195 sync merge commit).

  • web: backendFetch releases its 10 s deadline once the response settles (fixes finished requests being retained for 10 s), and --max-old-space-size=192 on the start command
  • backend: sqlx pool idle_timeout 60 s, max_lifetime 10 min
  • AI gateway: GOMEMLIMIT=64MiB default

After deploy, Railway memory for the web, backend, proxy and Postgres services will be compared against the pre-deploy 7-day baseline before promoting to main.

Verification

All CI jobs green on #196 (typecheck/lint/build, Rust backend, web unit, DB package, unused code/duplicates, browser tests, browser smoke).

🤖 Generated with Claude Code

lnieuwenhuis and others added 7 commits October 23, 2026 13:11
sqlx keeps idle pool connections for 10-30 minutes by default, and each one
holds 2-6 MB in the Postgres server process (more as its prepared-statement
cache fills). Close idle connections after 60s and recycle each connection
after 10 minutes. The pool size (SEC-09) and acquire timeout are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each multi-MB food-photo request allocates ~30x its size in garbage while
the body is translated, and Go's default GC keeps that resident long after
the request. A 64MiB soft limit cut retained memory after photo traffic from
~100 MB to ~77 MB locally (10 MB images: 180 MB to 80 MB) at negligible
latency cost, and keeps large requests from being OOM-killed under a hard
limit. Operators can still override it with the GOMEMLIMIT variable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backendFetch combined the caller's signal with AbortSignal.timeout(10s).
Once fetch attaches its abort listener, Node keeps a pending timeout signal
alive until the timer fires, and through it the finished request, response
and gzip buffers. Under load that held ~1,600 completed requests at once,
roughly doubling the buffer peak and leaving glibc's arena at that
high-water mark (settled memory ~165 MB vs ~107 MB locally).

Use a plain AbortController plus a timer that is cleared once the body ends,
errors or is cancelled. The deadline still covers the body, still rejects
with a TimeoutError, and still forwards the caller's abort reason.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Railway gives the web service an 8 GB limit, so V8 sizes its heap for that
and keeps a traffic peak's heap resident until the next deploy (production
sits flat at ~145 MB overnight). With the backend fetch fix the live heap
after load is ~45-75 MB; capping old space at 192 MB took local settled
memory from ~108 MB to ~80 MB and the peak from ~280 MB to ~130 MB, at
+1 ms p50 / +3 ms p95 under a 4-way concurrent burst. Set on the command
line rather than NODE_OPTIONS so the startup migration child is unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chore: sync staging into dev
AbortSignal.timeout never kept the process alive; the replacement timer
should not either. Also pin that cancelling the response body (the barcode
disconnect path) releases the deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
perf: cut Railway memory (web fetch deadline leak, heap cap, pool idle timeout, proxy GOMEMLIMIT)
@lnieuwenhuis
lnieuwenhuis merged commit ee87050 into staging Oct 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant