Compress trace batches (zstd or gzip) before they wait in the upload queue #44

Problem

In the JS SDK, a trace batch waiting to upload holds its runs uncompressed. When uploads fall behind, that memory grows until the process is killed.

We run LangGraph deep agents in Node.js 26 on 4 GB ECS Fargate tasks, using langsmith@0.10.5 with the default maxIngestMemoryBytes. When about eight scheduled agent runs start on a task at the same time, the task's memory climbs from about 1.8 GB to the 4 GB container limit within 5–10 minutes, and the container is killed (exit 137, not a V8 heap OOM). In the minute before each kill, the client logs Queue size limit (1073741824 bytes) exceeded dozens of times.

LangSmith accepted every upload during the burst: all 202, with no 429 or 5xx. The uploads just took longer: p95 went from under 1 s to 4–8 s, with the slowest at 54 s. Agent traces are large because every model call carries the full message history, so 5 concurrent uploads (traceBatchConcurrency) could not keep up.

Where the memory goes (src/lib/client.ts at 0c8c86e)

• Serialized, then queued uncompressed. processRunOperation (L2429) → drainAutoBatchQueue (L2157) → multipartIngestRuns (L2794) serializes every run of every drained batch into Blob parts (L2907) before the request is queued in batchIngestCaller. _sendMultipartRequest (L3069) compresses only while sending (CompressionStream("gzip"), L3111). So the part of the pipeline that grows during a backlog holds uncompressed bytes.
• The limit counts an estimate, not what is held. The AsyncCaller limit (src/lib/utils/async_caller.ts L103) counts an estimate of each batch's size. The serialized Blobs live outside the V8 heap, which is why the container is killed instead of V8 throwing a heap OOM.
• Finished batches keep their runs alive. The done callback from AutoBatchQueue.pop (L1087) captures the popped runs, so they stay alive until their batch is sent.
• Retries recompress. The streaming gzip path rebuilds and re-gzips the body on every retry.

The Python SDK already does this for zstd

• Compresses as runs arrive. src/langsmith/_internal/_compressed_traces.py in langchain-ai/langsmith-python writes each run's multipart parts into a streaming zstd compressor and holds only the compressed buffer.
• Sends that buffer unchanged. _send_compressed_multipart_req POSTs it to /runs/multipart with Content-Encoding: zstd, X-Pre-Compressed-Size and X-Post-Compressed-Size.
• Settings. Defaults to level 1 (LANGSMITH_RUN_COMPRESSION_LEVEL) and one worker thread (perf(py): default zstd compression to a single worker thread langsmith-sdk#3395).
• Supported on both sides. /info on api.smith.langchain.com reports zstd_compression_enabled: true. Node.js has zstd in node:zlib since 22.15.0 / 23.8.0.

Proposal

Compress each multipart batch before it joins the upload queue, send the compressed buffer unchanged, and count its compressed size against maxIngestMemoryBytes. Support zstd and gzip the same way, so runtimes and servers without zstd still get the memory benefit. I have a working implementation and am opening a PR with it:

• ClientConfig.runCompression.
  • true / "auto" picks zstd when the runtime has it and the server reports zstd_compression_enabled, otherwise gzip when the server reports gzip_body_enabled.
  • "zstd" or "gzip" uses only that encoding.
  • { algorithm, level, windowLog } also sets the level (zstd ≤ 22, default 3; gzip 0–9, default 1) and, for zstd, the match window (10–27).
  • Env vars: LANGSMITH_RUN_COMPRESSION (auto / zstd / gzip), LANGSMITH_RUN_COMPRESSION_LEVEL (shared with Python) and LANGSMITH_RUN_COMPRESSION_WINDOW_LOG.
  • Respects LANGSMITH_DISABLE_RUN_COMPRESSION, and rejects invalid values when the client is constructed.
  • It is off by default. When it is off, or no encoding applies, uploads use today's streamed gzip path unchanged.
• src/lib/utils/compression.ts (Node). Writes the multipart body into node:zlib's zstd or gzip compressor one part at a time, so the uncompressed body is never joined into one buffer.
• compression.browser.ts. Selected through the browser field, like fs and worker_threads. It gzips with CompressionStream, which takes no level, so pre-compression also works in browsers and edge runtimes; zstd is unavailable there.
• _processBatch.
  • Compresses the batch, then empties the parts list, so the queued request holds only the compressed body. The same buffer is reused on retries.
  • zstd requests carry the Python size headers.
  • Feedback batched with the runs goes in the same compressed request.
  • Failed uploads are still written to LANGSMITH_FAILED_TRACES_DIR, with the Content-Encoding recorded in the replay headers.
• AutoBatchQueue.pop. Its done callback now captures only the resolvers.

Measurements

All on synthetic agent-shaped traces: each model-call run carries its agent's full growing history.

Compression only. One 13 MB batch: 96 runs from 8 interleaved agents.

Most of the bytes are repeated copies of each agent's history. gzip's 32 KB window can't reach back to the earlier copy, and zstd level 1's fast strategy doesn't find it either. That is why the implementation defaults zstd to level 3 and gzip to level 1: level 6 costs more CPU for no gain on this data. It may also be worth revisiting Python's level-1 zstd default for agent workloads.

Memory held by queued uploads, built from the implementation. 139 MB of run inputs from 16 agents × 30 calls, arriving in rounds 400 ms apart. Uploads are held open, so every batch waits in the queue. Figures are after GC, relative to before tracing.

Questions for maintainers

1. Opt-in or default? Should this be opt-in at first, or on by default (as zstd is in Python)? Turning on gzip pre-compression by default would change the upload path for every JS user. Beyond the memory win, it removes the chunked-upload 422 fallback, because bodies are always buffered.
2. Default levels and window. The implementation uses zstd 3 with zstd's default window, and gzip 1.
3. Per-batch frames vs a long-lived frame. The implementation compresses each drained batch separately. That's a small change, but a run can't reference history already sent in an earlier batch. Python's long-lived frame would compress better, especially with a larger windowLog. This PR or a follow-up?
4. 404 from /runs/multipart. In the pre-compressed path, a 404 drops that batch's runs, because the uncompressed runs are no longer kept for the /runs/batch fallback. Its feedback is still posted one by one, and later batches switch over. Acceptable?
5. Queue limit checked after compressing. The limit is checked after serializing and compressing, as it is today for serializing, so work is wasted when the queue is full. A cheap pre-check would avoid that.