Skip to content

fix(statusline): bound the /tmp pai-parallel leak (follow-up to #1297) - #1767

Open
gcaspar wants to merge 1 commit into
danielmiessler:mainfrom
gcaspar:fix/statusline-tmp-parallel-leak
Open

fix(statusline): bound the /tmp pai-parallel leak (follow-up to #1297)#1767
gcaspar wants to merge 1 commit into
danielmiessler:mainfrom
gcaspar:fix/statusline-tmp-parallel-leak

Conversation

@gcaspar

@gcaspar gcaspar commented Aug 4, 2026

Copy link
Copy Markdown

Follow-up to #1297, which was closed on 2026-06-21 for architectural reasons rather than on merit ("If the need still stands against the new model, we'd love a fresh PR once it lands"). The new model has landed, the leak is still present in the installer layout, and it took down a box last week — so here is that fresh PR.

Still present on main today

LifeOS/install/LIFEOS/LIFEOS_StatusLine.sh:

  • L584 _parallel_tmp="/tmp/pai-parallel-$$"
  • L585 mkdir -p "$_parallel_tmp"
  • L1067 rm -rf "$_parallel_tmp" 2>/dev/null

482 lines apart, several of them network calls, with no trap. Cleanup is on the exit path only, so any invocation that doesn't reach L1067 leaks a directory permanently.

A second failure mode worth knowing about: inodes, not bytes

#1297 reported this as a disk-space problem (~8 GB of block metadata on ext4). On a tmpfs it presents completely differently and is much easier to misdiagnose.

Headless Ubuntu 22.04 box, /tmp on a 1.9 GB tmpfs, 4 concurrent SSH sessions, refreshInterval: 1:

244,760  /tmp/pai-parallel-* directories
df -i    1048571 / 1048576 inodes used   (100%)   <-- exhausted
df -h    1.5G / 1.9G used                ( 81%)   <-- looks fine

Every shell tool died with ENOSPC: no space left on device while df -h showed 380 MB free, because Claude Code stages each Bash command and its output under /tmp. Anything shell-based (curl, jq, even echo) fails; only the non-shell file tools keep working. If you're debugging this, df -i is the tell — df -h will send you the wrong way.

Diagnosis is also slow once it happens: ls -la /tmp, du, and any recursive find all hang on ~250k entries. ls -U and find -maxdepth 1 are the usable tools.

Correction to #1297: a trap does help, for the dominant signal path

#1297 states that a trap ... EXIT INT TERM HUP "does not help" because Claude Code invokes via /bin/sh -c and the bash child blocked in wait never receives the signal. That holds for one termination pattern but not the one that actually leaks. Measured on the affected box, real script, 5 reps per cell, killed at 0.4 s into a run that takes ~0.78 s when healthy:

termination with trap without trap
SIGTERM to the sh parent only 0/5 leaked 0/5 leaked
SIGTERM to the process group 0/5 leaked 5/5 leaked
SIGKILL to the process group 3/5 leaked 3/5 leaked

Reading:

So neither layer alone is enough: the trap covers group-SIGTERM, the entry-path sweep from #1297 covers SIGKILL. This PR ships both.

The fix

_parallel_tmp="/tmp/pai-parallel-$$"
find /tmp/ -maxdepth 1 -type d -name 'pai-parallel-*' -mmin +1 -exec rm -rf {} + 2>/dev/null &
mkdir -p "$_parallel_tmp"
trap 'rm -rf "$_parallel_tmp" 2>/dev/null' EXIT INT TERM HUP

Notes on the details, all of which matter:

  • Per-PID dirs are kept. A fixed-name dir races: concurrent sessions clobber each other's files mid-source and statusline sections silently vanish. (Same conclusion as statusline-command.sh: /tmp leak from pai-parallel-$$ dirs that are never cleaned up #1297.)
  • The trailing slash on /tmp/ is required, not cosmetic. Per @jmmarkiewicz in statusline-command.sh: /tmp leak from pai-parallel-$$ dirs that are never cleaned up #1297: on macOS /tmp is a symlink to /private/tmp, and find(1) will not descend into a symlinked start point, so find /tmp -maxdepth 1 ... silently matches nothing — the same "looks like it ran, didn't" failure as the original bug.
  • The sweep is backgrounded so it never adds latency to a render.
  • The trap is safe against self-sabotage. Bash resets traps in subshells, so the backgrounded blocks below it cannot fire it and delete the scratch dir mid-render. Verified explicitly rather than assumed.
  • -mmin +1 is comfortably safe. A healthy render completes in ~0.78 s.

Verification

Worst case, every single render SIGKILLed mid-run (so the trap can never fire and L1067 is never reached):

30 renders, all SIGKILLed        -> 26 dirs leaked
after 70s idle                   -> 26 dirs   (orphaned, nothing reclaims them)
after ONE healthy render         ->  0 dirs   (entry-path sweep reaps the backlog)

Accumulation is bounded by one minute of render rate instead of being unbounded. On the affected box, steady state went from 244,760 dirs to ~50, and inode usage from 100% to 3%.

bash -n clean. Tested on Ubuntu 22.04 / bash 5.1. The macOS find behaviour is quoted from #1297 rather than retested here.

Operator note, separate from this patch

refreshInterval: 1 is the amplifier — it turns any statusline bug into a 1 Hz bug, and a healthy render already takes ~0.78 s, so renders nearly overlap. Worth considering a higher default independently of this fix.

Cleanup lived only on the exit path -- the rm -rf sits ~480 lines after the
mkdir, past several network calls -- so any render Claude Code killed leaked its
scratch dir permanently. At refreshInterval=1 across 4 sessions this reached
244,760 directories and exhausted the tmpfs inode table while df -h still showed
81% free, which kills every shell-based tool in the session.

Adds two layers, because neither is sufficient alone:

- Entry-path sweep of dirs older than a minute. The only layer that covers
  SIGKILL, which cannot be trapped. The trailing slash on /tmp/ is required or
  this is a silent no-op on macOS, where /tmp is a symlink find(1) will not
  descend into.
- A trap, measured to fully close the group-SIGTERM path: 5/5 renders leaked
  without it, 0/5 with it.

Per-PID dirs are kept deliberately; a fixed name races and concurrent sessions
clobber each other's files mid-source.

Follow-up to danielmiessler#1297, which was closed for architectural reasons before the
installer layout landed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant