Skip to content

fix(release): decide lock takeover on owner liveness, not lockfile age - #1055

Open
BryanFRD wants to merge 4 commits into
fix/release-lock-in-worktreesfrom
fix/release-lock-pid-liveness
Open

fix(release): decide lock takeover on owner liveness, not lockfile age#1055
BryanFRD wants to merge 4 commits into
fix/release-lock-in-worktreesfrom
fix/release-lock-pid-liveness

Conversation

@BryanFRD

@BryanFRD BryanFRD commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Closes #792.

ReleaseLock wrote a PID and a hostname into the lockfile and then never read either. take_over_if_stale looked only at the mtime, which is written once at acquire and so records the start time rather than any liveness signal. Two consequences, both reproduced as failing tests before the fix:

  • a release still running after the TTL lost its lock to a concurrent run, which is exactly the half-pushed tag set the lock exists to prevent;
  • a run that crashed inside the TTL kept blocking until the TTL elapsed, even though its PID was demonstrably gone.

Takeover is now decided by asking whether the owner is still alive:

  • Lock written by this host. Liveness is authoritative. A dead owner is taken over at once, however recent the lockfile. A live one is never taken over, however old.
  • Lock written by another host, or unparseable. This host cannot ask about that PID, so STALE_LOCK_TTL decides, as before.
  • Hostname unknown. Treated as another host. It is the fallback when neither HOSTNAME nor COMPUTERNAME is set, so two machines can both claim it and their PIDs would be compared meaninglessly.

STALE_LOCK_TTL goes from 30 minutes to 6 hours. It is now only the fallback for an owner on a machine we cannot query, where hours is the honest number.

Liveness uses kill(pid, 0) on unix (treating EPERM as alive, since the process exists but belongs to another user) and OpenProcess + WaitForSingleObject on Windows. WaitForSingleObject avoids the GetExitCodeProcess ambiguity where a process that legitimately exits with 259 is indistinguishable from STILL_ACTIVE. Both libc and windows-sys were already in the dependency tree, so this adds no new compilation.

One thing the issue asked for that this does not do

The issue also asked to refresh the lockfile mtime through the run so the TTL measures staleness rather than duration. That is unnecessary under this design and I left it out: on the same host the TTL no longer applies at all, and the same-host case is the one both reported bugs are in. Adding a refresh would mean threading a touch through the release pipeline (the long stretch is inside a single phase, so the existing phase boundaries are the wrong seam anyway) to buy nothing. Happy to add it if you disagree.

There is one deliberate trade-off. If a dead owner's PID has been reused by an unrelated live process, the lock blocks until --force-unlock rather than expiring on the TTL. That direction is chosen on purpose: blocking wrongly costs one documented manual step, while stealing a live lock corrupts a release. It is also the direction the issue asks for, since "a live release past 30 minutes loses its lock" is the bug being fixed.

Verification

Against the previous mtime-only rule:

test monorepo::run::lock::tests::a_live_owner_keeps_its_lock_however_old_the_lockfile_is ... FAILED
test monorepo::run::lock::tests::a_dead_owner_on_this_host_is_taken_over_without_waiting_for_the_ttl ... FAILED

a_dead_pid() spawns a real process, reaps it, and asserts process_is_alive reports it dead before the test proceeds, so a test that would otherwise pass vacuously fails loudly instead. another_hosts_lock_is_still_judged_on_the_ttl_alone uses this process's own live PID under a foreign hostname, so it fails if the host check is ever dropped. an_unnamed_host_is_never_trusted_for_liveness and an_unreadable_lockfile_falls_back_to_the_ttl cover the two fallback paths.

These ran on Windows, so the OpenProcess path is exercised rather than only compiled.


Top of a stack of four. Based on #1054.

@ferrfleet ferrfleet Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design is the right call, including the "block rather than steal" trade-off and leaving the mtime refresh out. Two blocking findings, both about the liveness check being wrong at the host boundary: the env-var hostname makes the same-host branch dead code on Linux (and, I think, red CI on this commit), and the Windows path reads an access-denied process as dead. One nit on pid parsing.

Comment thread src/monorepo/run/lock.rs
Comment thread src/monorepo/run/lock.rs
Comment thread src/monorepo/run/lock.rs Outdated

@ferrfleet ferrfleet Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Liveness over mtime is the right call, and the tests are the kind that fail for the right reason (a_dead_pid asserting its own premise, the foreign-host test using a live local PID). One blocking issue on the Windows path, plus two nits.

One more thing not covered by the host check: equal hostnames do not imply the same PID namespace. Two containers with a pinned hostname sharing a bind-mounted repo compare PIDs from different namespaces, where low PIDs collide readily, and that can steal a live lock as well as block on a dead one. Stamping something namespace-specific alongside the host (the /proc/self/ns/pid inode on Linux, absent elsewhere) and requiring it to match would close it, and an unrecognised or missing marker falls back to the TTL exactly like a foreign host. Not needed for this PR, but the deliberate trade-off in the description reads as if the same-host case is fully sound, and this is the case where it is not.

Skipping the mtime refresh is fine as argued, given the TTL is no longer the mechanism on the host that matters.

Comment thread src/monorepo/run/lock.rs Outdated
Comment thread src/monorepo/run/lock.rs Outdated
Comment thread src/monorepo/run/lock.rs
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

SonarQube — aucune nouvelle issue

Comparaison entre le projet bac à sable de cette PR et la branche par défaut : SonarQube Community n'analyse pas les PR, ce delta est calculé côté CI. Détail

@BryanFRD
BryanFRD force-pushed the fix/release-lock-pid-liveness branch from 144ecfd to 03442d4 Compare September 7, 2026 12:22
@BryanFRD

BryanFRD commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

All four applied.

Access denied on Windows (03442d4): OpenProcess returning null now checks GetLastError() == ERROR_ACCESS_DENIED and reads that as alive, the counterpart of the unix EPERM branch. Verified this is load-bearing rather than theoretical: with the branch reverted to a plain return false, the new test fails here.

test a_process_this_user_may_not_open_still_reads_as_alive ... FAILED
pid 4 is always running, whether or not this user can open it

The test uses pid 1 on unix and pid 4 on Windows, per your suggestion.

Process-group pids (03442d4): parse_lock_info now rejects 0 and anything above i32::MAX, so such a lockfile reads as unparseable and takes the TTL path. a_pid_that_would_address_a_process_group_is_not_trusted covers both plus a valid pid.

Recursion on a failed removal (03442d4): both branches return remove_file(...).is_ok() now. No test: the condition needs a directory that refuses unlink, and there is no way to arrange that identically on unix and Windows. A test that no-ops on one of them would be worse than none, so the fix ships without one.

Docs: the takeover paragraph is rewritten in docs-en and docs-fr, saying same-host takeover is immediate and the 6 hour timeout is only the fallback for a host this machine cannot query. docs-v5 and docs-v6 are left alone. The common-dir change from the layer below got its own docs update in #1054, so each PR carries the paragraph it invalidates.

Hostname: as you noted, already fixed in 91f5859 before this round.

@ferrfleet ferrfleet Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All four findings are fixed in 03442d4 and the threads are resolved. Nothing new in the changed code, and Test and Coverage are green on this commit.

@BryanFRD
BryanFRD force-pushed the fix/release-lock-pid-liveness branch from 03442d4 to 5d526a1 Compare September 7, 2026 13:15
@BryanFRD
BryanFRD force-pushed the fix/release-lock-pid-liveness branch from 5d526a1 to 3e0aa6f Compare September 7, 2026 13:25
@BryanFRD
BryanFRD force-pushed the fix/release-lock-pid-liveness branch from 3e0aa6f to a3b67ec Compare September 7, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(release): stale-lock takeover ignores the PID it writes and the lock is never refreshed — a >30min release gets its lock stolen

1 participant