fix(worker): a job that completed while the queue was unreachable is neither reported failed nor run again

[edgehero]

A job that finished while Valkey was unreachable could be reported failed, and a scheduled one could run a second time, paid. Found while testing #577; no issue was filed for it.

What happened

When Valkey stays away longer than BullMQ's lock renewal window while a job is finishing, the processor still writes its run record, but BullMQ can no longer mark the job completed: moveToCompleted fails with "Missing lock" and the job stays active without a lock. The stall checker then acts on it:

• a plain job gets BullMQ's deferred failure "job stalled more than allowable limit", so the worker posted the failure comment and paged the operator for a job that had completed;
• a scheduled job is put back to wait and runs again, a second paid run, and its record overwrites the first one.

The fix

The run record is already a durable store: one file per job id carrying the attempt that wrote it, so no new store is needed.

• When a job fails with exactly that stall reason and a record for that same attempt exists with an outcome other than failed, the worker logs job_lost_lock_after_completion as the job's terminal line and posts nothing. A recorded paid policy stop still pages, because the first finish never reached the completed handler that would have paged it.
• When a stalled job comes back to the processor and its record already settles this attempt, the recorded result is returned without starting a container, reserving anything or writing a new record.
• On a fleet with PI_WORKER_NAME, the run mirror in Valkey is the second place to look when the local file does not settle the attempt. Any fault there counts as no record, which keeps today's behaviour.
• A record only counts when its job id and attempt match and it started no earlier than five minutes before the job was created (the two times come from two hosts' clocks), so a reused job id never matches an old record in practice. When a stall failure finds a record it does not accept, job_lost_lock_record_rejected names why (unreadable, id-mismatch, other-attempt, failed or older-than-job) and where it was read, so a comment that is still posted can be explained.

Specs

AMENDED: DES-TERMINAL-COMMENTS-AND-FAILURE-HOOK (the residual that accepted one duplicate is replaced by the record check; what remains is named: a fleet host that exits before Valkey returns or another host failing the job before the writer reconnects, clock skew beyond the tolerance, a run that dies between its comment and its record, and a suppressed job still listed in BullMQ's failed set while its record says completed), CONST-RETRY-INFRA-ONLY (Why), REQ-LOCAL-JOB-VISIBILITY (the new terminal line). UNCHANGED, checked: INT-RUN-HISTORY-FILE-CONTRACT, INT-ON-FAILURE-HOOK-CONTRACT.

Tests

The listener cases (suppressed, attempt mismatch, a failed record, another failure reason, the mirror fallback, the policy page), the processor short circuit, the record reader never throwing, BullMQ's stall reason pinned against the installed source, and a live Valkey test that deletes a running job's lock for a plain and a scheduled job and proves exactly one run, no failure comment and no page. Each rule was checked by reverting it and watching its test fail.