Skip to content

fix: stop the download queue from deadlocking after failed jobs - #271

Merged
mkb79 merged 1 commit into
masterfrom
fix/download-queue-deadlock
Aug 5, 2026
Merged

fix: stop the download queue from deadlocking after failed jobs#271
mkb79 merged 1 commit into
masterfrom
fix/download-queue-deadlock

Conversation

@mkb79

@mkb79 mkb79 commented Aug 5, 2026

Copy link
Copy Markdown
Owner

Closes #235, closes #239. Takes a first step on #256 — see the caveat below.

The bug

consume() re-raised on error when --ignore-errors was not given:

except Exception as e:
    logger.error(e)
    if not ignore_errors:
        raise          # leaves the `while True` loop
finally:
    QUEUE.task_done()

That does not just abort the job, it kills the consumer task. task_done() still ran for the current item, but the consumer never took another one. Once every --jobs consumer had died this way, await QUEUE.join() waited forever for items nobody would pick up.

Reproduced against the real consume()/QUEUE:

Scenario Before
-j 3, 3 failures first, then 5 good jobs hangs, 5 items stranded
-j 1, 1 failure, then 2 good jobs hangs
-j 3, 1 failure completes — two consumers survive
--ignore-errors completes — nothing re-raises

The default is -j 3, so three early failures were enough. That matches #235 ("a lot of Skip download and then suddenly audible hangs") and its offshoot #239.

The failure was invisible too: the exception sat in a dead task, and the asyncio.gather(..., return_exceptions=True) that would have collected it only runs after QUEUE.join() returns — which in this state never happens.

The fix

DownloadRun holds the abort event, the collected errors and the skipped count, shared by all consumers. The consumer never leaves its loop: on failure it records the error, and if --ignore-errors is not set it sets the abort event. Consumers then stop starting jobs and only acknowledge what is left, so QUEUE.join() can return. Downloads already in flight finish rather than being cancelled mid-write.

Waiting on QUEUE.join() alone was the deeper flaw — any consumer ending early strands the queue. drain_queue() now waits on the join and the consumers together, so a worker that dies for any reason is reported instead of hanging the run. --jobs also rejects values below 1; -j 0 previously queued work that no consumer would ever pick up.

This also makes --ignore-errors mean what its help text says. Before, its absence aborted nothing: one consumer died while the others kept downloading until they died too.

Exit code

A run in which a job raised now ends in AudibleCliException, which cli.main() maps to exit code 2 — with --ignore-errors as well.

⚠️ #256 is only partially addressed here. Failures that are only logged never reach DownloadRun and still exit zero: an unknown ASIN or title, a cover with no URL, and downloads that return a non-success Status (DownloadError, DownloadErrorStatusCode, DownloadSizeMismatch, DownloadContentTypeMismatch). Capturing those means threading the run state through every download function and separating real failures from legitimate DestinationAlreadyExists skips — worth its own PR, so #256 stays open. The changelog entry is worded to match what actually holds.

Relation to #257

#257 also targets #256 and overlaps in consume(), error tracking and the command epilogue, so the two should not be merged independently. It keeps the re-raise and therefore the deadlock, and its return 1 from the Click callback does not become the process exit status — Click discards callback return values in standalone mode. Its useful part is the additional error capture points, which are exactly what the #256 follow-up needs.

Verification

A harness driving the real consume()/QUEUE/drain_queue() — not a reimplementation:

  • all three previously hanging scenarios now drain completely
  • first failure stops the 20 queued jobs behind it; two slow in-flight jobs still run to completion
  • --ignore-errors runs everything and only counts the failures
  • an error-free run is unchanged and raises nothing
  • a consumer dying for an unrelated reason is reported rather than hanging
  • -j 0 and -j -1 are rejected by Click
  • AudibleCliException → exit 2 and RuntimeError → exit 3 confirmed through cli.main()

ruff check src plugin_cmds shows no new findings; cli's pre-existing C901 complexity drops from 32 to 31 because the draining moved out of it.

No tests were committed — the repo still has no pytest dependency and no CI job to run them. That gap is tracked separately.

`consume()` re-raised on error when `--ignore-errors` was not given. That
does not just abort the job, it leaves the `while True` loop and kills the
consumer task. `task_done()` still ran for the current item, but the consumer
never took another one. Once every `--jobs` consumer had died this way,
`await QUEUE.join()` waited forever for items nobody would pick up. With the
default `-j 3` three early failures were enough; with `-j 1` a single one.

The failure was also invisible: the exception sat in a dead task, and the
`asyncio.gather(..., return_exceptions=True)` that would have collected it
only runs after `QUEUE.join()` returns, which in this state never happens.

Keep the consumer alive and move the abort decision into a `DownloadRun`
object shared by all consumers. On the first failure it sets an event;
consumers then stop starting jobs and only drain what is left, so
`QUEUE.join()` can return. Downloads already in flight are allowed to finish
rather than being cancelled mid-write.

Waiting on `QUEUE.join()` alone was the deeper flaw: any consumer that ends
early strands the queue. `drain_queue()` now waits on the join and the
consumers together, so a worker that dies for any reason is noticed and
reported instead of hanging the run. `--jobs` also rejects values below `1`,
which previously queued work that no consumer would ever pick up.

This also makes `--ignore-errors` mean what its help text says. Before, the
flag's absence did not abort anything: one consumer died while the others
carried on downloading until they died too.

Finally, a run in which a job raised now ends in `AudibleCliException`, which
`cli.main()` maps to exit code 2. That holds with `--ignore-errors` as well,
so those failures are visible to scripts instead of being reported as
success. Failures that are only logged, such as an unknown ASIN or a download
rejected by its HTTP status, are not covered yet and still exit zero.

Closes #235
Closes #239
@mkb79
mkb79 merged commit 38efea3 into master Aug 5, 2026
@mkb79
mkb79 deleted the fix/download-queue-deadlock branch August 5, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cli does not exit correctly in some cases when finished Bug: Download Hangs

1 participant