Repository navigation
Conversation
run() terminated the single catalog writer by queueing `workers` sentinels on write_queue in the "no producers" branch, then computed writers_total = max(1, len(gov_workers_tasks)) -- which is 1 when there are no gov items. The writer breaks on the first sentinel it sees, so it exited before the trailing phases ran. VACINACAO items are deliberately deferred to a serial phase that pushes onto write_queue after the concurrent drain, and the retry pass does the same. With neither of those consumers alive, their entries had no reader: the parquet was uploaded to S3 but no catalog row was written and no SyncOutcome was emitted, so the artifact stayed invisible to PySUS. Once the bounded queue (maxsize=workers*2) filled, the put blocked forever and the run hung with no timeout. The branch was also a copy-paste of the one above it, which correctly sends sentinels to raw_queue to stop the processors. Sent the writer's single shutdown sentinel at the end of run(), right before awaiting the writer, and set writers_total to 1 to match the one consumer that actually exists. Test added for a VACINACAO-only run: it asserts every deferred item is reported as uploaded and reaches the catalog. It fails with 0 uploaded against the previous code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SyncEngine.run()has a single consumer ofwrite_queue— the serialcatalog_writercoroutine — which shuts itself down after seeingwriters_totalsentinels.The "no producers" branch queued the writer's shutdown sentinel too early:
writers_totalis1when there are no gov items, so the writer consumed one sentinel and exited. The second block is a copy-paste of the first, which is correct forraw_queue.Impact
VACINACAO items are deliberately deferred to a serial phase that runs after the concurrent drain and pushes onto
write_queue(as does the retry pass). With the writer already gone, those entries had no reader:SyncOutcomewas emitted — the artifact stayed invisible toPySUSand the run reported nothing happening.maxsize=workers*2) filled,putblocked forever. The run hung with no timeout.This affects any run whose pending items are all VACINACAO — e.g.
sync_clients.py --saude-only, orrun(datasets=["VACINACAO"]). The existing tests always pair VACINACAO with adadosgovitem, which keepsgov_workers_tasksnon-empty and hides the bug.Fix
The writer's single shutdown sentinel is now queued at the end of
run(), immediately before awaiting the writer, andwriters_totalis set to1to match the one consumer that exists.Test
test_run_vacinacao_only_syncs_every_fileruns a VACINACAO-only sync of 6 items and asserts all 6 are reporteduploadedand all 6 reach the catalog. It fails withassert 0 == 6against the previous code.1734 passed, 6 skippedblackandisortclean