Skip to content

fix: quarantine files on EXIT_NONZERO instead of aborting chunk - #1438

Open
shashwatMishra96 wants to merge 1 commit into
DeusData:mainfrom
shashwatMishra96:fix/quarantine-exit-nonzero
Open

fix: quarantine files on EXIT_NONZERO instead of aborting chunk#1438
shashwatMishra96 wants to merge 1 commit into
DeusData:mainfrom
shashwatMishra96:fix/quarantine-exit-nonzero

Conversation

@shashwatMishra96

Copy link
Copy Markdown

When a worker process exits with a nonzero code (e.g. internal parse-limit abort on a pathological file), treat it the same as CRASH/HANG: quarantine the offending file via the existing two-consecutive-strikes mechanism and continue indexing the rest of the chunk.

Previously, EXIT_NONZERO caused the supervisor to abort the entire chunk, losing all indexed data for that directory. This is impactful for large-scale repos (Android AOSP: 253GB, 60M+ files) where machine-generated files routinely trigger internal limits.

The two-consecutive-strikes intersection mechanism already guards against false quarantines — a systemic nonzero exit (e.g. bad CLI arg) produces no recurring suspect, so the intersection is empty and the supervisor gives up (same behavior as before).

Tested on Android QSSI17 (253GB, 24 chunks) + QCM6490 vendor (244 chunks). Result: 11.35M nodes indexed, 0 data loss, 2 genuinely pathological files quarantined (checkpatch.pl, a 140K-line generated .ts file).

What does this PR do?

Checklist

  • Every commit is signed off (git commit -s) — required, CI rejects
    unsigned commits (DCO, see CONTRIBUTING.md)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by a test (reproduce-first for bug fixes)

When a worker process exits with a nonzero code (e.g. internal parse-limit
abort on a pathological file), treat it the same as CRASH/HANG: quarantine
the offending file via the existing two-consecutive-strikes mechanism and
continue indexing the rest of the chunk.

Previously, EXIT_NONZERO caused the supervisor to abort the entire chunk,
losing all indexed data for that directory. This is impactful for
large-scale repos (Android AOSP: 253GB, 60M+ files) where machine-generated
files routinely trigger internal limits.

The two-consecutive-strikes intersection mechanism already guards against
false quarantines — a systemic nonzero exit (e.g. bad CLI arg) produces no
recurring suspect, so the intersection is empty and the supervisor gives up
(same behavior as before).

Tested on Android QSSI17 (253GB, 24 chunks) + QCM6490 vendor (244 chunks).
Result: 11.35M nodes indexed, 0 data loss, 2 genuinely pathological files
quarantined (checkpatch.pl, a 140K-line generated .ts file).

Signed-off-by: Shashwat Mishra <shashwat78.nits@gmail.com>
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant