Skip to content

fix: bound AioContextPool wait and fail fast on empty pool - #1787

Open
aetosdios27 wants to merge 1 commit into
zilliztech:mainfrom
aetosdios27:fix/aio-context-pool-hang
Open

fix: bound AioContextPool wait and fail fast on empty pool#1787
aetosdios27 wants to merge 1 commit into
zilliztech:mainfrom
aetosdios27:fix/aio-context-pool-hang

Conversation

@aetosdios27

Copy link
Copy Markdown

issue: #1772

What

  • AioContextPool's constructor now throws diskann::ANNException if every io_setup() call fails, instead of publishing a pool with zero usable contexts.
  • pop()'s wait predicate now includes stop_, so the destructor's notify_all() can actually wake and release a caller blocked waiting for a context.
  • stop_ = true in the destructor is now guarded by ctx_mtx_, removing the data race with the waiter's predicate check.

Why

AioContextPool swallowed every io_setup() failure and could publish an empty pool. pop() then waited on ctx_q_.size() forever — a predicate that can never become true once no context was ever successfully allocated, since the only producer (push()) is reachable exclusively via a context obtained from pop() in the first place. Every DiskANN load/search in the process then hung permanently, with no error, no timeout, and — because stop_ wasn't part of the wait predicate — no way to unblock even at shutdown.

This is reachable in practice whenever the process's AIO budget (fs.aio-max-nr) is exhausted by a concurrent AIO user, which the default pool sizing makes easy to hit (512 contexts × 128 events == 65536, the default fs.aio-max-nr).

Fail-fast turns this into a deterministic, reportable error instead of a silent hang. AioContextPool::GetGlobalAioPool()'s constructor is only ever first invoked from LinuxAlignedFileReader's constructor inside IndexNode::Deserialize(), which is only ever called from Index<T>::Deserialize() — wrapped in GuardedCall, which catches std::exception (ANNException derives from std::runtime_error) and converts it to Status::knowhere_inner_error. So the new throw cannot escape a noexcept boundary; it surfaces as a normal load failure.

Not included in this PR, on purpose:

  • A wait_for timeout in pop() — a legitimately busy-but-healthy pool (all contexts checked out under real concurrency) would start failing spuriously. Fail-fast at construction already makes the reported repro fail deterministically without this.
  • Reworking the default pool sizing (512 × 128 == 65536) so it no longer consumes the entire default fs.aio-max-nr — that looks like the actual root cause of field triggers, but it's a wider behavior change than a bug fix and deserves its own PR.

Validation

  • Standalone concurrency check: copied the pool's actual constructor/pop()/destructor logic into an isolated harness with io_setup()/io_destroy() mocked out (no conan/faiss/folly/grpc needed). Confirms: (1) all-io_setup()-failures now throws instead of publishing an empty pool, (2) a pop() blocked on an empty queue is released with nullptr when the pool is destroyed, (3) normal push/pop is unaffected.
  • Negative control: reverted the harness to the original predicate (ctx_q_.size() without stop_) and destructor (unguarded stop_ = true) — reproduces the reported hang exactly: the whole process fails to exit and has to be killed by timeout, matching "destroying a condition_variable with a waiter still blocked is UB and the destructor may not return." The fixed version completes the same test in <1s.
  • pre-commit run — no-op; thirdparty/ is excluded from formatting (.pre-commit-config.yaml:20).
  • Full make WITH_UT=True project build and the issue's exact repro (holding fs.aio-max-nr via the C helper, then Indexing.SearchDiskAnnWithInvalidParam) — not completed; the from-scratch conan build (faiss/folly/grpc/openblas) didn't finish in a reasonable time in my sandbox. Traced every call site by hand instead to confirm the new throw can't escape a noexcept boundary (caught by GuardedCall in Index<T>::Deserialize). Recommend running this against CI or a local dev build before merge.

If every io_setup() call fails during construction, AioContextPool
previously published an empty pool. pop() then waited on a predicate
that could never become true, hanging every DiskANN load/search in
the process forever with no error and no way out (destructor's
notify_all() was ineffective since stop_ was not part of the wait
predicate, and was also written without holding the mutex).

Fail fast instead: throw if construction ends with zero usable
contexts, include stop_ in pop()'s wait predicate so shutdown can
free a blocked waiter, and guard stop_ with the mutex.

Fixes zilliztech#1772

Signed-off-by: aetosdios27 <aetosdios27@gmail.com>
@sre-ci-robot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aetosdios27
To complete the pull request process, please assign cqy123456 after the PR has been reviewed.
You can assign the PR to them by writing /assign @cqy123456 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot

Copy link
Copy Markdown
Collaborator

Welcome @aetosdios27! It looks like this is your first PR to zilliztech/knowhere 🎉

@aetosdios27

Copy link
Copy Markdown
Author

/kind bug

@sre-ci-robot sre-ci-robot added the kind/bug This PR is a bug fix label Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants