fix(hgraph): relax Tune source coverage

[jac0626]

Change Type

• Bug fix
• New feature
• Improvement/Refactor
• Documentation
• CI/Build/Infra

Linked Issue

• Fixes: [bug](hgraph): allow Tune to use incomplete FP32 sources #2718

What Changed

• Select the Tune source with direct raw, precise FP32, then base FP32 checks; only fall through when the current candidate is absent or empty.
• Do not require the selected source to cover the published ID range.
• Preserve the pre-fix(hgraph): tune from available fp32 codes #2466 training count and rebuild the complete published HGraph ID range.
• Add a regression test for an Add submission failure that leaves HGraph metadata ahead of codes.

Test Evidence

• Other (describe below)

Test details:

clang-format-15 --dry-run --Werror src/algorithm/hgraph/hgraph.cpp src/algorithm/hgraph/hgraph_add_test.cpp
git diff --check
cmake --build build --target unittests functests --parallel 6
./build/tests/unittests '[hgraph_tune_incomplete_source]' --allow-running-no-tests
./build/tests/unittests '[tune]' --allow-running-no-tests
./build/tests/functests '[tune_codes]' --allow-running-no-tests

Results: 8 regression assertions, 7 deduplicated-Tune assertions, and 8040 Tune-source assertions passed.

Compatibility Impact

• API/ABI compatibility: none
• Behavior changes: Tune no longer rejects an otherwise available raw/FP32 source solely because its logical count is behind the published HGraph count. Training and rebuilding retain the pre-fix(hgraph): tune from available fp32 codes #2466 ranges.

Performance and Concurrency Impact

• Performance impact: consistent indexes are unchanged; inconsistent indexes rebuild the full published ID range, matching legacy Tune behavior.
• Concurrency/thread-safety impact: none; Tune keeps its existing Add serialization.

Documentation Impact

• No docs update needed
• Updated docs:
  • README.md
  • DEVELOPMENT.md
  • CONTRIBUTING.md
  • Other:

Risk and Rollback

• Risk level: medium
• Residual risk: Add-state consistency and invalid individual slots remain outside this PR.
• Rollback plan: revert these commits to restore strict source coverage rejection.

Checklist

• I have linked the relevant issue.
• I have added/updated tests for the bug fix.
• I have considered API compatibility impact.
• I have updated docs if behavior/workflow changed.
• My commit messages follow project conventions.

[vsag-bot]

/label status/waiting-for-review
/waiting-on reviewer
/request-review @jiaweizone
/request-review @wxyucs
/request-review @inabao
/request-review @LHT129

[mergify]

Merge Protections

🟢 All 3 merge protections satisfied — ready to merge.

Show 3 satisfied protections

🟢 Require kind label

• label~=^kind/

🟢 Require version label

• label~=^version/

🟢 Require linked issue for feature/bug PRs

• body~=(?im)(?:^|[\s\-\*])(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s*:?\s+(?:#\d+|[\w.\-]+/[\w.\-]+#\d+|https?://github\.com/[\w.\-]+/[\w.\-]+/issues/\d+)

[Copilot]

Pull request overview

This PR addresses an HGraph Tune failure mode where total_count_ (and ID metadata) can be ahead of the available FP32/raw codes after a failed Add, and Tune should still proceed by rebuilding only the readable prefix.

Changes:

• Adjust Tune’s source selection to pick the available raw/FP32 source with the largest available prefix.
• Bound Tune training and code rebuilding by the selected source’s available count (rather than the published HGraph count).
• Add a regression test covering Add submission failure leaving HGraph metadata ahead of codes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/algorithm/hgraph/hgraph.cpp	Relaxes Tune’s source selection and limits rebuild/training to the selected source’s available count.
src/algorithm/hgraph/hgraph_add_test.cpp	Adds a regression test using a rejecting thread pool to reproduce Add failure followed by a successful Tune.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jac0626]

/retest

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/algorithm/hgraph/hgraph_add_test.cpp:60

• ArmableRejectingThreadPool::Enqueue returns a default-constructed std::future<void>, which is invalid. If any code path calls Enqueue directly and later waits/gets on the returned future, this will throw std::future_error. Return a ready future instead to satisfy the ThreadPool contract.

        if (reject_submissions_.load(std::memory_order_acquire)) {
            throw std::bad_alloc();
        }
        task();
        return {};

src/algorithm/hgraph/hgraph.cpp:214

• The fallback tune-source selection (second pass with require_coverage=false) always picks the first available source (raw/high_precise/basic) rather than the one with the largest readable prefix. This contradicts the PR/issue requirement to choose the available raw/FP32 source with the largest readable prefix, and can also pick a shorter source even when a longer FP32 source is available.

        auto select_tune_source = [&](bool require_coverage) {
            auto select = [&](const FlattenInterfacePtr& codes, bool require_fp32) {
                if (tune_source != nullptr or codes == nullptr or
                    (require_fp32 and codes->GetQuantizerName() != QUANTIZATION_TYPE_VALUE_FP32) or
                    (require_coverage and not covers_active_ids(codes))) {

src/algorithm/hgraph/hgraph.cpp:274

• current_count is loaded from total_count_ (the published HGraph count), but when Tune falls back to an incomplete source, decode_tune_source(i, ...) will still be called for i >= tune_source->TotalCount() during both training (for i < train_count) and rebuilding (for i < current_count). In the Add-failure scenario this PR targets, GetCodesById can return nullptr and Tune will throw an INTERNAL_ERROR instead of rebuilding the readable prefix. Tune should bound both the training sample loop and the rebuild loop by min(published_count, tune_source->TotalCount()).

            for (int64_t i = 0; i < current_count; ++i) {
                decode_tune_source(i, insert_buffer.data());
                new_code->InsertVector(static_cast<const void*>(insert_buffer.data()), i);
            }

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

src/algorithm/hgraph/hgraph.cpp:270

• Rebuild currently iterates [0, current_count) even when the chosen tune_source is a fallback that may only have a shorter readable prefix. For FlattenDataCell + MemoryIO, GetCodesById reads from the resized buffer without validating writes, so this can decode uninitialized memory and silently rebuild corrupted vectors.

When using an incomplete source, cap the rebuild loop to the source’s readable count (e.g., TotalCount()).

            Vector<float> insert_buffer(dim_, 0, allocator_);
            for (int64_t i = 0; i < current_count; ++i) {
                decode_tune_source(i, insert_buffer.data());
                new_code->InsertVector(static_cast<const void*>(insert_buffer.data()), i);
            }