diff --git a/docs/docs/en/src/resources/autotune.md b/docs/docs/en/src/resources/autotune.md index dc720e26d5..f2160a2b6d 100644 --- a/docs/docs/en/src/resources/autotune.md +++ b/docs/docs/en/src/resources/autotune.md @@ -192,6 +192,34 @@ truth and defines per-query `recall_at_k` as the intersection of the returned an in their first `top_k` entries, divided by `top_k`; the reported value is the average over all queries. Ground truth is optional when recall is neither a constraint nor the objective. +Typed HGraph requests may reproduce a filtered workload by providing either one `FilterPtr` or one +exclusion `BitsetPtr` per query: + +```cpp +request.workload = {queries, filtered_ground_truth, 10, 48}; +request.workload.query_filters = query_filters; +// Or: request.workload.query_invalid_bitsets = query_invalid_bitsets; +``` + +The two vectors are mutually exclusive. An empty vector means that filter form is absent; otherwise +its size must equal the query count, and a null entry leaves that query unfiltered. A set bit in +`query_invalid_bitsets` excludes the corresponding external index ID. Bitsets must not be mutated +during tuning and should contain only IDs from the evaluated index. + +Every non-null `FilterPtr` is reused across candidates and may be called concurrently, so it must be +deterministic, reusable, and thread-safe. Its `ValidRatio()` must represent that query's actual +selectivity because HGraph uses it to choose search paths. This version uses +`CheckValid(int64_t)` and rejects `use_extra_info_filter=true`; the ID passed to `CheckValid` is the +index's external label. + +Each query must admit at least `top_k` IDs, and ground truth must be computed using its corresponding +filter or bitset. The returned parameters must be used with a representative filtered workload of +the same kind. `TuneIndex` supports the same HGraph workload; when either filter form is present and +`index_spaces` is omitted, only HGraph candidates are generated. Filters and bitsets are not stored +in the index, so the caller must still pass the applicable value to each final `KnnSearch`. +Caller-provided filters are available only through the typed API and are not serialized into CLI +requests or reports. IVF and Pyramid filtered tuning are not supported in this version. + The CLI's `index_path` field remains an offline adapter: it uses the concrete create parameters to create and deserialize an index, then enters the same search-only flow. @@ -203,14 +231,17 @@ unnamed hierarchy. To select different `ef_search` values for different paths, r AutoTune request per representative path workload, with the matching ground truth. V1 does not aggregate path-specific recommendations into one result. -A complete example is available at -[`examples/cpp/327_feature_autotune_existing_index.cpp`][existing-index-example]. +The `FilterPtr` example is available at +[`examples/cpp/327_feature_autotune_existing_index.cpp`][existing-index-example]. For direct +per-query exclusion bitsets, see +[`examples/cpp/330_feature_autotune_existing_index_bitset.cpp`][bitset-example]. For Pyramid path tuning, see [`examples/cpp/328_feature_autotune_existing_pyramid.cpp`][pyramid-example]. It tunes the same Pyramid index separately for 512-vector and 4096-vector leaf subgraphs under the same recall target, illustrating why different paths may need different `ef_search` values. [existing-index-example]: https://github.com/antgroup/vsag/blob/main/examples/cpp/327_feature_autotune_existing_index.cpp +[bitset-example]: https://github.com/antgroup/vsag/blob/main/examples/cpp/330_feature_autotune_existing_index_bitset.cpp [pyramid-example]: https://github.com/antgroup/vsag/blob/main/examples/cpp/328_feature_autotune_existing_pyramid.cpp ## Metrics @@ -260,5 +291,6 @@ result. ## V1 Boundaries V1 evaluates one KNN workload and performs a full sweep except for the supported HGraph -`ef_search` adaptive search. It does not provide filtered or range-search workloads, adaptive query -sampling, cross-request build cache, or model-based candidate generation. +`ef_search` adaptive search. The typed API supports per-query ID filters for HGraph; the CLI and +other index types do not yet support filtered workloads. V1 does not provide range-search +workloads, adaptive query sampling, cross-request build cache, or model-based candidate generation. diff --git a/docs/docs/zh/src/resources/autotune.md b/docs/docs/zh/src/resources/autotune.md index 378fa17ab3..098d8a8d45 100644 --- a/docs/docs/zh/src/resources/autotune.md +++ b/docs/docs/zh/src/resources/autotune.md @@ -182,6 +182,31 @@ if (result.has_value() && result->status == vsag::autotune::TuneStatus::SUCCESS) 返回结果和 ground truth 前 `top_k` 个 ID 的交集大小除以 `top_k`,最终指标是所有 query 的平均值。recall 既不是约束也不是目标时,ground truth 可省略。 +typed HGraph 请求可以为每条 query 提供一个 `FilterPtr`,或者一个表示排除集合的 +`BitsetPtr`,以复现带过滤的 workload: + +```cpp +request.workload = {queries, filtered_ground_truth, 10, 48}; +request.workload.query_filters = query_filters; +// 或:request.workload.query_invalid_bitsets = query_invalid_bitsets; +``` + +两种 vector 互斥。vector 为空表示未使用该种过滤方式;否则数量必须与 query 数量相同, +其中空指针表示该 query 不做过滤。`query_invalid_bitsets` 中置位的 bit 表示排除对应的索引 +外部 ID。调优期间不能修改 bitset,并且其中应只包含待评测索引中的 ID。 + +每个非空 `FilterPtr` 会跨候选重复使用,并且可能被并发调用,因此必须确定、可重复使用且 +线程安全。`ValidRatio()` 必须能代表该 query 的实际选择率,因为 HGraph 会用它选择搜索路径。 +本版本使用 `CheckValid(int64_t)` 并拒绝 `use_extra_info_filter=true`;`CheckValid` 收到的 ID +是索引的外部 label。 + +每条 query 必须至少允许 `top_k` 个 ID 通过,ground truth 必须使用对应 filter 或 bitset +生成;推荐参数应继续用于具有同类代表性过滤分布的 workload。`TuneIndex` 同样支持这种 +HGraph workload;存在任一种过滤输入且省略 `index_spaces` 时只生成 HGraph 候选。filter 和 +bitset 都不会保存到索引中,调用方在最终的每次 `KnnSearch` 中仍需传入对应值。调用方提供的 +过滤输入只支持 typed API,不会写入 CLI 请求或报告。本版本不支持 IVF 和 Pyramid 的 +filtered tuning。 + CLI 的 `index_path` 仍是离线适配:它先使用具体 create 参数创建并反序列化 Index,再进入 同一条 search-only 流程。 @@ -192,14 +217,17 @@ Pyramid 原生的默认/root 搜索。typed 请求直接从 `workload.queries->G 发起一次 typed AutoTune 请求,并提供与该 path 对应的 ground truth。V1 不把多个 path 的 推荐聚合成一条结果。 -完整示例见 -[`examples/cpp/327_feature_autotune_existing_index.cpp`][existing-index-example]。 +`FilterPtr` 完整示例见 +[`examples/cpp/327_feature_autotune_existing_index.cpp`][existing-index-example];直接传入每条 +query 的排除 bitset 见 +[`examples/cpp/330_feature_autotune_existing_index_bitset.cpp`][bitset-example]。 Pyramid path 调优示例见 [`examples/cpp/328_feature_autotune_existing_pyramid.cpp`][pyramid-example]。 它对同一个 Pyramid 索引中的 512-vector 和 4096-vector 叶子子图使用相同 recall 目标分别 调优,用于展示不同 path 可能需要不同的 `ef_search`。 [existing-index-example]: https://github.com/antgroup/vsag/blob/main/examples/cpp/327_feature_autotune_existing_index.cpp +[bitset-example]: https://github.com/antgroup/vsag/blob/main/examples/cpp/330_feature_autotune_existing_index_bitset.cpp [pyramid-example]: https://github.com/antgroup/vsag/blob/main/examples/cpp/328_feature_autotune_existing_pyramid.cpp ## 指标 @@ -245,5 +273,5 @@ typed `TuneIndex` 和 `TuneSearch` 绝不会写报告文件,而是通过返回 ## V1 边界 V1 评测一个 KNN workload;除已支持的 HGraph `ef_search` 自适应搜索外,其他候选仍完整遍历。 -它暂不支持过滤或范围查询 workload、query sampling、跨请求 build cache,以及基于模型的 -候选生成。 +typed API 支持 HGraph 的逐 query ID filter;CLI 和其他索引类型暂不支持 filtered workload。 +V1 仍不支持范围查询 workload、query sampling、跨请求 build cache,以及基于模型的候选生成。 diff --git a/examples/cpp/327_feature_autotune_existing_index.cpp b/examples/cpp/327_feature_autotune_existing_index.cpp index 79bc5bc1e1..c0ad9f7859 100644 --- a/examples/cpp/327_feature_autotune_existing_index.cpp +++ b/examples/cpp/327_feature_autotune_existing_index.cpp @@ -16,11 +16,35 @@ #include #include +#include #include #include #include "autotune.h" +namespace { + +class ParityFilter : public vsag::Filter { +public: + explicit ParityFilter(int64_t parity) : parity_(parity) { + } + + bool + CheckValid(int64_t id) const override { + return id % 2 == parity_; + } + + float + ValidRatio() const override { + return 0.5F; + } + +private: + int64_t parity_; +}; + +} // namespace + int main() { constexpr int64_t DIM = 4; @@ -57,6 +81,11 @@ main() { ->Dim(1) ->Ids(ground_truth_ids.data()) ->Owner(false); + std::vector filters; + filters.reserve(QUERY_COUNT); + for (int64_t i = 0; i < QUERY_COUNT; ++i) { + filters.emplace_back(std::make_shared(ground_truth_ids[i] % 2)); + } const std::string create_params = R"( { @@ -84,6 +113,7 @@ main() { vsag::autotune::SearchRequest request; request.index = index; request.workload = {queries, ground_truth, 1, 1}; + request.workload.query_filters = filters; request.parameter_space = R"({"hgraph":{"ef_search":[4,8,16]}})"; request.constraints = {{vsag::autotune::Metric::RECALL_AT_K, 1.0}}; request.objective = vsag::autotune::Metric::LATENCY_AVG_MS; @@ -110,7 +140,7 @@ main() { ->Dim(DIM) ->Float32Vectors(query_vectors.data()) ->Owner(false); - auto neighbors = index->KnnSearch(query, 1, result.parameters); + auto neighbors = index->KnnSearch(query, 1, result.parameters, filters[0]); if (!neighbors.has_value()) { std::cerr << "Search failed: " << neighbors.error().message << std::endl; return 1; diff --git a/examples/cpp/330_feature_autotune_existing_index_bitset.cpp b/examples/cpp/330_feature_autotune_existing_index_bitset.cpp new file mode 100644 index 0000000000..da49b1bc4f --- /dev/null +++ b/examples/cpp/330_feature_autotune_existing_index_bitset.cpp @@ -0,0 +1,132 @@ +// Copyright 2024-present the vsag project +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#include + +#include +#include +#include +#include +#include + +#include "autotune.h" + +int +main() { + constexpr int64_t DIM = 4; + constexpr int64_t BASE_COUNT = 16; + constexpr int64_t QUERY_COUNT = 4; + vsag::Options::Instance().set_block_size_limit(2UL * 1024 * 1024); + + std::vector base_ids(BASE_COUNT); + std::vector base_vectors(BASE_COUNT * DIM); + for (int64_t i = 0; i < BASE_COUNT; ++i) { + base_ids[i] = 1000 + i; + for (int64_t j = 0; j < DIM; ++j) { + base_vectors[i * DIM + j] = static_cast(i * DIM + j); + } + } + + std::vector query_vectors(base_vectors.begin(), + base_vectors.begin() + QUERY_COUNT * DIM); + std::vector ground_truth_ids(base_ids.begin(), base_ids.begin() + QUERY_COUNT); + + auto base = vsag::Dataset::Make() + ->NumElements(BASE_COUNT) + ->Dim(DIM) + ->Ids(base_ids.data()) + ->Float32Vectors(base_vectors.data()) + ->Owner(false); + auto queries = vsag::Dataset::Make() + ->NumElements(QUERY_COUNT) + ->Dim(DIM) + ->Float32Vectors(query_vectors.data()) + ->Owner(false); + auto ground_truth = vsag::Dataset::Make() + ->NumElements(QUERY_COUNT) + ->Dim(1) + ->Ids(ground_truth_ids.data()) + ->Owner(false); + + std::vector invalid_bitsets; + invalid_bitsets.reserve(QUERY_COUNT); + for (int64_t i = 0; i < QUERY_COUNT; ++i) { + auto invalid = vsag::Bitset::Make(); + for (const auto id : base_ids) { + invalid->Set(id, id % 2 != ground_truth_ids[i] % 2); + } + invalid_bitsets.emplace_back(std::move(invalid)); + } + + const std::string create_params = R"( + { + "dim": 4, + "dtype": "float32", + "metric_type": "l2", + "index_param": { + "base_quantization_type": "fp32", + "max_degree": 8, + "ef_construction": 40 + } + })"; + auto created = vsag::Factory::CreateIndex("hgraph", create_params); + if (!created.has_value()) { + std::cerr << "Failed to create index: " << created.error().message << std::endl; + return 1; + } + auto index = created.value(); + auto built = index->Build(base); + if (!built.has_value()) { + std::cerr << "Failed to build index: " << built.error().message << std::endl; + return 1; + } + + vsag::autotune::SearchRequest request; + request.index = index; + request.workload = {queries, ground_truth, 1, 1}; + request.workload.query_invalid_bitsets = invalid_bitsets; + request.parameter_space = R"({"hgraph":{"ef_search":[4,8,16]}})"; + request.constraints = {{vsag::autotune::Metric::RECALL_AT_K, 1.0}}; + request.objective = vsag::autotune::Metric::LATENCY_AVG_MS; + request.config.max_trials = 3; + + const auto tuned = vsag::autotune::TuneSearch(request); + if (!tuned.has_value()) { + std::cerr << "AutoTune failed: " << tuned.error().message << std::endl; + return 1; + } + if (tuned->status == vsag::autotune::TuneStatus::NO_FEASIBLE_CANDIDATE) { + std::cerr << "No candidate satisfied the constraints. Best effort:\n" + << tuned->best_effort.dump(2) << std::endl; + return 2; + } + + const auto& result = tuned.value(); + std::cout << "recommended search_params: " << result.parameters << '\n' + << "validated metrics: " << result.metrics.dump() << '\n' + << "trials evaluated: " << result.report["trials"].size() << std::endl; + + auto query = vsag::Dataset::Make() + ->NumElements(1) + ->Dim(DIM) + ->Float32Vectors(query_vectors.data()) + ->Owner(false); + auto neighbors = index->KnnSearch(query, 1, result.parameters, invalid_bitsets[0]); + if (!neighbors.has_value()) { + std::cerr << "Search failed: " << neighbors.error().message << std::endl; + return 1; + } + std::cout << "first neighbor id: " << neighbors.value()->GetIds()[0] << std::endl; + return 0; +} diff --git a/examples/cpp/CMakeLists.txt b/examples/cpp/CMakeLists.txt index ee8eec932c..4208b2f339 100644 --- a/examples/cpp/CMakeLists.txt +++ b/examples/cpp/CMakeLists.txt @@ -163,6 +163,10 @@ if (TARGET vsag::autotune) add_executable (328_feature_autotune_existing_pyramid 328_feature_autotune_existing_pyramid.cpp) target_link_libraries (328_feature_autotune_existing_pyramid vsag::autotune) + + add_executable (330_feature_autotune_existing_index_bitset + 330_feature_autotune_existing_index_bitset.cpp) + target_link_libraries (330_feature_autotune_existing_index_bitset vsag::autotune) endif () add_executable(329_feature_ivf_precise_bucket 329_feature_ivf_precise_bucket.cpp) diff --git a/tools/autotune/autotune.cpp b/tools/autotune/autotune.cpp index 239ed9ac1c..54e64bd7e1 100644 --- a/tools/autotune/autotune.cpp +++ b/tools/autotune/autotune.cpp @@ -419,6 +419,7 @@ make_context(eval::EvalDatasetPtr dataset, request.base_count = base_count; request.query_count = static_cast(request.dataset->GetNumberOfQuery()); request.ground_truth_k = request.dataset->GetGroundTruthK(); + request.has_query_filters = request.dataset->HasQueryFilters(); require(request.top_k > 0, "request.workload.top_k must be positive"); require(request.top_k <= static_cast(std::numeric_limits::max()), "request.workload.top_k is too large"); @@ -472,6 +473,10 @@ make_context(eval::EvalDatasetPtr dataset, {"keep_intermediate", request.keep_intermediate}, {"max_trials", request.max_trials}}}, {"output", {{"include_raw_evaluation", request.include_raw_eval}}}}; + if (request.has_query_filters) { + request.effective_request["workload"]["filtered_query_count"] = + request.dataset->GetFilteredQueryCount(); + } for (const auto& [name, value] : request.constraints) { request.effective_request["constraints"][name] = value; } @@ -524,8 +529,12 @@ ParseRequest(const IndexRequest& input) { const auto metric = normalize(input.metric_type); require(metric == "l2" || metric == "ip" || metric == "cosine", "request.metric_type must be l2, ip, or cosine"); - auto dataset = eval::EvalDataset::FromDatasets( - input.base, input.workload.queries, input.workload.ground_truth, metric); + auto dataset = eval::EvalDataset::FromDatasets(input.base, + input.workload.queries, + input.workload.ground_truth, + metric, + input.workload.query_filters, + input.workload.query_invalid_bitsets); IndexTuningRequest request; request.context = make_context(std::move(dataset), static_cast(input.base->GetNumElements()), @@ -536,7 +545,9 @@ ParseRequest(const IndexRequest& input) { input.config, false); if (input.index_spaces.empty()) { - request.indexes = {{"hgraph"}, {"ivf"}}; + request.indexes = request.context.has_query_filters + ? std::vector{{"hgraph"}} + : std::vector{{"hgraph"}, {"ivf"}}; } else { for (uint64_t i = 0; i < input.index_spaces.size(); ++i) { request.indexes.emplace_back(parse_index_space(input.index_spaces[i], i, false)); @@ -544,6 +555,8 @@ ParseRequest(const IndexRequest& input) { } const auto dim = static_cast(request.context.dataset->GetDim()); for (auto& index : request.indexes) { + require(not request.context.has_query_filters || index.name == "hgraph", + "filtered workloads currently support only hgraph"); merge_dataset_field(index.create_params, "dim", dim, index.name); merge_dataset_field(index.create_params, "dtype", vsag::DATATYPE_FLOAT32, index.name); merge_dataset_field(index.create_params, "metric_type", metric, index.name); @@ -560,8 +573,13 @@ ParseRequest(const SearchRequest& input) { require(input.index != nullptr, "request.index is required"); const auto element_count = input.index->GetNumElements(); require(element_count > 0, "request.index must not be empty"); - auto dataset = - eval::EvalDataset::FromSearchDatasets(input.workload.queries, input.workload.ground_truth); + const auto concrete_index_name = index_name(input.index); + auto dataset = eval::EvalDataset::FromSearchDatasets(input.workload.queries, + input.workload.ground_truth, + input.workload.query_filters, + input.workload.query_invalid_bitsets); + require(not dataset->HasQueryFilters() || concrete_index_name == "hgraph", + "filtered workloads currently support only hgraph"); SearchTuningRequest request; request.context = make_context(std::move(dataset), static_cast(element_count), @@ -572,7 +590,7 @@ ParseRequest(const SearchRequest& input) { input.config, true); IndexSpace space; - space.name = index_name(input.index); + space.name = concrete_index_name; space.search_parameter_space = input.parameter_space; request.index_input = parse_index_space(space, 0, true); request.index = input.index; diff --git a/tools/autotune/autotune.h b/tools/autotune/autotune.h index c89507fae5..9e8798736e 100644 --- a/tools/autotune/autotune.h +++ b/tools/autotune/autotune.h @@ -19,9 +19,11 @@ #include #include "nlohmann/json.hpp" +#include "vsag/bitset.h" #include "vsag/dataset.h" #include "vsag/errors.h" #include "vsag/expected.hpp" +#include "vsag/filter.h" #include "vsag/index.h" namespace vsag::autotune { @@ -59,6 +61,15 @@ struct Workload { uint64_t top_k{0}; /// Number of evaluator search threads. uint64_t concurrency{1}; + /// Optional per-query ID filters for HGraph. When non-empty, the vector must contain exactly + /// one entry per query; a null entry means that query is unfiltered. Filters may be invoked + /// concurrently and repeatedly during tuning, and ValidRatio() must describe their actual + /// selectivity. + std::vector query_filters; + /// Optional per-query HGraph exclusion bitsets. A set bit excludes the corresponding external + /// ID. This vector follows the same size and null-entry rules as query_filters, and the two + /// vectors are mutually exclusive. + std::vector query_invalid_bitsets; }; /// Evaluation options shared by index and search tuning. diff --git a/tools/autotune/autotune_candidate.cpp b/tools/autotune/autotune_candidate.cpp index b2957d3476..0691745922 100644 --- a/tools/autotune/autotune_candidate.cpp +++ b/tools/autotune/autotune_candidate.cpp @@ -291,6 +291,17 @@ supports_adaptive_ef_search_objective(const std::string& objective) { objective == "search_seconds" || objective == "build_and_search_seconds"; } +bool +uses_extra_info_filter(const JsonType& search_params) { + if (!search_params.contains("hgraph") || !search_params["hgraph"].is_object()) { + return false; + } + const auto& hgraph = search_params["hgraph"]; + return hgraph.contains("use_extra_info_filter") && + hgraph["use_extra_info_filter"].is_boolean() && + hgraph["use_extra_info_filter"].get(); +} + int64_t positive_int64(const JsonType& value, const std::string& path) { if (!value.is_number_integer()) { @@ -408,6 +419,11 @@ generate_candidates(const RequestContext& context, ? take_hgraph_ef_search_range(search_space, context) : std::nullopt; expand(search_space, [&](const JsonType& search_params) { + if (context.has_query_filters && uses_extra_info_filter(search_params)) { + throw std::invalid_argument( + "filtered workloads currently support only ID filters; " + "hgraph.use_extra_info_filter must be false"); + } Candidate candidate{index.name, create_params, search_params, ef_search_range}; JsonType identity{{"index_name", candidate.index_name}, {"create_params", candidate.create_params}, diff --git a/tools/autotune/autotune_internal.h b/tools/autotune/autotune_internal.h index 9355bdc504..73ee8f7a1b 100644 --- a/tools/autotune/autotune_internal.h +++ b/tools/autotune/autotune_internal.h @@ -55,6 +55,7 @@ struct RequestContext { uint64_t query_count{0}; uint64_t ground_truth_k{0}; bool enable_recall{false}; + bool has_query_filters{false}; bool keep_intermediate{false}; bool include_raw_eval{false}; }; diff --git a/tools/autotune/autotune_test.cpp b/tools/autotune/autotune_test.cpp index e0e5ef10aa..9786dc90d4 100644 --- a/tools/autotune/autotune_test.cpp +++ b/tools/autotune/autotune_test.cpp @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include @@ -308,6 +309,67 @@ struct MemoryFixture { vsag::DatasetPtr ground_truth; }; +class CountingAllowListFilter : public vsag::Filter { +public: + CountingAllowListFilter(std::vector valid_ids, float valid_ratio) + : valid_ids_(std::move(valid_ids)), valid_ratio_(valid_ratio) { + } + + bool + CheckValid(int64_t id) const override { + checks_.fetch_add(1, std::memory_order_relaxed); + return std::find(valid_ids_.begin(), valid_ids_.end(), id) != valid_ids_.end(); + } + + float + ValidRatio() const override { + return valid_ratio_; + } + + uint64_t + Checks() const { + return checks_.load(std::memory_order_relaxed); + } + +private: + std::vector valid_ids_; + float valid_ratio_; + mutable std::atomic checks_{0}; +}; + +std::vector> +make_query_filters(const MemoryFixture& fixture, uint64_t first_rank, uint64_t count) { + std::vector> filters; + filters.reserve(MemoryFixture::QUERY_COUNT); + for (int64_t query = 0; query < MemoryFixture::QUERY_COUNT; ++query) { + const auto begin = fixture.neighbors.begin() + query * MemoryFixture::GROUND_TRUTH_K + + static_cast(first_rank); + filters.emplace_back(std::make_shared( + std::vector(begin, begin + static_cast(count)), + static_cast(count) / static_cast(MemoryFixture::BASE_COUNT))); + } + return filters; +} + +std::vector +make_query_invalid_bitsets(const MemoryFixture& fixture, uint64_t first_rank, uint64_t count) { + std::vector bitsets; + bitsets.reserve(MemoryFixture::QUERY_COUNT); + for (int64_t query = 0; query < MemoryFixture::QUERY_COUNT; ++query) { + auto invalid = vsag::Bitset::Make(); + for (const auto id : fixture.base_ids) { + invalid->Set(id); + } + const auto begin = fixture.neighbors.begin() + query * MemoryFixture::GROUND_TRUTH_K + + static_cast(first_rank); + for (uint64_t i = 0; i < count; ++i) { + invalid->Set(begin[static_cast(i)], false); + } + bitsets.emplace_back(std::move(invalid)); + } + return bitsets; +} + } // namespace TEST_CASE("AutoTune candidate rules only fill missing fields") { @@ -885,6 +947,122 @@ TEST_CASE("AutoTune searches an in-memory existing index") { Catch::Matchers::ContainsSubstring("search_parameter_space.ivf is unsupported")); } +TEST_CASE("AutoTune evaluates per-query filters and bitsets for HGraph") { + vsag::Options::Instance().logger()->SetLevel(vsag::Logger::kOFF); + ScopedBlockSizeLimit block_size_limit(256UL * 1024); + ScopedPath search_workspace(temp_path("autotune-filtered-search-workspace")); + ScopedPath index_workspace(temp_path("autotune-filtered-index-workspace")); + MemoryFixture fixture; + constexpr uint64_t TOP_K = 3; + constexpr uint64_t FIRST_FILTERED_RANK = 3; + std::vector filtered_ground_truth_ids; + filtered_ground_truth_ids.reserve(MemoryFixture::QUERY_COUNT * TOP_K); + for (int64_t query = 0; query < MemoryFixture::QUERY_COUNT; ++query) { + const auto begin = + fixture.neighbors.begin() + query * MemoryFixture::GROUND_TRUTH_K + FIRST_FILTERED_RANK; + filtered_ground_truth_ids.insert( + filtered_ground_truth_ids.end(), begin, begin + static_cast(TOP_K)); + } + auto filtered_ground_truth = vsag::Dataset::Make() + ->NumElements(MemoryFixture::QUERY_COUNT) + ->Dim(TOP_K) + ->Ids(filtered_ground_truth_ids.data()) + ->Owner(false); + auto filters = make_query_filters(fixture, FIRST_FILTERED_RANK, TOP_K); + std::vector query_filters(filters.begin(), filters.end()); + + const std::string create_params = + R"({"dim":8,"dtype":"float32","metric_type":"l2","index_param":{)" + R"("base_quantization_type":"fp32","max_degree":8,"ef_construction":40,)" + R"("build_thread_count":2}})"; + auto created = vsag::Factory::CreateIndex("hgraph", create_params); + REQUIRE(created.has_value()); + REQUIRE(created.value()->Build(fixture.base).has_value()); + + vsag::autotune::SearchRequest search_request; + search_request.index = created.value(); + search_request.workload = {fixture.queries, filtered_ground_truth, TOP_K, 2}; + search_request.workload.query_filters = query_filters; + search_request.parameter_space = R"({"hgraph":{"ef_search":8,"brute_force_threshold":1.0}})"; + search_request.constraints = {{vsag::autotune::Metric::RECALL_AT_K, 1.0}}; + search_request.objective = vsag::autotune::Metric::LATENCY_AVG_MS; + search_request.config.workspace_path = search_workspace.Get(); + search_request.config.max_trials = 1; + + const auto search_result = vsag::autotune::TuneSearch(search_request); + REQUIRE(search_result.has_value()); + REQUIRE(search_result->status == vsag::autotune::TuneStatus::SUCCESS); + REQUIRE(search_result->metrics["recall_at_k"] == 1.0); + REQUIRE(search_result->report["request"]["workload"]["filtered_query_count"] == + MemoryFixture::QUERY_COUNT); + for (const auto& filter : filters) { + REQUIRE(filter->Checks() > TOP_K); + } + + const auto query_invalid_bitsets = + make_query_invalid_bitsets(fixture, FIRST_FILTERED_RANK, TOP_K); + search_request.workload.query_filters.clear(); + search_request.workload.query_invalid_bitsets = query_invalid_bitsets; + const auto bitset_search_result = vsag::autotune::TuneSearch(search_request); + REQUIRE(bitset_search_result.has_value()); + REQUIRE(bitset_search_result->status == vsag::autotune::TuneStatus::SUCCESS); + REQUIRE(bitset_search_result->metrics["recall_at_k"] == 1.0); + REQUIRE(bitset_search_result->report["request"]["workload"]["filtered_query_count"] == + MemoryFixture::QUERY_COUNT); + + auto extra_info_request = search_request; + extra_info_request.parameter_space = + R"({"hgraph":{"ef_search":8,"use_extra_info_filter":true}})"; + const auto extra_info_result = vsag::autotune::TuneSearch(extra_info_request); + REQUIRE_FALSE(extra_info_result.has_value()); + REQUIRE(extra_info_result.error().type == vsag::ErrorType::INVALID_ARGUMENT); + REQUIRE(extra_info_result.error().message == + "filtered workloads currently support only ID filters; " + "hgraph.use_extra_info_filter must be false"); + + auto index_request = fixture.Request(index_workspace.Get()); + index_request.workload.ground_truth = filtered_ground_truth; + index_request.workload.query_invalid_bitsets = query_invalid_bitsets; + index_request.index_spaces[0].create_parameter_space = + R"({"index_param":{"base_quantization_type":"fp32","max_degree":8,)" + R"("ef_construction":40,"build_thread_count":2}})"; + index_request.index_spaces[0].search_parameter_space = search_request.parameter_space; + index_request.constraints = {{vsag::autotune::Metric::RECALL_AT_K, 1.0}, + {vsag::autotune::Metric::BUILD_SECONDS, 1000.0}}; + index_request.config.max_trials = 1; + + const auto index_result = vsag::autotune::TuneIndex(index_request); + REQUIRE(index_result.has_value()); + REQUIRE(index_result->status == vsag::autotune::TuneStatus::SUCCESS); + REQUIRE(index_result->metrics["recall_at_k"] == 1.0); + REQUIRE(index_result->report["request"]["workload"]["filtered_query_count"] == + MemoryFixture::QUERY_COUNT); + + auto default_spaces = index_request; + default_spaces.index_spaces.clear(); + const auto parsed = vsag::autotune::internal::ParseRequest(default_spaces); + REQUIRE(parsed.indexes.size() == 1); + REQUIRE(parsed.indexes[0].name == "hgraph"); + + default_spaces.index_spaces = {{"ivf", "{}", "{}"}}; + REQUIRE_THROWS_WITH(vsag::autotune::internal::ParseRequest(default_spaces), + "filtered workloads currently support only hgraph"); + + search_request.workload.query_invalid_bitsets.pop_back(); + REQUIRE_THROWS_WITH(vsag::autotune::internal::ParseRequest(search_request), + "query_invalid_bitsets must contain exactly one entry per query"); + + search_request.workload.query_invalid_bitsets = query_invalid_bitsets; + search_request.workload.query_filters = query_filters; + REQUIRE_THROWS_WITH(vsag::autotune::internal::ParseRequest(search_request), + "query_filters and query_invalid_bitsets are mutually exclusive"); + + search_request.workload.query_invalid_bitsets.clear(); + search_request.workload.query_filters.pop_back(); + REQUIRE_THROWS_WITH(vsag::autotune::internal::ParseRequest(search_request), + "query_filters must contain exactly one entry per query"); +} + TEST_CASE("AutoTune uses default candidates for an in-memory IVF index") { vsag::Options::Instance().logger()->SetLevel(vsag::Logger::kOFF); ScopedBlockSizeLimit block_size_limit(256UL * 1024); @@ -975,6 +1153,11 @@ TEST_CASE("AutoTune searches an existing Pyramid index for one path workload") { input.constraints.pop_back(); input.workload.queries = fixture.queries; REQUIRE_NOTHROW(vsag::autotune::internal::ParseRequest(input)); + + auto filters = make_query_filters(fixture, 0, 3); + input.workload.query_filters.assign(filters.begin(), filters.end()); + REQUIRE_THROWS_WITH(vsag::autotune::internal::ParseRequest(input), + "filtered workloads currently support only hgraph"); } TEST_CASE("AutoTune writes concrete trials for an adaptive ef_search range") { diff --git a/tools/eval/case/search_eval_case.cpp b/tools/eval/case/search_eval_case.cpp index 389b796e6e..e93cf1b092 100644 --- a/tools/eval/case/search_eval_case.cpp +++ b/tools/eval/case/search_eval_case.cpp @@ -222,6 +222,20 @@ SearchEvalCase::do_knn_search() { return std::make_pair(std::move(query), query_vector); }; + auto search = [this, topk](const DatasetPtr& query, uint64_t query_id) { + const auto& invalid = this->dataset_ptr_->GetQueryInvalidBitset(query_id); + if (invalid != nullptr) { + return this->index_->KnnSearch( + query, static_cast(topk), config_.search_param, invalid); + } + const auto& filter = this->dataset_ptr_->GetQueryFilter(query_id); + if (filter != nullptr) { + return this->index_->KnnSearch( + query, static_cast(topk), config_.search_param, filter); + } + return this->index_->KnnSearch(query, static_cast(topk), config_.search_param); + }; + bool statistics_collected = false; for (auto& monitor : this->monitors_) { const bool is_latency_monitor = @@ -244,8 +258,7 @@ SearchEvalCase::do_knn_search() { auto i = static_cast(id) % query_count; auto query_and_vector = prepare_query(i); auto& query = query_and_vector.first; - auto [result, latency_ms] = MeasureSearch( - [&]() { return this->index_->KnnSearch(query, topk, config_.search_param); }); + auto [result, latency_ms] = MeasureSearch([&]() { return search(query, i); }); if (not result.has_value()) { search_failure.Record(result.error().message); continue; @@ -274,7 +287,7 @@ SearchEvalCase::do_knn_search() { auto query_and_vector = prepare_query(i); auto& query = query_and_vector.first; const void* query_vector = query_and_vector.second; - auto result = this->index_->KnnSearch(query, topk, config_.search_param); + auto result = search(query, i); if (not result.has_value()) { search_failure.Record(result.error().message); continue; @@ -306,7 +319,7 @@ SearchEvalCase::do_knn_search() { auto i = static_cast(id) % query_count; auto query_and_vector = prepare_query(i); auto& query = query_and_vector.first; - auto result = this->index_->KnnSearch(query, topk, config_.search_param); + auto result = search(query, i); if (not result.has_value()) { search_failure.Record(result.error().message); continue; diff --git a/tools/eval/eval_dataset.cpp b/tools/eval/eval_dataset.cpp index 780b421fbe..f0fc709c06 100644 --- a/tools/eval/eval_dataset.cpp +++ b/tools/eval/eval_dataset.cpp @@ -189,6 +189,24 @@ validate_offsets(const std::vector& offsets, } } +void +validate_query_filter_inputs(const vsag::DatasetPtr& queries, + const std::vector& query_filters, + const std::vector& query_invalid_bitsets) { + if (not query_filters.empty() and not query_invalid_bitsets.empty()) { + throw std::invalid_argument( + "query_filters and query_invalid_bitsets are mutually exclusive"); + } + const auto query_count = static_cast(queries->GetNumElements()); + if (not query_filters.empty() and query_filters.size() != query_count) { + throw std::invalid_argument("query_filters must contain exactly one entry per query"); + } + if (not query_invalid_bitsets.empty() and query_invalid_bitsets.size() != query_count) { + throw std::invalid_argument( + "query_invalid_bitsets must contain exactly one entry per query"); + } +} + } // namespace float @@ -241,7 +259,9 @@ EvalDatasetPtr EvalDataset::FromDatasets(const vsag::DatasetPtr& base, const vsag::DatasetPtr& queries, const vsag::DatasetPtr& ground_truth, - const std::string& metric_type) { + const std::string& metric_type, + const std::vector& query_filters, + const std::vector& query_invalid_bitsets) { if (base == nullptr || queries == nullptr) { throw std::invalid_argument("base and queries datasets are required"); } @@ -260,11 +280,14 @@ EvalDataset::FromDatasets(const vsag::DatasetPtr& base, throw std::invalid_argument( "ground_truth must contain one non-empty id row for every query"); } + validate_query_filter_inputs(queries, query_filters, query_invalid_bitsets); auto dataset = std::make_shared(); dataset->base_dataset_ = base; dataset->query_dataset_ = queries; dataset->ground_truth_dataset_ = ground_truth; + dataset->query_filters_ = query_filters; + dataset->query_invalid_bitsets_ = query_invalid_bitsets; dataset->vector_type_ = DENSE_VECTORS; dataset->train_data_type_ = vsag::DATATYPE_FLOAT32; dataset->test_data_type_ = vsag::DATATYPE_FLOAT32; @@ -319,7 +342,9 @@ EvalDataset::FromDatasets(const vsag::DatasetPtr& base, EvalDatasetPtr EvalDataset::FromSearchDatasets(const vsag::DatasetPtr& queries, - const vsag::DatasetPtr& ground_truth) { + const vsag::DatasetPtr& ground_truth, + const std::vector& query_filters, + const std::vector& query_invalid_bitsets) { if (queries == nullptr || queries->GetNumElements() <= 0) { throw std::invalid_argument("queries dataset is required and must not be empty"); } @@ -333,10 +358,13 @@ EvalDataset::FromSearchDatasets(const vsag::DatasetPtr& queries, throw std::invalid_argument( "ground_truth must contain one non-empty id row for every query"); } + validate_query_filter_inputs(queries, query_filters, query_invalid_bitsets); auto dataset = std::make_shared(); dataset->query_dataset_ = queries; dataset->ground_truth_dataset_ = ground_truth; + dataset->query_filters_ = query_filters; + dataset->query_invalid_bitsets_ = query_invalid_bitsets; dataset->vector_type_ = DENSE_VECTORS; dataset->train_data_type_ = vsag::DATATYPE_FLOAT32; dataset->test_data_type_ = vsag::DATATYPE_FLOAT32; diff --git a/tools/eval/eval_dataset.h b/tools/eval/eval_dataset.h index 1d117da0c4..991976074b 100644 --- a/tools/eval/eval_dataset.h +++ b/tools/eval/eval_dataset.h @@ -15,16 +15,20 @@ #pragma once +#include #include #include #include +#include #include "H5Cpp.h" #include "common.h" #include "nlohmann/json.hpp" #include "simd/basic_func.h" +#include "vsag/bitset.h" #include "vsag/constants.h" #include "vsag/dataset.h" +#include "vsag/filter.h" namespace vsag::eval { @@ -47,10 +51,15 @@ class EvalDataset { FromDatasets(const vsag::DatasetPtr& base, const vsag::DatasetPtr& queries, const vsag::DatasetPtr& ground_truth, - const std::string& metric_type); + const std::string& metric_type, + const std::vector& query_filters = {}, + const std::vector& query_invalid_bitsets = {}); static EvalDatasetPtr - FromSearchDatasets(const vsag::DatasetPtr& queries, const vsag::DatasetPtr& ground_truth); + FromSearchDatasets(const vsag::DatasetPtr& queries, + const vsag::DatasetPtr& ground_truth, + const std::vector& query_filters = {}, + const std::vector& query_invalid_bitsets = {}); static void Save(const EvalDatasetPtr& dataset, const std::string& filename); @@ -188,6 +197,36 @@ class EvalDataset { return query_dataset_ == nullptr ? nullptr : query_dataset_->GetPaths(); } + [[nodiscard]] const vsag::FilterPtr& + GetQueryFilter(uint64_t query_id) const { + static const vsag::FilterPtr no_filter; + return query_filters_.empty() ? no_filter : query_filters_[query_id]; + } + + [[nodiscard]] const vsag::BitsetPtr& + GetQueryInvalidBitset(uint64_t query_id) const { + static const vsag::BitsetPtr no_bitset; + return query_invalid_bitsets_.empty() ? no_bitset : query_invalid_bitsets_[query_id]; + } + + [[nodiscard]] bool + HasQueryFilters() const { + return GetFilteredQueryCount() > 0; + } + + [[nodiscard]] uint64_t + GetFilteredQueryCount() const { + const auto filter_count = + std::count_if(query_filters_.begin(), query_filters_.end(), [](const auto& filter) { + return filter != nullptr; + }); + const auto bitset_count = std::count_if( + query_invalid_bitsets_.begin(), query_invalid_bitsets_.end(), [](const auto& bitset) { + return bitset != nullptr; + }); + return static_cast(filter_count + bitset_count); + } + [[nodiscard]] const void* GetOneTrainById(int64_t id) const { if (train_ids_are_identity_) { @@ -381,6 +420,8 @@ class EvalDataset { vsag::DatasetPtr base_dataset_; vsag::DatasetPtr query_dataset_; vsag::DatasetPtr ground_truth_dataset_; + std::vector query_filters_; + std::vector query_invalid_bitsets_; std::unordered_map train_id_to_row_; std::vector sparse_train_; diff --git a/tools/eval/eval_dataset_test.cpp b/tools/eval/eval_dataset_test.cpp index 3bd2b3aab8..1f7fd6a423 100644 --- a/tools/eval/eval_dataset_test.cpp +++ b/tools/eval/eval_dataset_test.cpp @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -39,6 +40,32 @@ using vsag::SparseVector; using vsag::eval::EvalDataset; using vsag::eval::EvalDatasetPtr; +class ParityFilter : public vsag::Filter { +public: + explicit ParityFilter(int64_t parity) : parity_(parity) { + } + + bool + CheckValid(int64_t id) const override { + checks_.fetch_add(1, std::memory_order_relaxed); + return id % 2 == parity_; + } + + float + ValidRatio() const override { + return 0.5F; + } + + uint64_t + Checks() const { + return checks_.load(std::memory_order_relaxed); + } + +private: + int64_t parity_; + mutable std::atomic checks_{0}; +}; + EvalDatasetPtr BuildSparseDataset(bool with_token_sequences, bool all_empty = false) { // Build a tiny sparse dataset (3 train, 2 test) and optionally attach @@ -212,12 +239,29 @@ TEST_CASE("EvalDataset builds a query-only view for id recall", "[ut][eval_datas auto ground_truth = vsag::Dataset::Make()->NumElements(2)->Dim(2)->Ids(ground_truth_ids.data())->Owner(false); - auto dataset = EvalDataset::FromSearchDatasets(queries, ground_truth); + auto even_filter = std::make_shared(0); + std::vector filters{even_filter, nullptr}; + auto dataset = EvalDataset::FromSearchDatasets(queries, ground_truth, filters); REQUIRE(dataset->GetTrain() == nullptr); REQUIRE(dataset->GetTest() == query_vectors.data()); REQUIRE(dataset->GetNumberOfBase() == 0); REQUIRE(dataset->GetNumberOfQuery() == 2); REQUIRE(dataset->GetGroundTruthK() == 2); + REQUIRE(dataset->HasQueryFilters()); + REQUIRE(dataset->GetFilteredQueryCount() == 1); + REQUIRE(dataset->GetQueryFilter(0) == even_filter); + REQUIRE(dataset->GetQueryFilter(1) == nullptr); + REQUIRE(dataset->GetQueryInvalidBitset(0) == nullptr); + + auto invalid = vsag::Bitset::Make(); + invalid->Set(20); + std::vector bitsets{invalid, nullptr}; + auto bitset_dataset = EvalDataset::FromSearchDatasets(queries, ground_truth, {}, bitsets); + REQUIRE(bitset_dataset->HasQueryFilters()); + REQUIRE(bitset_dataset->GetFilteredQueryCount() == 1); + REQUIRE(bitset_dataset->GetQueryFilter(0) == nullptr); + REQUIRE(bitset_dataset->GetQueryInvalidBitset(0) == invalid); + REQUIRE(bitset_dataset->GetQueryInvalidBitset(1) == nullptr); int64_t result_ids[]{10, 99}; vsag::eval::SearchRecord record{ @@ -229,6 +273,17 @@ TEST_CASE("EvalDataset builds a query-only view for id recall", "[ut][eval_datas REQUIRE_THROWS_WITH(EvalDataset::FromSearchDatasets(nullptr, ground_truth), "queries dataset is required and must not be empty"); + filters.pop_back(); + REQUIRE_THROWS_WITH(EvalDataset::FromSearchDatasets(queries, ground_truth, filters), + "query_filters must contain exactly one entry per query"); + filters = {std::make_shared(1), nullptr}; + REQUIRE_NOTHROW(EvalDataset::FromSearchDatasets(queries, ground_truth, filters)); + bitsets.pop_back(); + REQUIRE_THROWS_WITH(EvalDataset::FromSearchDatasets(queries, ground_truth, {}, bitsets), + "query_invalid_bitsets must contain exactly one entry per query"); + bitsets.push_back(nullptr); + REQUIRE_THROWS_WITH(EvalDataset::FromSearchDatasets(queries, ground_truth, filters, bitsets), + "query_filters and query_invalid_bitsets are mutually exclusive"); } TEST_CASE("EvalDataset rejects overflowing ground-truth ID counts", "[ut][eval_dataset]") { @@ -430,7 +485,10 @@ TEST_CASE("EvaluateSearch validates inputs and propagates search errors", "[ut][ std::vector ground_truth_ids{0, 3}; auto ground_truth = vsag::Dataset::Make(); ground_truth->NumElements(2)->Dim(1)->Ids(ground_truth_ids.data())->Owner(false); - auto dataset = EvalDataset::FromDatasets(base, queries, ground_truth, "l2"); + auto even_filter = std::make_shared(0); + auto odd_filter = std::make_shared(1); + std::vector filters{even_filter, odd_filter}; + auto dataset = EvalDataset::FromDatasets(base, queries, ground_truth, "l2", filters); const std::string create_params = R"( { @@ -478,6 +536,23 @@ TEST_CASE("EvaluateSearch validates inputs and propagates search errors", "[ut][ REQUIRE(qps_only["index_info"].is_object()); REQUIRE(qps_only["index_info"].empty()); REQUIRE(omp_get_max_threads() == caller_thread_count); + REQUIRE(even_filter->Checks() > 0); + REQUIRE(odd_filter->Checks() > 0); + + auto first_invalid = vsag::Bitset::Make(); + auto second_invalid = vsag::Bitset::Make(); + for (const auto id : base_ids) { + first_invalid->Set(id, id != ground_truth_ids[0]); + second_invalid->Set(id, id != ground_truth_ids[1]); + } + std::vector bitsets{first_invalid, second_invalid}; + auto bitset_dataset = EvalDataset::FromDatasets(base, queries, ground_truth, "l2", {}, bitsets); + config.enable_qps = false; + config.enable_recall = true; + const auto bitset_result = vsag::eval::EvaluateSearch(index, bitset_dataset, config); + REQUIRE(bitset_result["recall_avg"].get() == 1.0); + config.enable_recall = false; + config.enable_qps = true; config.search_param = R"({"hgraph":{"ef_search":0}})"; REQUIRE_THROWS_WITH(