Optimize cuvs-lucene CAGRA_HNSW index build; add example - #2481

Open

jamxia155 wants to merge 49 commits into

NVIDIA:mainfrom

jamxia155:cuvs-lucene-cagra-hnsw-build-optimization-with-example

Commits on Aug 18, 2026

cuvs-lucene__139: This code allows us to construct the HNSW graph on GPU without loading the full set of data on the Java Heap, but instead allows us to stream the set of data to the Java Heap.

EC2 Default User
authored and
jamxia155
committed
c579e87
Consolidate duplicated HNSW graph and field-writing methods

nvzm123
authored and
jamxia155
committed
8598e4f
Add example: chunked sequential ingestion of large .fbin files
jamxia155
committed
f66bdce
Parallelize level-0 HNSW graph serialization in writeGraph
jamxia155
committed
d0b5eb9
Add native flat buffering for single-segment CAGRA_HNSW builds
jamxia155
committed
cd23490
Parallelize CAGRA-to-HNSW conversion in GPUBuiltHnswGraph
jamxia155
committed
3ad0d47
Add prefetching + reused-array to the fbin ingest example
jamxia155
committed
948b41d
Honor cagraGraphBuildAlgo override in HEURISTIC strategy
jamxia155
committed
33c85a2
Expand the fbin ingest example into a full optimized CAGRA-HNSW build
jamxia155
committed
1b9b9d8
Expand OptimizedCagraHnswBuildExample and add early index-sort check
jamxia155
committed
56bc260
Fix merge errors
jamxia155
committed
8ffba09
Add a Lucene-version tripwire and round-trip test for NativeFlatVectorsWriter
jamxia155
committed
cfd3440
Add equivalence test for writerThreads-parallelized graph construction/write
jamxia155
committed
4b4a0d7
Add functional coverage for native flat buffering (numInputVectors)
jamxia155
committed
64ee08d
Fix merge-time vector-count bug causing intermittent EOF during search
jamxia155
committed
22e10fe
Restore unintended M-derivation change in createMultiLayerHnswGraph
jamxia155
committed
725fc9d
Restore unnecessary removals
jamxia155
committed
9faae91
Fix inconsistent handling of explicit CuVSIvfPqParams under HEURISTIC+IVF_PQ
jamxia155
committed
77cbc65
Guard against native flat buffering with quantized fields
jamxia155
committed
3ab2330
Merge remote-tracking branch 'origin/main' into cuvs-lucene-cagra-hnsw-build-optimization-with-example
jamxia155
committed
68245a1
Apply Spotless formatting
jamxia155
committed
563428c
Merge remote-tracking branch 'origin/main' into cuvs-lucene-cagra-hnsw-build-optimization-with-example
jamxia155
committed
b99dae2

Commits on Aug 21, 2026

Commits on Aug 23, 2026

Commits on Aug 25, 2026

Commits on Aug 26, 2026

Commits on Aug 27, 2026

Commits on Aug 28, 2026

Split native-flat-buffered writer out of Lucene99AcceleratedHNSWVectorsWriter
jamxia155
committed
96456b8
Fix RMM-enable race in CagraHnswBulkIndexWriter
jamxia155
committed
d706808
Document RMM enable-method ownership on CuVSProvider
jamxia155
committed
a1f34b3
Split NativeFieldWriter out of FieldWriter
jamxia155
committed
f8b5cd3
Fix native buffer leak on native flat buffering failure
jamxia155
committed
10af80e

Commits on Aug 29, 2026

Commits on Sep 1, 2026

Don't attempt to close the writer if we know that we cannot flush the buffer
Igor Motov
committed
2c3c9cd
Fix potential leak in AcceleratedHnswGraphOutput
Igor Motov
committed
73ada6c
Prevent dataset leaks for small datasets
Igor Motov
committed
325d107
Improve thread pool cleanup
Igor Motov
committed
d2ce349
Merge branch 'main' into cuvs-lucene-cagra-hnsw-build-optimization-with-example
imotov
authored
b225365