Hemant/rebuild updated by hemant-endee · Pull Request #172 · endee-io/endee

hemant-endee · 2026-04-09T05:24:27Z

No description provided.

github-actions · 2026-04-09T05:25:24Z

VectorDB Benchmark — Dense — Passed

Triggered by @omnish-endee · Commit 4a3982f

Step	Status
Provision Servers	Up
Deploy Endee Server	Done
Run Benchmark	Passed
Results	See below
Teardown	Done ✓

No result file found on benchmark server.

hemant-endee · 2026-04-09T08:51:42Z

/correctness_benchmarking dense

omnish-endee · 2026-04-10T07:21:43Z

/correctness_benchmarking dense

omnish-endee · 2026-04-10T08:41:21Z

/correctness_benchmarking dense

omnish-endee · 2026-04-10T09:47:43Z

/correctness_benchmarking dense

github-actions · 2026-04-21T09:59:03Z

VectorDB Benchmark - Ready To Run

CI Passed ([lint + unit tests] (https://github.com/endee-io/endee/actions/runs/24984978794)) - benchmark options unlocked.

Post one of the command below. Only members with write access can trigger runs.

Available Modes

Mode	Command	What runs
Dense	`/correctness_benchmarking dense`	HNSW insert throughput · query P50/P95/P99 · recall@10 · concurrent QPS
Hybrid	`/correctness_benchmarking hybrid`	Dense + sparse BM25 fusion · same suite + fusion latency overhead

Infrastructure

Server	Role	Instance
Endee Server	Endee VectorDB — code from this branch	`t2.large`
Benchmark Server	Benchmark runner	`t3a.large`

Both servers start on demand and are always terminated after the run — pass or fail.

How Correctness Benchmarking Works

1. Post /correctness_benchmarking <mode>
2. Endee Server Create  →  this branch's code deployed  →  Endee starts in chosen mode
3. Benchmark Server Create  →  benchmark suite transferred
4. Benchmark Server runs correctness benchmarking against Endee Server
5. Results posted back here  →  pass/fail + full metrics table
6. Both servers terminated   →  always, even on failure

After a new push, CI must pass again before this menu reappears.

hemant-endee · 2026-04-21T10:18:36Z

/correctness_benchmarking dense

github-actions · 2026-04-21T10:18:55Z

VectorDB Benchmark — Dense — Failed

Triggered by @hemant-endee · Commit 117a81f

Step	Status
Provision Servers	Up
Deploy Endee Server	Done
Run Benchmark	Failed
Results	See reason below
Teardown	Done ✓

Benchmark job status: skipped

* Rebuild index with new config * fix 1 * index name in get stattu api correction * docs changes * Rebuild Status Persistence

…me as addVectors)

…y + IndexManager to executeJob instead of unpacking fields

shaleenji · 2026-04-24T12:42:16Z

+void Rebuild::cleanupTempFiles(const std::string& data_dir) {
+    if (!std::filesystem::exists(data_dir)) {
+        return;
+    }
+    try {
+        std::string temp_filename = std::string(settings::DEFAULT_SUBINDEX) + ".idx.temp";
+        std::string ts_prefix     = std::string(settings::DEFAULT_SUBINDEX) + ".idx.";
+        for (const auto& entry : std::filesystem::recursive_directory_iterator(data_dir)) {
+            if (!entry.is_regular_file()) continue;
+            const std::string fname = entry.path().filename().string();
+            bool is_temp = (fname == temp_filename);
+            bool is_ts   = fname.size() > ts_prefix.size()
+                           && fname.substr(0, ts_prefix.size()) == ts_prefix
+                           && std::all_of(fname.begin() + ts_prefix.size(), fname.end(), ::isdigit);
+            if (is_temp || is_ts) {
+                std::filesystem::remove(entry.path());
+            }
+        }
+    } catch (const std::exception& e) {
+        LOG_WARN(1803, "rebuild", "Failed to cleanup temp files on startup: " << e.what());
+    }
+}


should not throw error if there are not cleanup files left in the directory.

Fixed. Narrowed catch to std::filesystem::filesystem_error so it only triggers on real filesystem errors, not unrelated exceptions.

shaleenji · 2026-04-26T12:34:02Z

+        std::filesystem::copy_file(p.timestamped_path, p.index_path,
+            std::filesystem::copy_options::overwrite_existing);


why is renaming the index file not enough here ?

Fixed. Replaced copy_file + remove with std::filesystem::rename

shaleenji · 2026-04-26T12:37:15Z

+        std::filesystem::copy_file(p.timestamped_path, p.index_path,
+            std::filesystem::copy_options::overwrite_existing);
+
+        // Cannot call reloadIndex() here — we hold operation_mutex and reloadIndex acquires


st.stop_requested() is not checked in phase 3. makes the shutdown time unbounded for large indexes.

Fixed. Added stop_requested() check after Phase 2 loop, before the Phase 3 saveIndex call.

shaleenji · 2026-04-26T12:40:19Z

+        new_alg->saveIndex(p.timestamped_path);
+        std::filesystem::copy_file(p.timestamped_path, p.index_path,
+            std::filesystem::copy_options::overwrite_existing);
+
+        // Cannot call reloadIndex() here — we hold operation_mutex and reloadIndex acquires
+        // indices_mutex_, while deleteIndex holds indices_mutex_ then acquires operation_mutex.
+        // Calling reloadIndex here would deadlock with a concurrent delete on the same index.
+        auto fresh_alg = std::make_unique<hnswlib::HierarchicalNSW<float>>(p.index_path, 0);
+        IndexManager::wireVectorFetchers(fresh_alg.get(), entry->vector_storage);


new_alg already has all the data and was just persisted.

can use entry->alg = std::move(new_alg) directly.

doesnt seem useful to re load again

Fixed. Removed disk reload. new_alg is fully built and fetchers are already wired in Phase 2 — moved directly into entry->alg.

shaleenji · 2026-04-26T12:42:42Z

@@ -0,0 +1,124 @@
+# Index Rebuild


Things that are missing in the doc. make it more comprehensive

Disk space: rebuild needs 2× the index size on disk (timestamped + canonical co-exist briefly).

Memory: 2× the graph in RAM during Phase 2 (old + new).

Expected duration: rough order of magnitude per million vectors helps capacity planning.

What "manually restart" means after a crash — operationally how does an operator know a rebuild was incomplete? (There's no persisted state.)

Fixed. Added Capacity and Timing section covering disk space (2×), memory (2×), and expected duration.

shaleenji · 2026-04-26T12:43:29Z

+std::string Rebuild::formatTime(std::chrono::system_clock::time_point tp) {
+    return timeToISO8601(tp);
+}


no one is using this function

Fixed. Removed formatTime. Call sites in getProgress now call timeToISO8601 directly.

shaleenji · 2026-04-26T12:44:50Z

+    rebuild_.setActiveRebuild(username, index_id, current_count);
+
+    // Spawn thread — lambda calls rebuild_.executeJob directly (execution lives in Rebuild)
+    std::jthread t([this, params = std::move(params)](std::stop_token st) mutable {


here mutable is not required. executeJob takes const RebuildJobParams&, so the lambda doesn't actually mutate

Fixed. Removed mutable — executeJob takes const RebuildJobParams& so the lambda never mutates.

shaleenji · 2026-04-26T12:46:25Z

+        case RebuildStatus::IN_PROGRESS: return "in_progress";
+        case RebuildStatus::COMPLETED:   return "completed";
+        case RebuildStatus::FAILED:      return "failed";
+        default:                         return "unknown";


can never reach this line since RebuildStatus has only three defined values

Fixed. Removed default case. Added __builtin_unreachable() so compiler warns if a new enum value is added without updating the switch.

… doc review comments

shaleenji requested changes Apr 21, 2026

View reviewed changes

shaleenji reviewed Apr 23, 2026

View reviewed changes

Comment thread src/core/rebuild.hpp Outdated

shaleenji reviewed Apr 23, 2026

View reviewed changes

Comment thread src/main.cpp Outdated

shaleenji reviewed Apr 23, 2026

View reviewed changes

Comment thread src/core/rebuild.hpp Outdated

shaleenji requested changes Apr 23, 2026

View reviewed changes

Comment thread src/core/rebuild.hpp Outdated

Comment thread src/core/rebuild.hpp Outdated

Comment thread src/core/rebuild.hpp Outdated

hemant-endee added 8 commits April 23, 2026 16:37

Rebuild index with new config (#136)

85aded6

* Rebuild index with new config * fix 1 * index name in get stattu api correction * docs changes * Rebuild Status Persistence

using jthread with stop token

bb2f3d6

Shared parallel addPoint utility function — static chunk partition(sa…

cbaa744

…me as addVectors)

comments resolved

cb9f73d

rebuild also handles execute_rebuild_job

fd2c034

correction on cleantempfiles and error code

3b32a2e

rebuild status enum and logs code change

13fe6ce

rebuild.cpp

6f4083e

hemant-endee force-pushed the hemant/rebuild_updated branch from 117a81f to 6f4083e Compare April 23, 2026 16:09

Using Rebuild as friend class on Indexmanager helps to pass CacheEntr…

2202923

…y + IndexManager to executeJob instead of unpacking fields

shaleenji requested changes Apr 26, 2026

View reviewed changes

hemant-endee added 2 commits April 27, 2026 09:43

fix(rebuild): using new_alg directly, stop_request before phase3, and…

c2adbac

… doc review comments

test cases

af444b6

		std::filesystem::copy_file(p.timestamped_path, p.index_path,
		std::filesystem::copy_options::overwrite_existing);

Conversation

hemant-endee commented Apr 9, 2026

Uh oh!

github-actions Bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VectorDB Benchmark — Dense — Passed

Uh oh!

hemant-endee commented Apr 9, 2026

Uh oh!

omnish-endee commented Apr 10, 2026

Uh oh!

omnish-endee commented Apr 10, 2026

Uh oh!

omnish-endee commented Apr 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VectorDB Benchmark - Ready To Run

Available Modes

Infrastructure

How Correctness Benchmarking Works

Uh oh!

hemant-endee commented Apr 21, 2026

Uh oh!

github-actions Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VectorDB Benchmark — Dense — Failed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

github-actions Bot commented Apr 9, 2026 •

edited

Loading

github-actions Bot commented Apr 21, 2026 •

edited

Loading

github-actions Bot commented Apr 21, 2026 •

edited

Loading