Repository navigation
feat: add nanobeir bench - #62
Conversation
 [Low impact] The PR adds an operator-run benchmark workflow without changing model behavior. The PR appears safe to merge, with non-blocking fixes recommended for hidden errors and reuse of old local models. Reviews (1) · Last reviewed commit: "add nanobeir bench" · Reviewed by Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if quantize == "model2vec-int8": | ||
| model, quantize = str(model2vec_int8(model)), "none" | ||
| cmd = [str(binary), f"data={data}", f"model={model}", f"quantize={quantize or 'none'}"] | ||
| out = subprocess.run(cmd, check=True, text=True, capture_output=True).stdout |
There was a problem hiding this comment.
build() and score() capture the child process output but never show it when the command fails. A compile error or model-loading failure leaves the user with only CalledProcessError, hiding why the benchmark stopped. Print the captured output before raising, or let errors reach the terminal.
| out = WORK / "models" / f"{model.replace('/', '--')}-int8" | ||
| if not (out / "model.safetensors").exists(): |
There was a problem hiding this comment.
model2vec_int8() reuses its saved model based only on the input path and the presence of model.safetensors. Replacing a local model at the same path and rerunning with :model2vec-int8 still scores the old saved copy. Check whether the source changed before reusing it, or rebuild local models on each run.
Adds a little NanoBEIR bench