diff --git a/docs/working-with-non-anthropic-models.md b/docs/working-with-non-anthropic-models.md new file mode 100644 index 0000000..d9b63e4 --- /dev/null +++ b/docs/working-with-non-anthropic-models.md @@ -0,0 +1,248 @@ +# Working with non-Anthropic models + +VulnHunter's skills (`/vulnhunt`, `/vulnhunter-fix`) run inside Claude Code. Claude +Code normally speaks the Anthropic Messages API, and gateways such as +[OpenRouter](https://openrouter.ai) expose an Anthropic-compatible endpoint. With a +few model overrides, we can run the same VulnHunter workflow against selected +third-party models (GLM, Kimi, DeepSeek, Gemma, Nemotron, OpenAI's open models, …). + +This doc explains **how** to do that and **which models actually work**. + +> TL;DR — in our evaluations as of 2026-07-28, the models we'd consider for a real scan +> are **Claude Opus 5** and **Opus 4.8** (native), **GLM-5.2**, and — on a single strong +> run — **Kimi K3** (the latter two via OpenRouter). Opus 5 found by far the most real +> vulnerabilities in absolute terms, at by far the highest cost; GLM-5.2 and Kimi K3 led +> on cost-effectiveness and, on the vulns all models were measured against, on +> severity-weighted coverage. Opus produced the most polished and consistent reports. No +> single run should be treated as complete. Other completed evaluations — including +> **Qwen 3.7 Max** — were noisy, incomplete, or non-starters. OpenAI frontier models +> have not yet completed the same evaluation. + +--- + +## How it works + +Before launching Claude Code, configure OpenRouter's Anthropic-compatible endpoint, +supply credentials, and pin both the main agent and subagents to the model under test. +The following shell commands follow OpenRouter's current +[Claude Code integration guide](https://openrouter.ai/docs/guides/coding-agents/claude-code-integration): + +```bash +export ANTHROPIC_BASE_URL="https://openrouter.ai/api" +export ANTHROPIC_AUTH_TOKEN="$(<"$HOME/.openrouter/api_key")" +export ANTHROPIC_API_KEY="" +export ANTHROPIC_MODEL="z-ai/glm-5.2" # <- the model under test +# ...other candidates commented out... +export ANTHROPIC_DEFAULT_OPUS_MODEL="$ANTHROPIC_MODEL" +export ANTHROPIC_DEFAULT_HAIKU_MODEL="$ANTHROPIC_MODEL" +export ANTHROPIC_DEFAULT_SONNET_MODEL="$ANTHROPIC_MODEL" +export CLAUDE_CODE_SUBAGENT_MODEL="$ANTHROPIC_MODEL" + +# Treat third-party and open-weight models as untrusted tool callers. Do not assume +# they provide Anthropic-equivalent safety behavior; deny arbitrary command execution. +claude --verbose \ + --allowedTools "Read,Write,Edit,Grep,Glob,Agent,AskUserQuestion" \ + --disallowedTools "Bash" +``` + +Three things to understand: + +1. **The main agent, aliases, and subagents are pinned separately.** `ANTHROPIC_MODEL` + selects the main model. The three `ANTHROPIC_DEFAULT_*` variables map Claude Code's + Opus, Sonnet, and Haiku aliases to that model. `CLAUDE_CODE_SUBAGENT_MODEL` forces + every subagent to use it as well; without that variable, subagent definitions or + per-invocation settings can select a different model. See Anthropic's + [model configuration reference](https://code.claude.com/docs/en/model-config). +2. **`Bash` is explicitly blocked for static mode.** `--allowedTools` only + pre-approves tools; omitting `Bash` from that list does **not** remove it. The + `--disallowedTools "Bash"` flag removes it from the model's toolset. This is a + containment measure, not just a convenience: third-party and open-weight models do + not necessarily have the same safety training, runtime safeguards, or + instruction-following behavior as Anthropic's models. An unexpected tool call or a + failure to follow VulnHunter's protocol must not turn a static scan into arbitrary + command execution. Blocking `Bash` also causes `/vulnhunt` to take its static path: + it writes exploit tests and PoCs but does not install dependencies or execute tests. + Remove the deny and add `Bash` to `--allowedTools` only when the model, target, and + execution environment are trusted. + + "Static" is more accurate than "read-only" here. `Write` and `Edit` remain + available so VulnHunter can create result artifacts, which means a misbehaving model + can still alter files. This configuration limits process execution but does not + enforce source-tree immutability or provide a complete security boundary. Use a + disposable checkout or an OS-level sandbox, grant access only to the target and + results paths, and inspect `git diff` after the run. +3. **`Agent` is load-bearing.** The pipeline fans out to 19–30 subagents via the + `Agent` (Task) tool. Without it, `/vulnhunt` cannot run its recon/hunt/verify/sweep + phases. Never drop `Agent` from the allow-list. + +## Quick start + +```bash +# 1. Store the OpenRouter key used by the launch configuration without placing it +# in shell history. +install -d -m 700 ~/.openrouter +read -rsp 'OpenRouter API key: ' router_key && printf '\n' +(umask 077; printf '%s\n' "$router_key" > ~/.openrouter/api_key) +unset router_key + +# 2. Pre-create the results directory and record the metadata VulnHunter cannot +# collect while Bash is blocked. +cd /path/to/target-repo +scan_dir="$PWD/$(basename "$PWD")_VULNHUNT_RESULTS_$(date '+%Y-%m-%d-%H%M%S')" +mkdir -p "$scan_dir" +printf 'Results: %s\nBranch: %s [%s]\nRepository: %s\n' \ + "$scan_dir" \ + "$(git branch --show-current 2>/dev/null || printf unknown)" \ + "$(git rev-parse --short HEAD 2>/dev/null || printf unknown)" \ + "$(git remote get-url origin 2>/dev/null || basename "$PWD")" +``` + +From the target repository, run the environment exports and `claude` command shown in +"How it works." If Claude Code has a cached Anthropic login, run `/logout` once, exit, +and relaunch with the same configuration. Then invoke: + +```text +> /vulnhunt in read-only mode, mock up your metadata, bypass model check +``` + +This invocation explicitly opts into the non-recommended model and tells VulnHunter to +bypass its interactive model gate. Because `Bash` is blocked, use the pre-created +results directory and repository metadata printed in step 2 if the model asks for +concrete values. + +Native Claude models (Opus 4.8, etc.) don't need the OpenRouter configuration — just +run `claude` normally; `/vulnhunt` already gates itself to Opus-class models by default. + +--- + +## Currently recommended models + +| Model | Access | Verdict | +|---|---|---| +| **Claude Opus 5** | native `claude` | **Recommended when breadth matters more than budget.** In one run it produced 32 adjudicated-real findings — more than double any other model — with 0.865 precision, no invalid citations, and the best recon/sweep/report stages we've scored. Most of that lead came from a large region of the codebase no other model searched; measured only on the vulnerabilities every model was scored against, it placed third. It cost **$110** (about 7× Kimi K3, and $3.44 per true positive — worse than Opus 4.8). Run it when you want maximum coverage in one pass and can absorb the cost. | +| **Claude Opus 4.8** | native `claude` | **Recommended for report quality and consistency.** Reliably completed the pipeline and produced the most polished reports. Its main weakness was recon coverage: both evaluated runs missed some of the most severe findings. It was also the most expensive option. | +| **GLM-5.2** | `z-ai/glm-5.2` (OpenRouter) | **Recommended for severity-weighted coverage and value.** It produced the strongest severe-vulnerability coverage at substantially lower cost than Opus, with generally strong precision. Run-to-run variance is the main caveat. | +| **Kimi K3** | `moonshotai/kimi-k3` (OpenRouter) | **Recommended, on a single run.** In one evaluation it led the field on true-positive count and cost-effectiveness with strong precision, and it **uniquely discovered two exploitable High-severity vulnerabilities no other model found** (a token path-prefix bypass and an ambient-credential clone). Caveats: only one run so far; it was slow (~5.5 h); and its severe-vuln credit came from findings it discovered rather than the previously-known set. A clear generational jump over Kimi k2.7-code (below) — do not confuse the two. | + +**Caveat on GLM (and any single run):** finding overlap between runs was low. The most +severe issue appeared consistently, but the secondary findings changed substantially. +The sample is too small to call that behavior reliable. For thoroughness, **run it 2–3 +times and union the results**. In our evaluation, repeated GLM runs still compared +favorably with a single Opus 4.8 run on both cost and severe-vulnerability coverage — and +two or three of them still cost less than half of one Opus 5 run. + +These recommendations come from a limited evaluation, not a general model benchmark. +Opus agents also participated in adjudicating the results while Opus was one of the +models under evaluation. Blind voting reduced that conflict but did not eliminate it; +see the full report's objectivity caveat. + +### Completed evaluations not recommended + +| Model | OpenRouter id | Why not | +|---|---|---| +| Kimi k2.7-code | `moonshotai/kimi-k2.7-code` | Completed, but was noisy and slow. It repeatedly treated trusted or operator-controlled inputs as attacker-reachable and was outperformed by GLM. **Superseded by Kimi K3 (recommended, above) — a distinct, much stronger model.** | +| Qwen 3.7 Max | `qwen/qwen3.7-max` | **Failed in practice.** Recon was strong, but the hunt stage mass-dismissed nearly every real finding: 1 true positive (plus 1 false positive) against 37 known vulnerabilities, missing all five High-severity ones — despite spawning 40 subagents and 1,100+ tool calls. It also self-mislabeled its own model in the report and declared the codebase "well-hardened." High effort, near-zero yield. | +| Nemotron-3-ultra | `nvidia/nemotron-3-ultra-550b-a55b` | Produced low-precision, incomplete output with invalid citations and internal inconsistencies. Not usable. | +| Gemma-4-31b | `google/gemma-4-31b-it` | **Failed.** Misunderstood the threat model, missed the real attack surface, and emitted a false all-clear. A null result presented as a clean bill of health is worse than no scan. | +| DeepSeek-v4-pro | `deepseek/deepseek-v4-pro` | **Failed.** Could not follow the pipeline instructions and produced no usable output. | + +Full data, methodology, and per-stage scoring: [`eval_rep/EVAL_REPORT.md`](../eval_rep/EVAL_REPORT.md) +and the live scorecard dashboard linked from it. + +--- + +## What makes a model "work" + +A model needs **all** of the following to run VulnHunter's pipeline usefully. The first +four are table-stakes capabilities; the last four are the ones our eval showed actually +separate the working models from the failures — a model can look capable on a benchmark +and still fail here on stamina, orchestration, consistency, or judgment. + +### Baseline capabilities + +1. **A 1M-class context window was necessary, but not sufficient, in our evaluation.** + Every model that both completed the pipeline and maintained acceptable + false-positive discipline had at least a 1M-token context window. Models with + smaller windows either produced substantial noise or failed the workflow. That is + an observed correlation, not proof that context size caused the difference; one + 1M-class model also produced low-precision, incomplete output. Treat 1M as a + screening requirement for the current pipeline, not as a guarantee of + instruction-following or security judgment. +2. **Strong instruction-following.** The pipeline is a strict multi-phase protocol with + ordered gates, disposition rules, and exact output-file naming. Models that improvise + (e.g. redefining the threat model) derail immediately — this is what sank Gemma. +3. **Multi-step reasoning.** Every finding is a data-flow trace from an + attacker-controlled source through assignments/calls/transforms to a dangerous sink. + That's sustained deductive chaining, not pattern-matching. +4. **Code understanding.** Must read the target languages, follow call graphs and + indirect dispatch, and reason accurately about API and runtime semantics. + +### Eval-derived requirements (where models actually fail) + +5. **Agentic subagent orchestration.** `/vulnhunt` dispatches 19–30 `Agent` subagents + (recon, per-partition INJ/NAV/LOG traces, verify, sweep) and must integrate their + structured results. This is the backbone of the pipeline. Strong models orchestrated + the full subagent set cleanly; weaker models stalled or collapsed early. If a model + can't reliably spawn, delegate to, and merge subagents, nothing else matters. +6. **Long-horizon stamina.** Real runs span six phases and hundreds of tool calls, often + taking well over an hour. Models must stay coherent to the end. Some evaluated + models stopped after early phases or failed to produce a final report. Completing + the pipeline at all is a real, discriminating bar. +7. **Output consistency & valid citations.** Every artifact must be internally + self-consistent — summary counts matching the number of PoC files, stable finding + IDs, and `file:line` citations that point at code that actually exists. Duplicate + IDs, count mismatches, or invalid citations make the output unsafe to trust and can + poison downstream automation such as `/vulnhunter-fix`. +8. **Trust-boundary judgment & false-positive discipline.** The security-specific + reasoning skill: distinguishing attacker-controlled input from operator/trusted + input, and dismissing non-issues *with cited evidence* rather than flooding the + report. The recommended models were substantially more precise than the weaker + candidates, which repeatedly treated trusted inputs as attacker-controlled. A + high-recall model with poor trust-boundary judgment is a false-positive generator, + not a scanner. + +**Rule of thumb:** if a model can't reliably orchestrate subagents (#5) and finish the +run (#6), it fails outright regardless of raw intelligence. If it can, then consistency +(#7) and trust-boundary judgment (#8) determine whether its output is trustworthy. + +--- + +## Roadmap + +- **OpenAI frontier models — planned, not yet evaluated.** The intended target is an + OpenAI frontier model, not the open `gpt-oss` weights. This is not just another model + slug in the OpenRouter configuration above: current approved access is path-specific + (for example, Responses API or Codex). We + expect the harness to be either Codex or one of the existing harnesses—Claude Code + or the Claude Agent SDK—connected through an Anthropic-compatible API adapter. The + chosen path must preserve VulnHunter's orchestration, tool, permission, and model + semantics before its results can be called apples-to-apples. The open + `openai/gpt-oss-120b` model is available separately, but it is a different model and + access path. Evaluate the frontier run with the same methodology once the harness is + selected and validated; until then, treat OpenAI results as unknown. + +To score a new model consistently with the existing evaluation, use the incremental +harness documented in +[`eval_rep/eval/README.md`](../eval_rep/eval/README.md): run the model with the chosen +launch configuration, then `add_run_workflow.js` → `merge_run.py` → `score.py` → +`make_dashboard.py`. + +## Gotchas + +- **Cost is not perfectly cross-provider comparable.** The recorded non-Anthropic + runs did not expose prompt-cache accounting comparable to Anthropic's, so their token + totals do not line up. Compare billed dollars for these runs, not raw token counts. +- **Static by default, not filesystem read-only.** The launch configuration denies + `Bash`, so exploit tests are written but not executed. "PASS (static)" means a + data-flow trace, not a live reproduction. `Write` and `Edit` are still available for + result artifacts. +- **One model, all tiers and subagents.** The `ANTHROPIC_DEFAULT_*` overrides pin the + aliases; `CLAUDE_CODE_SUBAGENT_MODEL` pins subagents. Keep both. +- **Authentication can conflict with a cached login.** OpenRouter currently recommends + `ANTHROPIC_AUTH_TOKEN` for its key and an explicitly empty `ANTHROPIC_API_KEY`. If + `/status` still shows Anthropic authentication, run `/logout`, exit, and relaunch. +- **Pin one model at a time.** Set `ANTHROPIC_MODEL` exactly once in the launch + configuration. If it is assigned more than once, the last value wins. +- **Prices and provider behavior move.** The costs above describe the recorded runs, + not current quotes. Record model IDs, provider routing, Claude Code version, and + timestamps with every new evaluation.