Skip to content

QA.md

Latest commit

 

History

History
671 lines (624 loc) · 43.2 KB

QA.md

File metadata and controls

671 lines (624 loc) · 43.2 KB
Raw

QA Playbook — diffctx

Project-specific facts for /qa. Generic methodology lives in ~/.claude/qa-refs/.

Applicability matrix

Check Applies Notes
CI yes GitHub Actions on the mirror (ci.yml, action-smoke, CodeQL per push; cd.yml/publish-* manual dispatch; nightly-full-eval cron). "Rust diffctx tests" is ~6 min on a warm cargo cache; a Rust-touching commit invalidates it (+~10 min compile). Beyond ~30 min on "Build and test" = stuck runner — gh run cancel + gh run rerun --failed (an 85-min hang resolved to a 6-min green rerun with zero changes). A re-run stuck in queued can be API-unrecoverable: cancel AND force-cancel 409 with "re-run that has not yet queued", and an all-jobs-cancelled CodeQL run later reports "cannot be retried" — don't fight it; the next push supersedes, verify the corpus gate on the new SHA
Browser QA / stumble yes there IS a deployed surface: the GitHub Pages landing page (diffctx.com, a Pages custom domain since 2026-09-23; built by the pages build and deployment run on every push). This row read "no web frontend" until 2026-08-26 and it was simply wrong — the page is where a stranger decides whether to adopt the tool, which is exactly what the stumble probe measures. It is ONE page: only demo/ and llms.txt are additionally published; the docs/ subdirectories are Markdown that Jekyll renders on Pages too (product/token-budget.html, product/cli.html, product/faq.html answer 200 — verified 2026-09-16; this row claimed they 404 until then), and since #240 the nav links product/cli.html
PWA yes the landing page carries manifest.webmanifest, icons/ and sw.js (network-first, precached shell as the offline fallback; Pages has no build step, so nothing stamps a version and the worker never serves a live visit from cache). tests/test_publishing_manifests.py::TestLandingPagePwa pins the set. The Jekyll-rendered product/*.html pages get the worker's scope but not its registration
Post-deploy autoqa no sensor nothing runs the crawler against the Pages site, so the link sweep is done in-pass (fetch the page, follow same-origin links, HEAD every external). The fonts.googleapis.com / fonts.gstatic.com bare hosts are rel=preconnect hints and 404 on a bare GET — always false positives, never file them
CD / K8s / ArgoCD no ships to PyPI, npm, crates.io, Docker Hub via cd.yml/publish-extras.yml; those workflows' publish smokes are the probes
Backend smoke no —
SonarQube yes self-hosted sonar.nikolay-eremeev.com (WARP-only), project key diffctx; main only. Since 2026-09-28 the per-push sensor is gone: the autoqa-daily cron's sonar lane (00:15 Europe/Berlin) scans the Forgejo HEAD of that moment, so the gate and issue list lag a push by up to a day

Stumble probe tasks

Fixed wording — the number is only a trend if the task never moves. Alternate across passes; STUMBLE_GOAL is the Ultimate Goal sentence from CLAUDE.md, verbatim. STUMBLE_URL is the Pages landing page.

  1. (entry) "You have a git repository and a pull request that touches about 40 files. Using only this site, work out the exact command that would give you a review-ready context of that pull request, including how to point it at the right commit range and how to control how much output you get. Then decide, from evidence on this site, whether you would trust that output instead of reading the raw diff."
  2. (uncomfortable) "Decide whether this tool would beat simply pasting git diff into your model, and find the number on this site that settles it. Say what that number was measured on."

Forge

  • origin = Forgejo (source of truth, push here); github = mirror, but all CI, PRs and Dependabot live on GitHub, and the roadmap issues do too.
  • Forgejo issues are NOT empty: external reporters file there (it is the public-facing host). Enumerate both arms every pass — GET /repos/nikolay-e/diffctx/issues?type=issues&state=open. Triage on Forgejo, then cross-link to the GitHub issue that carries the work item and its gate, rather than duplicating the tracker: the Forgejo copy closes with that link and what was read live, and a report with no GitHub twin gets one first ("Reported on Forgejo #N, carried here as the work item"). Most of these reports come from the owner's own sessions on private repos, which sit on this machine: reproduce on the named repo read-only before calling one unreproducible, and keep repo names out of GitHub.
  • The redact-check PreToolUse hook blocks a tracker post whose command line mentions a deny-listed secret name — including the Keychain service name used to fetch the API token. Fetch the token in a separate Bash call (cache to a umask 077 scratch file, delete after), then post without naming the service.
  • Push main to both remotes (mirror sync is periodic; direct push is immediate).
  • Renovate auto-merges its Forgejo PRs (renovate.json: patch/minor automerge, squash) — possibly hours after opening them, so an "open" PR list is a snapshot, not a queue to hand-merge. When the whole fleet's bot PRs sit open past the next nightly run (2026-09-22: 82 across 16 repos), the stall is in gitops' shared Renovate preset, not here (2026-09-22 cause: with no status checks on the branch Renovate rates it yellow and never merges itself; the merges everyone saw were Forgejo's event-driven automerge firing on Monday rebases — ignoreTests: true in the preset is the fix) — hand it to the gitops session, then merge this repo's own by hand after checking each bump (ruff check at the new rev, cargo check on the bumped crate, the action SHA against git ls-remote --tags of the peeled tag). A pin that must NOT land needs a packageRules constraint (the python allowedVersions: <3.14 CI-matrix gate is the precedent), not a closed PR.
  • Forgejo's merge API can return 405 with an empty message on a merge that SUCCEEDED. Never read the HTTP code as the outcome — re-fetch the PR (state/merged) before retrying or escalating; a blind retry loop would report failure on a completed merge.
  • The v1.13.0 tag points at a commit containing unreleased work — the 2026-08-08 history rewrite re-anchored tags onto epoch commits, so git describe/tag position cannot tell you what a release contains. Released state is read from the registry (PyPI upload date/version), not from the tag.

Dependency channels (who owns which file)

Three lock/manifest pairs, and mixing them up is how a channel breaks silently:

  • pyproject.toml + uv.lock — every CI job installs with uv sync --locked --no-build. --locked refuses to re-resolve, so a pyproject edit landed without uv lock turns CI red instead of drifting; that IS the drift gate, there is no separate check. --no-build keeps every dependency on a wheel (no third-party setup.py runs); the root project is exempt and still compiles through maturin. Renovate owns this pair too since 2026-09-29 (0d75e73a removed Dependabot and automerge.yml); a Dependabot alert on a transitive pin (PyJWT, 2026-09-29) is closed by uv lock --upgrade-package <name>; plugin/uv.lock follows at the next release.
  • eval dependency group in pyproject.toml, locked in uv.lock — the research harness's Python deps. One resolver for the whole repo; the separate requirements-eval.txt + .lock pair (and its own bot) is gone.
  • Cargo.toml + Cargo.lock, GitHub Actions pins, pre-commit revs, Docker digests — Renovate on Forgejo (renovate.json, automerge). An epoch squash of main leaves every open Renovate branch based on a commit that no longer exists, and Renovate reads that as a manual edit: the Dependency Dashboard (Forgejo #4) lists them under "PR Edited (Blocked)" and no run touches them again (all ten sat on f047eeae for two days, 2026-09-30). Tick their rebase-branch boxes in the dashboard body after every rewrite; the next daily run rebuilds them on main. Never hand-merge a conflicted bot branch.
  • plugin/ is not Renovate's. cd.yml writes plugin/uv.lock once PyPI serves the release, seeded with uv.lock so it locks the set CI tested, and bumps the launcher pin in plugin/pyproject.toml with the version. Renovate bumped the former constraints.txt one package at a time (pydantic and pydantic-core as two PRs, either alone unresolvable); renovate.json disables plugin/**.
  • A Renovate automerge never waits for the GitHub CI. Forgejo carries no pre-commit/pytest run, so a bumped linter can land red on main: markdownlint 0.49.1 (3d33c528, 2026-09-15) tightened MD013 and failed the next two main runs on two CHANGELOG lines. A red "Pre-commit hooks" job right after a chore(deps): update pre-commit hook merge is that class — fix the files, not the pin.

Tests

  • pytest — integration-only (the run prints the count; don't pin it here, it rots). Known valid skip: test_clipboard.py (Windows only).
  • Local flake mode: -n auto on the M4 Pro means 14 xdist workers, each CLI subprocess spawning a full rayon pool — under load the global pytest-timeout (30 s since 27d6a3d6, was 10 s) fires on varying tests. Timeout-only failures with a changing set across runs = oversubscription, not a regression; never run cargo test concurrently with pytest, it makes this worse.
  • cargo test --lib in crates/diffctx-native — inline units; the run prints the count, and a count written here rotted once already.
  • YAML corpus: CI gates the FULL corpus on every push (the run prints the case count; pinning it here rotted — it read 2725 while the corpus had grown to 2902) (cargo test --release --test yaml_cases), per-case against known_below_threshold.txt, enforced bidirectionally; nightly re-runs with DIFFCTX_YAML_IGNORE_BASELINE=1 to track baseline size. DIFFCTX_YAML_CASES_LIMIT=20 sampling survives only in the local pre-commit hook.
  • Active-xfail semantics in the yaml harness: a case with an active xfail: (reason/category set) reports test "ok" while its ORACLE still fails — the marker absorbs the failure; test "FAILED" on such a case means the oracle now PASSES ("drop the xfail"). Bare xfail: {} is a no-op (is_active false, baseline gates instead) — remove on sight.
  • Rust edits fight the auto-format hook: the PostToolUse formatter orders use items differently from repo rustfmt; git commit then fails cargo-fmt. Run cargo fmt before every commit that touches Rust. Python edits hit the same trap: the hook applies isort's aligned-parens import style, which repo black rewrites back — run pre-commit run black --files <touched> after Python edits.
  • DIFFCTX_PROBE_TAU mirrors DIFFCTX_PROBE_MODE in the yaml harness: measurement-only tau override, meaningful only with DIFFCTX_YAML_IGNORE_BASELINE=1; default run still measures the shipped constant.
  • A test that reads the CLI's stdout decodes it as UTF-8 explicitly (encoding="utf-8" next to text=True): the CLI writes UTF-8 and Windows decodes with cp1252, so an assertion on — passes everywhere but the five Windows legs.
  • Python API fragments carry content/kind/lines/path/role/symbol only — no token_count. Integration tests measuring budget use must proxy via len(content) (~4 chars/token).
  • Version-bearing files: the CD bump patches every file listed in the cd.yml bump block — that block is the enumeration, incl. docs/product/github-action.md (pins gated by test_version_consistency). Adding a new file that embeds the version requires BOTH the cd.yml bump block and that test.
  • eval calibrate at 8 workers can reboot the machine (OS memory pressure when heavy-tail instances co-schedule; same class as the 11-13-worker sweep kills). Use 5 workers; checkpoints resume cleanly per cell.

Diff-context review

  • Every pull request in this repo carries its own diffctx review context (action-smoke.yml → pr-review-context: the published composite action run on the PR's base..head at --budget 8000, uploaded as the diffctx-review-context artifact and pinned to the PR as one sticky comment marked <!-- diffctx-review-context -->). A /qa pass that reviews a PR reads THAT artifact first and the raw diff only for what it withheld — that is the dogfood. Two things to judge each time, because they are product findings: whether the selection is what a reviewer of this change would want (a missing caller, a pulled-in irrelevant sibling, an omitted changed file at 8k), and whether the comment's numbers agree with a local run of the same range. The action installs the released wheel (diffctx-version default), so a PR that changes selection is reviewed by the version before it — compare against a source install when the diff touches the engine.

  • Review with the released package the plugin runs, pinned to the same version: uvx 'diffctx[mcp]==X.Y.Z' . --diff <range> --mode impact -f md (X.Y.Z = the version pinned in plugin/hooks/diffctx-impact.sh). The plugin diffctx@diffctx is the one diffctx on this machine: no uv tool, pipx or mise copy, which went stale silently and re-introduced fixed bugs into reviews (a 1.12.2 tool leaked .diffctx/ignore-excluded context; a source install's receipt made uv tool upgrade rebuild a dev build). A pinned uvx resolves the published wheel every time. .venv/bin/diffctx is the development build, never the reviewer; to review unreleased work, run it deliberately and say so.

  • Do not review with --budget -1. Selection plus the post-passes are ~97% of wall clock on a wide range (#121); unlimited removes the stop and every cell hits the 300s deadline. A wide range at the default budget is the reviewable artifact.

  • This repo's own .diffctx/ignore excludes *.yaml/*.yml (the corpus would drown every self-eat) and tests — so the entire tests/ tree, crates/diffctx-native/tests/, every oracle case and all CI/workflow YAML are invisible to self-eat, hidden even from changed_files by the security contract. A range that touches only those legitimately yields rc=4 and a bare skeleton. Concretely: a commit changing 33 files can show 9. Review test and workflow changes with plain git diff; don't file this as a bug, and don't read the short changed-files list as the whole change.

  • Reading a self-eat diff of a scoping change: total emitted lines and context-file count move in OPPOSITE directions. Shrinking fragments frees budget the greedy immediately spends admitting more files, so a genuine over-dump fix shows up as fewer lines AND more files. Measured across three ranges when bounded gap chunks landed: 717→656, 543→531, 582→535 lines, while context files went 4→8 and 16→17. Neither number alone is the verdict — this is why #149's gate pairs over-dump rate with precision. Two mechanisms drive the breadth half: freed budget, and apply_fragment recording the excerpt's identifiers rather than the whole core's, so needs the trimmed body used to cover read as uncovered and the greedy goes looking for them elsewhere.

  • Self-eat context breadth collapsed after the #211 post-pass/singleton gates (measured 2026-08-19: 61 visible changed files -> 85 output files, i.e. ~0.4 context files per changed file; the 2026-08-07 pre-gate figure was ~4-5x). A wide-context self-eat read is no longer designed behavior — treat a return to multi-x breadth as a regression of the #211 gates, not as the old known shape.

  • env_overrides.rs carries name-consistency tests: any new read_env_*("DIFFCTX_*") must appear in the parameter-strategy.md Tier-3 table (or the TIER1_EXTRAS_READ_BUT_NOT_TABLED allowlist) and vice versa.

  • The release gap is a channel of its own. main accumulating security fixes while PyPI serves the previous wheel is a user-facing state no test reports: on 2026-09-03 five ### Security bullets and the --budget contract fix had been unreleased since 1.15.0 (2026-08-19), and the dogfood Action installs that released wheel, so our own PR review ran the vulnerable version. Every pass reads curl -s https://pypi.org/pypi/diffctx/json | jq -r .info.version against the [Unreleased] changelog block and, when Security bullets sit there, says so in the report. Dispatching the release is the owner's call (memory no-release-without-asking); tracking it is not — #261.

  • The fleet carries this repo's own Action; sweep it, don't assume it. Fifteen sibling repos run .github/workflows/diffctx.yml (the consumer shape of action-smoke.yml's pr-review-context). On 2026-09-04 the sweep found it had never produced a context anywhere: six repos had runs and every one was skipped by the bot gate — ~98% of pull requests in this fleet are Dependabot — and nine had no runs at all. Every copy now carries a workflow_dispatch trigger, so a pass can exercise it without opening a PR: gh workflow run diffctx.yml -R nikolay-e/<repo> and check the run produced a diffctx-review-context artifact. Two facts worth keeping: the bot gate is right on cost but its comment used to claim diffctx returns empty on a dependency bump, which is false (measured on certamen#134: 4 fragments of package.json); and the action pins a diffctx release SHA, so every repo needs a bump when a release ships (#261).

Real usage is the oracle

The YAML corpus is a regression gate over mostly synthetic, LLM-written expectations and has no impact-mode cases, so a pass that judges impact runs it on real history: --mode impact over the last ~30 commits of a few sibling repos (read-only, DIFFCTX_NO_MARKER=1), each changed symbol's callers compared with git grep -w at that commit. Read every grep "miss" before calling it one. On 2026-10-06 almost all of them were the answer truncated at the 2k cap (truncated), files the consumer repo's own .diffctx/ignore withholds (withheld_files), or same-name methods. Two were real: a spread call read as a member access, and a silent zero for withheld callers. A failing corpus case gets the same question first: is the case right?

A missed caller has three causes that read alike, and two probes split them. RUST_LOG=diffctx=debug,_diffctx=debug prints, per changed symbol, impact <path>::<name>: kept N, dropped {reason: count}: no such line means the graph holds no reverse edge at all (a builder gap); a not_forward or binding_rejected count names the filter that dropped it. Re-running with DIFFCTX_MAX_EDGES_PER_NODE=100000 tells the per-node edge cap from both: a caller that appears only then was cut by the cap, which the answer does not report (#446). Copying the two files into a scratch repo is the third probe: found there and missed in the real tree means scale, not syntax.

The diff review's file list is checked by arithmetic, not by eye: changed_files + changed_files_omitted + deleted_files + withheld_files against git diff --name-only <range> | wc -l. Impact drops lock files from every count (#495), so a gap equal to the changed lock files is that, and any other gap is a dropped file.

Bug channels (enumerated 2026-08-05)

  1. GitHub issues (gh issue list) — canonical tracker, takes Fixes #N.
  2. Forgejo issues (API, token forgejo-token in Keychain) — external reporters; cross-link to GitHub work items. Forgejo #4 is Renovate's standing Dependency Dashboard — infrastructure, not a bug; never triage it as one.
  3. No in-app bug queue, no client telemetry, no bot reports (CLI/library product). PyPI/npm/crates user reports arrive via the two trackers.
  4. GitHub alert feeds (gh api repos/<r>/dependabot/alerts?state=open, code-scanning/alerts?state=open, secret-scanning/alerts?state=open) — all three were empty on 2026-08-30; a pass reads them, it does not assume them.
  5. SonarQube project diffctx on sonar.nikolay-eremeev.com (WARP-only; without a hosts entry use curl --resolve sonar.nikolay-eremeev.com:443:10.43.76.7), Bearer token from Keychain sonarqube-token: api/qualitygates/project_status, api/issues/search?componentKeys=…&resolved=false, api/hotspots/search?projectKey=…&status=TO_REVIEW. Issues raised on the day's own commits count as intake for that pass. The gate scores NEW code only, so older findings sit under a green gate; read the issue list, never the gate alone. Analysis is not part of diffctx CI on the GitHub mirror and, since 2026-09-28, not per push either: the autoqa-daily cron in argo-workflows (00:15 Europe/Berlin) scans the Forgejo HEAD (its inventory node lists the sha it took; scan:false = unchanged since the last analysis). api/project_analyses/search?project=diffctx names the analysed revision — compare it to HEAD before reading a verdict as current. A fix's issues stay open until the next morning's run. The per-push sensor went live on 2026-09-15 07:44Z, after that day's last diffctx push, so the project did not exist until the first push of the same evening (ed4cd2f5) created it: gate OK, 92 issues — 83 rust:S3776 cognitive complexity (tracked, see the issue filed that pass), 7 python:S2245 + 1 python:S2612 re-marked false positive (the marks did not migrate from SonarCloud), docker:S6471 on Dockerfile.eval accepted (operator-run research image).
  6. SonarCloud still analyses every GitHub push under the legacy key nikolay-e_TreeMapper (organization nikolay-e; a search for "diffctx" finds nothing) and keeps its own "Sonar way" gate — security, reliability and maintainability ratings on new code, which the self-hosted "Fleet" gate does not have. Both gates are intake. Public API, no token: https://sonarcloud.io/api/qualitygates/project_status?projectKey=nikolay-e_TreeMapper and …/api/issues/search?componentKeys=nikolay-e_TreeMapper&resolved=false&inNewCodePeriod=true. On 2026-09-23 it held a C security rating (an http-capable curl -L) while the self-hosted gate read OK.
  7. The stumble ledger is a channel: [stumble] <task> issues carry the per-batch median and the deduped gripes. A convergent gripe (the same slug from independent runs) is the signal — on 2026-08-30 four of six slugs were one defect, an unsourced headline number, and it was real.
  8. The Forgejo [autoqa] repo nikolay-e/diffctx: open findings issue (ci-bot, from the same autoqa-daily run): gitleaks over the whole reachable history plus grype over the Actions pins. It rewrites its own body each day and closes itself when clean, so triage it before the next run, and read the attached findings.json for the tail the body truncates. Two traps: it reaches every commit any tag points at — the pre-rewrite bitcheck/* anchors kept 61 third-party tokens in retired benchmark diffs "ours" for a day (fixed by re-pinning bitcheck on the rewritten history); and grype reads a sha-pinned action at the version in its # vX comment (autoqa issue 68).

Issue triage invariants

  • The label set and the tracker rule live in #394 (root: invariants, epics, versions). Every open issue has one primary home — epic:*, invariant:*, deferred or blocked:measurement — plus bug / enhancement from the templates; nothing else. Finer classes (regression, known limitation, validation gap) go in the body. A broken instrument is invariant:eval-integrity, never a research or benchmark item: that misfiling is how a tracker with zero bugs once read as "no defects left".
  • Fixed in source, verified on the repro, released are three states. An issue fixed on main sits in the next release's milestone and closes with a comment naming that release.
  • A closure comment does not tick the body. #245's items 2/5/6 stayed [ ] for three weeks after the comment that closed them, and an outside audit read the issue as closed-with-open-work. When a comment closes a checklist item, edit the body in the same pass; gh issue edit --body-file.

Traps this repo has already sprung

  • A -1 at the head of --budgets is an option to argparse. The first full-mode eval-sweep run (2026-09-16) died in every cell in seconds with --budgets: expected one argument; the cell's run.log is in its uploaded artifact (gh run download <run> -n cell-…), and the job log needs gh api --allow-escape-sequences …/jobs/<id>/logs. The workflow passes --budgets="${BUDGETS}" now; anything else that hands a list whose first element can be negative to argparse needs the = form.

  • grep | head -5 under shell: bash (pipefail) fails once the sixth line exists. The CD sdist gate printed five Rust paths for the log and died on the write error; the 1.16.0 release stopped at "Build sdist" with nothing published. grep -m5 prints the same lines without a pipe.

  • cd.yml tags on GitHub only, and nothing carries tags to Forgejo. Until 2026-10-02 a Forgejo push mirror pruned every tag Forgejo lacked, which drafted the release (assets 404 for npm install and Scoop) — 1.16.0 and 1.18.2 (#332). That mirror is gone (gitops dae29cbf2; github-reverse-sync moves branches only), and publish-extras.yml now recreates a missing tag at the "Release version X.Y.Z" commit and un-drafts the release, with a ::warning::. Still push the tag to Forgejo by hand after a release (git fetch github --tags && git push origin vX.Y.Z); re-adding a push mirror brings the pruning back.

  • A dependency PR merged right after a release fails uv sync --locked. Its uv.lock still names the pre-release project version; the bump changed pyproject.toml underneath it. uv lock (one-line diff) and commit; merge dependency PRs before dispatching a release, not after.

  • Cancelling a running release publishes nothing — the bump commits reach github/main only in the finalize job, so a cancelled cd.yml leaves version.py on the old version and the same version can be dispatched again.

  • uvx --from 'diffctx[mcp]' diffctx-mcp builds from source when uv's default interpreter is free-threaded. The wheels are cp310-abi3, which a 3.14t build cannot use, so uvx resolves the sdist, compiles the Rust extension (~75 s, needs cargo) and the MCP client gives up at its 30 s connect timeout — this session's own diffctx server "failed to connect" that way. The failure is the user's uv setup (uv python list: a managed +freethreaded install and no other managed one), not the package; the README shape is still right. Diagnose with the initialize handshake piped into the command under time.

  • Docs subpages load anchor-js from cdnjs through the default Jekyll theme (docs/ has no _config.yml, so Pages renders product/*.md with jekyll-theme-primer, whose layout inlines anchors.add()). With the CDN blocked the console shows ReferenceError: anchors is not defined; the page reads fine. A stumble run gripes about it; overriding the theme layout to drop one script buys nothing — do not file it.

  • The landing page scrolled sideways on a phone for months. At 390 px the nav links and the single-column lab grid (1fr resolves to min-content) pushed the document to ~595 px wide; nothing checked it. Browser QA of the page includes one mobile-width probe: document.documentElement.scrollWidth <= innerWidth at 390 px, plus a list of elements whose right edge passes the viewport outside an overflow: auto box.

  • Two ignore policies, not one. The withhold policy (secret names, .diffctx/ignore, gitignore, .git/) is the security floor everywhere. The noise policy (DEFAULT_IGNORE_PATTERNS: node_modules/, target/, lock files) belongs to tree-mode readers — get_tree_map and the legacy get_file_context glob. Collapsing them in either direction has now broken something twice: engine-only served uv.lock and every ignored directory, noise-only served .netrc. fetch_fragments is the deliberate exception — engine only, because refusing a fragment selection ranked is the other half of #228.

  • find_ignored_paths_with_source is the attribution lookup and drops ancestor-inherited exclusions on purpose (#153). Anything that walks the working tree must use find_ignored_paths_any instead, or it answers "not ignored" for the whole contents of every ignored directory.

  • A cargo fmt run invalidates a source-text mutation. Verifying a Rust gate on a known-bad input by string-replacing the fix will silently no-op if the formatter has since rewrapped the line — the "known-bad" run then tests the good code and passes. Re-read the file after cargo fmt and assert the mutation anchor before trusting a green.

  • A rebuild is part of restoring a mutation. uv sync --reinstall-package diffctx installs whatever the source said at that moment; restoring the file afterwards leaves the pre-fix extension in the venv, and the next pytest measures the wrong build.

  • rrf and pit read DIFFCTX_OP_GRAPH_DEPTH, internal-bm25 does not. The fusion modes take ego_depth_extended (env-overridable); bm25 takes the hardcoded ego_depth_default. Excluding depth cells for all three at once deletes 30 real configurations — it happened on 2026-08-30 and was reverted the same pass.

  • A Renovate/Dependabot merge on the mirror cancels the in-flight CI of the commit before it. ci.yml has concurrency: cancel-in-progress keyed on github.ref, so any push to main — a bot's included — cancels whatever main run is still going. A cancelled verdict on your own commit therefore means "superseded", not "failed": gh run rerun <id> it, or read the next commit's run, which covers the same tree plus the bump.

  • The first version of a Q-class edge change is measured, never trusted. link_by_name's rewrite passed lib tests and looked obviously right; the full corpus moved cargo_001 (suffix matched a sibling crate) and two Swift cases (import Database resolves to a directory, which the old fallback reached and the new code did not). Both were invisible below the full 2902-case run.

  • Fragment paths are absolute; the FragmentIndex is keyed repo-relative. A builder that derives a path reference from a fragment's own path (Path::new(f.path()).parent()) must pass it through base::index_key_of(path, repo_root) first, or the reference matches nothing — and a walk that climbs to / then degrades to the bare name and matches everything. Two channels shipped dead this way on 2026-09-02 while the corpus stayed green; the builder tests that caught it use absolute paths plus a repo root, which is the only shape that exercises the seam.

  • Sonar's unreachable-code rule is a real bug detector here, not lint. rows.append under a continue made stratified-analysis return nothing; no test touched it. A red Sonar gate on your own commits is intake, and the fix ships with a test that fails on the previous code.

  • A new eval test needs pytest.importorskip("numpy") (or whatever the eval group provides): CI's test job installs no eval group, so an unguarded import fails collection on every Python version.

  • "${arr[@]}" on an empty array is an unbound variable under bash 3.2 with set -u — macOS /bin/bash, which #!/usr/bin/env bash resolves to whenever Claude Code's PATH has no Homebrew bash (Dock, VS Code). The plugin's impact hook died that way on every macOS default install of 1.18.0, right of a pipe, under || true: exit 0, no output. Build one never-empty array (args=(hook "$event"); args+=(--gate)), and probe hook scripts with /bin/bash explicitly, not the shell on PATH.

  • The Claude plugin runs nothing it downloaded. The directory refused 1.18.1 for fetching the release binary at session start; it runs only code in the reviewed repository or a package pinned to an exact version. The hooks run uvx "diffctx[mcp]==X" hook <event> — the MCP server's own launch — with the version written plainly in each script (cd.yml bumps it, test_publishing_manifests pins it). The directory reads .mcp.json's uvx <package>==<version> as a locked launch only with no option but -q, --from and --python, against a uv.lock beside it: -c constraints.txt was "Launcher lock invalid". The "Runs a pinned npx or uvx package" hold stays on every version regardless, and every commit on the tracked branch is one more held version (~80 from 2026-09-26 to 1.18.4), which is why the directory follows plugin-release, moved by cd.yml at release, not main. Probe a hook change with tests/test_plugin_hook.py (it drives the script under /bin/bash), and remember a manual --mode impact run silences the hook on the same content for 15 minutes: use DIFFCTX_NO_MARKER=1 or a scratch DIFFCTX_CACHE_DIR when comparing the two.

  • Every child process gets a null stdin. Under an MCP client the server's stdin is the JSON-RPC pipe the client holds open; a git that inherited it never exited on Windows and every tool call reaching git hung (fixed 2026-09-23, git_command + every subprocess.run). A new spawn site that forgets .stdin(Stdio::null()) / stdin=subprocess.DEVNULL re-opens it, and only the Windows legs of tests/test_mcp_stdio.py will say so.

  • On Windows text=True decodes as cp1252, and write_text writes CRLF. Four "Windows-only" failures on 2026-09-23 were the tests': UTF-8 output read as cp1252 (— became �), and a 99 KB fixture that crossed the 100 KB cap once every \n became \r\n. Tests read subprocess output with encoding="utf-8" and write fixtures with newline="".

Known false positives

  • import-linter local failures are REAL — the old excuse ("fails locally on the namespace package while green in CI") is dead; lint-imports reports 6 kept, 0 broken locally. One failure caught from diffctx import _diffctx in mcp/server.py, which executes diffctx/__init__ and so pulls _app -> cli, breaking "MCP server must not import CLI". Import the submodule directly (from diffctx._diffctx import X) so the edge points at the submodule rather than the package root.
  • Sonar python:S2245 on paired_bootstrap (seeded random.Random for CI resampling) — false positive: determinism is the requirement, not a security context.
  • Sonar pythonsecurity:S8705/S8707 on eval/workflows/ harnesses ("LLM sandbox escape") — accepted risk: operator-run local research harnesses; argv exec with regex-validated sha (realworld_rerun) and operator-chosen output dir (run_final). Don't re-litigate unless the eval workflows become agent-facing.
  • Sonar python:S2612 on writer.py os.fchmod(fd, 0o666 & ~umask) — false positive: the mode is what open(..., "w") would have created; mkstemp's 0600 is the anomaly being corrected. Python NOSONAR takes no (rule) argument — # NOSONAR(python:S2612) is itself flagged (S7632) and suppresses nothing.
  • Sonar githubactions:S8543 on the publish-extras npm smoke: $VERSION is an exact just-published version, package has zero deps — false positive (NOSONAR is NOT supported by the githubactions analyzer; don't re-add it). Gotcha: editing the flagged line (or its neighbours) shifts the issue hash and Sonar re-raises the finding under a NEW issue key with the FP mark lost — re-fetch after every analysis touching that file and re-mark via api/issues/do_transition (falsepositive).
  • None of the resolutions above exist on the self-hosted server: they were made on the retired hosted instance and did not migrate, so every one re-raises until it is resolved again.
  • The landing page's doi.org/10.5281/zenodo.18824579 answers 403 to a scripted HEAD; a GET with a browser User-Agent follows two redirects to the Zenodo record (200). Not a broken link.

Recurring bug patterns (diagnose once, recognise thereafter)

  • Over-emission for a tiny diff is ONE mechanism, not per-language bugs. excerpt::generate_core_excerpts cuts a hunk window (+3 context lines, capped at 70% of the parent) but is consulted only as a budget fallback via build_signature_lookup, when the core exceeds budget x core_budget_fraction. So the same file ships whole at --budget 8000 and tightly excerpted at a small budget. Three issues are this one mechanism reached three ways: no grammar → whole-file chunk (#105); grammar parses but the body is flat → one chunk (#107); fine fragments exist but the hunk spans more lines than any of them, so find_core_for_hunk promotes to the enclosing definition (Forgejo issue 2). Don't re-diagnose per language. Status: fixed for the changed role — the excerpt is consulted on how little changed rather than on leftover budget, excerpts are generated for kinds that have a signature variant too (a signature drops the changed lines), render agrees with locate that an Excerpt is changed, and long uncovered runs are split into bounded chunks so a flat file has sub-file granularity at all. #105/#107/#114 closed on their own repros. The context role was fixed by a different lever (#123, closed 2026-08-04): not windowing (both windowing attempts measured 43 corpus failures each) but the file-level relation encoding (9b41040d) — import-class edges land on the file's representative fragment instead of fanning to every fragment at full weight, so an indivisible 400-line parent stops being selected as context at all.
  • coherence_post_pass is inert by accident and load-bearing for precision. It resolves a dangling semantic neighbour by lowercased symbol_name instead of by the id the graph edge already gives it. That is a real bug, but the name lookup mostly lands on already-selected fragments, so the pass adds nothing. Fixing the lookup alone activates a pass with no relevance bar that draws from filtered_fragments — i.e. it re-admits candidates the greedy declined — and cost 9 of 2725 oracle cases (recall=100%, forbidden_rate=100%). Land the id fix only together with a relevance bar or a cap.
  • Span-vs-content mismatch. line_count() comes from the id's span while slicing indexes content.lines(); nothing enforces agreement. Gate on the actual line vector, and never let end < start reach a FragmentId — line_count() underflows on it in every later stage.
  • Lexical path containment is not containment. Path::starts_with compares components and canonicalize fails on a non-existent path (the old side of a deletion). A lexical fallback used as an alternative to the canonical check makes the canonical check dead. Reject .. and absolute up front, then use the lexical form ONLY when canonicalization is impossible — otherwise an in-repo symlink pointing outside passes. Both are pinned by tests in git.rs; the tests must canonicalize their temp root or the hole hides on macOS (/var -> /private/var) and only fails on Linux CI.

Bit-equivalence gate for refactors (scripts/bitcheck.sh)

record on the pre-change build, rebuild, check. 24 cells (3 fixed ranges x 4 scoring modes x pack/locate) in ~20 s; clean drops the snapshots and the fixture worktree.

  • The bitcheck/<sha> tags that anchor the fixture must point INTO the rewritten history. Anchors from before an epoch squash keep the entire old tree reachable on both forges, and the daily secret scan of history then reports every retired benchmark blob — re-pin the ranges on main commits after every rewrite, then delete the old tags on both remotes.
  • The input is a worktree pinned to a fixed SHA, not the live checkout. diffctx analysing its own repo means its sources appear in its own output as context fragments, so editing the code under test also edits the input. Running against the live tree reported 8 moved cells for a change that could not affect them — a stashed file had rewritten a context fragment. Never "simplify" the fixture away.
  • The fixture cannot exercise every path: this repo's .diffctx/ignore hides tests, so no FooTest.java/widget-spec.js shape exists in it and a test-classifier change reads as bit-identical. Corpus + targeted unit tests carry that; bitcheck does not.
  • A claimed pure refactor lands with a bitcheck result in the same commit; an output-changing fix is judged by a known-answer test and by reading each corpus case it flips (AGENTS.md), never parked for moving output.

Measuring against a running build

Two ways to invalidate an overnight measurement, both observed in practice:

  • maturin develop swaps the extension under a running experiment. The eval harnesses invoke .venv/bin/diffctx, which loads src/diffctx/_diffctx.abi3.so. Rebuilding mid-run mixes two builds into one result file. cargo build / cargo test are safe — they write target/. Editing Python under src/diffctx/ is NOT safe for the CLI path (editable install), though mcp/ is, since the CLI never imports it.
  • A background cargo test compiles the live tree: editing sources after launching the corpus run means the binary's content is undefined — kill and relaunch after the last edit (one mid-compile edit cost a full 150s rerun).
  • Name the build, not the branch. Commit time is not build time. Compare the .so mtime against git log --date and state which commit's content the run executed; "measured on main" is meaningless when six commits landed while it ran.

Reading a benchmark whose cases time out

datasets/real-world-diff/v1/ is 109 real commits over react-native / gitpod / sentry, and its shipped snapshot is 66% hangs — so its quality numbers describe the survivors, not the benchmark. Rules that follow:

  • Report the hang rate first and the denominator always. "precision 0.81" over 43 produced cases out of 81 is not "precision 0.81".
  • Record elapsed per case. The snapshot used a 30s cap and current defaults use 300s; only the recorded seconds let one run be read at both.
  • Parallelise per repo (one detached worktree each), not per case — checkouts of a 2 GB repo dominate otherwise, and the three repos never contend.
  • gitpod and sentry are the timeout-bound pair; react-native completes. A liveness claim that does not say which repo is not a claim.

Corpus baseline discipline

known_below_threshold.txt is bidirectional: a listed case that starts passing fails with "remove it from that file". That is a legitimate baseline edit only when the improvement is intended and kept — if the cause gets reverted, restore the entry. Never edit the baseline to absorb an unexplained new failure; bisect the cause first (a scratch git worktree at each candidate commit + a single-case run is the fast path).

Known non-bugs (audited correct — do not re-file)

  • IntervalIndex::overlaps boundary asymmetry — deliberate, pinned by a verdict-matrix test; symmetric fix is net-zero on the corpus (Q-class, next calibration).
  • node_end_line +1 in tree-sitter spans is correct (end-exclusive to end-inclusive conversion).
  • Haskell type_synomym matches the upstream grammar's misspelling — "fixing" the spelling breaks extraction.
  • BM25 .ln_1p() is the Lucene IDF formulation, deliberate.
  • Float-sum nondeterminism pattern: any accumulation over FxHashMap iteration order must sort keys first — fixed sites are pinned by determinism tests; new code must follow suit.