Add BullshitBench validity defense + model checking diagnostics - #2
Merged
Conversation
Two complementary additions for evaluating Distribird's behavior: BullshitBench — parameter validity defense - New `validity_check` graph node classifies every parameter as VALID / SUSPICIOUS / LIKELY_INVALID / UNKNOWN at the end of the pipeline - Passive heuristics combine signals: papers found across refined queries, values extracted, enrichment LLM self-flags (is_recognized_parameter, recognition_confidence, empirically_measured), prior confidence - Optional second-opinion LLM probe runs only when passive verdict is SUSPICIOUS and budget permits - `PARAMETER_ENRICHMENT` prompt extended with three new fields, free signal from an existing LLM call; new `PARAMETER_VALIDITY_PROBE` prompt - Settings: `enable_validity_check`, `enable_validity_probe` (default True) - New fields: `parameter_validity`, `validity_reason`, `validity_signals`, `is_empirical` on PipelineResult - 17 tests (10 integration + 7 unit) plus a real-LLM smoke runner that verified 4/4 verdicts on real Semantic Scholar + LLM calls Model checking diagnostics - New `distributions/model_check.py` computes goodness-of-fit (KS test, AIC, log-likelihood, credible-interval coverage, mean absolute CDF deviation) for fitted priors against extracted literature values - MAP, mean, median, variance, 95% CI exposed via `ModelCheckResult` - Beta/Gamma boundary cases (alpha<=1) return median instead of degenerate mode at 0/boundary - New `export/table_export.py` produces LaTeX/Markdown tables for paper- ready presentation; `export/json_export.py` updated to serialize `ModelCheckResult` - Offline utility `check_model_from_result()` for retrofitting existing BatchResult JSON with diagnostics Tests: 217 passing (197 existing + 10 BullshitBench integration + 10 validity unit + ~40 model_check unit). No regressions. Lint clean.
There was a problem hiding this comment.
Pull request overview
This PR adds two evaluation/diagnostics features to Distribird’s pipeline: (1) a parameter validity classifier (“validity defense”) that labels requests as VALID/SUSPICIOUS/LIKELY_INVALID/UNKNOWN, and (2) model-checking diagnostics to assess fitted priors against extracted literature values, including exports for paper-ready tables.
Changes:
- Introduces a new
validity_checkpipeline node with passive heuristics plus an optional second-opinion LLM probe, and wires its outputs intoPipelineResult. - Adds
distributions/model_check.pyand corresponding models/tests to compute goodness-of-fit diagnostics (KS test, AIC, LL, coverage, CDF deviation, etc.). - Adds new export utilities for model-check tables and extends JSON export to optionally include
model_check.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_validity.py | Unit tests for passive validity heuristics + probe verdict application. |
| tests/test_model_check.py | Comprehensive unit tests for model-check computations and offline utilities. |
| tests/test_bullshitbench.py | Integration-style tests for end-to-end validity classification behavior. |
| src/distribird/models.py | Adds ParameterValidity, ModelCheckResult, and validity/model-check fields on PipelineResult; extends EnrichedContext. |
| src/distribird/export/table_export.py | New Markdown/LaTeX table exporters for model-check outputs. |
| src/distribird/export/json_export.py | Serializes model_check when present. |
| src/distribird/distributions/model_check.py | Implements model-check computations and offline batch/result utilities. |
| src/distribird/config.py | Adds toggles for validity check/probe. |
| src/distribird/agent/validity.py | Implements passive validity rules, optional LLM probe, and probe verdict application. |
| src/distribird/agent/state.py | Extends pipeline state to carry validity outputs; adds a budget helper. |
| src/distribird/agent/prompts.py | Extends enrichment prompt to request validity self-flags; adds validity probe prompt. |
| src/distribird/agent/pipeline.py | Ensures pipeline error fallback sets validity fields to safe defaults. |
| src/distribird/agent/nodes.py | Adds validity_check_node and integrates budget/warnings/tracing. |
| src/distribird/agent/graph.py | Inserts validity_check after synthesize and returns validity fields in PipelineResult. |
| src/distribird/agent/enrich.py | Plumbs enrichment self-flags into EnrichedContext and logs them. |
| examples/bullshitbench_run.py | Adds a real-LLM runner script for the validity benchmark. |
| bullshitbench_real_llm_results.md | Adds a sample real-LLM run report. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
added 6 commits
April 29, 2026 18:13
The shipped BullshitBench caught the standard empirical-only case via Rule 4 (LLM explicitly returns empirically_measured=False, papers exist, 0 values extracted → SUSPICIOUS). An audit found three slip-throughs: 1. LLM uncertain (`empirically_measured=None`): Rule 4 didn't fire; if a prior was synthesized at MEDIUM confidence the parameter could be marked VALID, masking model-internal nature. 2. LLM mis-flagged as empirical (`empirically_measured=True` for a calibration weight): Rule 5 fired and the probe was never invoked. 3. Recognition uncertain (`is_recognized_parameter=None`): Rule 5's loose `is not False` gate accepted None and returned VALID for a fitted prior. Changes: - `validity.py`: combined Rules 4 and 4b into one branch on `is_empirical is not True` with two reason templates; tightened Rule 5 to require `is_recognized is True` (strict), `is_empirical is not False`, and `values_extracted >= MIN_VALUES_FOR_VALID` (2). Reason strings extracted to module-level constants so tests can pin to exact wording. - `prompts.py`: extended `PARAMETER_ENRICHMENT` with a red-flag list (version suffixes, software prefixes, calibration/weight/latent terms) and a second worked example demonstrating empirically_measured=false for a Biome-BGCMuSo Q10 calibration weight. - `tests/test_validity.py`: 3 new unit tests covering Rule 4b (uncertain empirical), Rule 5 blocked when is_recognized=None, Rule 5 blocked when is_empirical=False. Existing tests migrated to use REASON_* constants instead of substring matching. - `tests/test_bullshitbench.py`: 2 new integration tests (uncertain empirical calibration weight, misclassified empirical with probe override). Helper `_mk_papers_no_values()` extracted to remove repeated mock setup. - `examples/bullshitbench_run.py`: 2 new real-LLM cases (`kalman_filter_state_covariance_q11` latent state, and `dssat_cropgro_root_growth_partition_factor_v45` version-specific calibration weight). Verification: 222 tests pass (217 existing + 13 unit + 12 integration), ruff clean. Real-LLM run with 6 cases scored 6/6 verdicts matching expected: two pure-nonsense → likely_invalid, three theoretical-only → suspicious, one real (specific_leaf_area) → valid with HIGH-confidence beta prior.
- json_export.py: `result_to_dict()` now exposes parameter_validity (as
enum value), validity_reason, validity_signals, and is_empirical so
API/CLI consumers can observe the validity verdict.
- table_export.py: introduce `_latex_escape()` covering the full set of
LaTeX special characters (\\ { } & % $ # _ ~ ^), used in
`batch_to_latex_table()`. Previously only `_` was escaped, which would
break compilation and allow LaTeX injection for parameter names
containing other specials.
Skipped Copilot suggestions:
- Moving the real-LLM results markdown to docs/ (file is committed
intentionally as evidence of the system working with real LLM).
- Casing nit on a previous version of the report — the v2 report
replaced that content entirely in the prior commit.
The validity_check node correctly classifies out-of-scope requests, but running it only at the END of the pipeline meant we paid the full cost of search, fulltext fetch, extraction, and synthesis for inputs that the enrichment LLM had already flagged as fabricated. On real-LLM benchmark runs the search and extraction stages dominate the wall-clock cost (often 80--95% of the total runtime), so even a clearly nonsensical request like `mumblesnort_factor` consumed ~8 minutes before being classified as LIKELY_INVALID. Add a `route_after_enrich` conditional edge that examines the enrichment LLM's `is_recognized_parameter` and `recognition_confidence` fields and, if the LLM clearly does not recognise the parameter (False with none/low confidence), routes directly to the terminal validity_check node. The classifier returns LIKELY_INVALID via the existing passive heuristics (no terminology, no papers, unrecognized name) without the probe firing. The terminal classifier still runs for all requests; recognised or ambiguous parameters take the full pipeline path so the probe can use real evidence when needed. A new integration test (`test_early_skip_avoids_search_and_extract`) asserts that the search and extract mocks are never called for the clear-nonsense scenario. The previous `test_plausible_sounding_fake_uses_probe` is updated to set is_recognized_parameter=True (LLM fooled by plausible name) so the early route does not fire and the probe path remains exercised. Tests: 223 pass (was 222); ruff clean.
Replaces the BullshitBench TODO with two new prose subsections and a benchmark table. System Design — Section "Validity classification: detecting out-of-scope requests" describes: - Two failure modes outside Distribird's contract (fabricated names, empirical-only model parameters) - The enrichment-prompt extension that elicits is_recognized_parameter, recognition_confidence, and empirically_measured as free signals - The two-stage gate: an early-skip route from Enrich directly to ValidityCheck when the LLM does not recognise the parameter, plus a terminal classifier that issues the final verdict using all collected signals (papers, values, prior confidence) - The optional LLM probe for ambiguous SUSPICIOUS verdicts and how the verdict is exposed on PipelineResult Pipeline figure — Adds a ValidityCheck node and a red dashed early-skip edge from Enrich showing the short-circuit path for unrecognised parameters. Caption explains that the bypass saves ~90% of wall-clock cost on out-of-scope requests. Experimental Evaluation — New subsection "Out-of-scope detection: BullshitBench" reports the real-LLM benchmark (6 parameters across nonsense, theoretical-only, and real-control categories) with verdicts matching expectations 6/6. Table tabulates verdict, paper/value counts, and wall-clock time per request. Conditions of Applicability — The "What Distribird is not designed for" callout now forward-references the validity classifier and the benchmark, so the limitation is paired with the active mitigation. Compiles cleanly (two minor overfull hboxes, both <12pt; pre-existing 3.6pt overfull at lines 509-526 is unchanged).
These artefacts are kept locally only: - paper/ — manuscript draft - examples/bullshitbench_run.py — local benchmark runner - bullshitbench_real_llm_results.md — local benchmark output Files remain on disk; they are simply no longer part of the repository.
Reformat four files via ruff (graph.py, nodes.py, validity.py, table_export.py) and adjust the recognition_confidence validator to narrow `object` to `str` via isinstance() instead of relying on a `# type: ignore` that newer mypy versions flagged as unused.
PatrikScully
added a commit
that referenced
this pull request
Apr 29, 2026
…ount - DAG diagram now shows Synthesize -> ValidityCheck -> END (matches the BullshitBench feature merged in PR #2). - New "Validity defense" bullet describes the four verdicts (VALID, SUSPICIOUS, LIKELY_INVALID, UNKNOWN) and the early-skip short-circuit. - Tests badge and prose counts updated 148 -> 223.
PatrikScully
added a commit
that referenced
this pull request
Apr 29, 2026
Minor bump for the BullshitBench / parameter-validity-defense feature merged in PR #2: new four-class verdict system (VALID, SUSPICIOUS, LIKELY_INVALID, UNKNOWN) plus early-skip routing. Also syncs src/distribird/__init__.py (was stale at 0.1.0) with pyproject.toml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two complementary additions for evaluating Distribird's behavior:
BullshitBench — parameter validity defense
validity_checkgraph node classifies every parameter as VALID / SUSPICIOUS / LIKELY_INVALID / UNKNOWN at the end of the pipelinePARAMETER_ENRICHMENTprompt extended with three new fields, free signal from an existing LLM call; newPARAMETER_VALIDITY_PROBEpromptenable_validity_check,enable_validity_probe(default True)parameter_validity,validity_reason,validity_signals,is_empiricalon PipelineResultModel checking diagnostics
distributions/model_check.pycomputes goodness-of-fit (KS test, AIC, log-likelihood, credible-interval coverage, mean absolute CDF deviation) for fitted priors against extracted literature valuesModelCheckResultexport/table_export.pyproduces LaTeX/Markdown tables for paper- ready presentation;export/json_export.pyupdated to serializeModelCheckResultcheck_model_from_result()for retrofitting existing BatchResult JSON with diagnosticsTests: 217 passing (197 existing + 10 BullshitBench integration + 10 validity unit + ~40 model_check unit). No regressions. Lint clean.