Add BullshitBench validity defense + model checking diagnostics

bullshitbench_real_llm_results.md

@@ -0,0 +1,154 @@
+# BullshitBench — Real-LLM Run
+
+Real end-to-end pipeline runs (LLM enrichment + Semantic Scholar + extraction + validity check)
+
+## Summary — 4/4 verdicts match expected
+
+| Parameter | Category | Expected | Verdict | Match | Papers | Values | Time |
+|---|---|---|---|---|---|---|---|
+| `mumblesnort_factor` | nonsense | likely_invalid | likely_invalid | OK | 0 | 0 | 50.8s |
+| `fake_quantum_correction_xyz` | nonsense | likely_invalid | likely_invalid | OK | 0 | 0 | 46.9s |
+| `biome_bgcmuso_carbon_pool_calibration_weight_v3` | theoretical | suspicious | suspicious | OK | 8 | 0 | 560.2s |
+| `specific_leaf_area` | real | valid | valid | OK | 80 | 4 | 1954.2s |
+
+## Per-case details
+
+### `mumblesnort_factor` — nonsense
+- **Description:** A fabricated coefficient that does not exist in any literature
+- **Domain:** general scientific testing
+- **Expected:** `likely_invalid`
+- **Verdict:** `likely_invalid` — MATCHES expected
+- **Reason:** No literature found across 5 refined queries
+- **Empirical:** False
+- **Pipeline:** 0 papers, 0 values, prior `truncated_normal` (confidence: none, informative: False)
+- **LLM recognized:** False (confidence: none)
+- **LLM empirically measured:** False
+- **Terminology:** fabricated parameter, dummy variable, placeholder coefficient
+- **Signals:**
+  ```json
+  {
+    "papers_found": 0,
+    "values_extracted": 0,
+    "n_queries_tried": 5,
+    "is_recognized_parameter": false,
+    "recognition_confidence": "none",
+    "empirically_measured": false,
+    "n_terminology": 3,
+    "prior_is_informative": false,
+    "prior_confidence": "none"
+  }
+  ```
+- **Warnings:**
+  - The requested parameter 'mumblesnort_factor' is fabricated and does not exist in scientific literature.
+  - All provided papers were excluded because they are completely unrelated to the target parameter.
+  - No informative evidence found; using uninformative prior.
+  - Parameter validity: LIKELY INVALID — No literature found across 5 refined queries
+- **Elapsed:** 50.8s
+
+### `fake_quantum_correction_xyz` — nonsense
+- **Description:** A made-up quantum correction term with no scientific basis
+- **Domain:** theoretical physics
+- **Expected:** `likely_invalid`
+- **Verdict:** `likely_invalid` — MATCHES expected
+- **Reason:** No literature found across 5 refined queries
+- **Empirical:** False
+- **Pipeline:** 0 papers, 0 values, prior `truncated_normal` (confidence: none, informative: False)
+- **LLM recognized:** False (confidence: none)
+- **LLM empirically measured:** False
+- **Terminology:** radiative correction, loop correction, quantum correction, higher-order correction, renormalization constant
+- **Signals:**
+  ```json
+  {
+    "papers_found": 0,
+    "values_extracted": 0,
+    "n_queries_tried": 5,
+    "is_recognized_parameter": false,
+    "recognition_confidence": "none",
+    "empirically_measured": false,
+    "n_terminology": 5,
+    "prior_is_informative": false,
+    "prior_confidence": "none"
+  }
+  ```
+- **Warnings:**
+  - The requested parameter is fictitious and has no physical equivalent, meaning no valid scientific literature can be used to build an empirical Bayesian prior for it.
+  - No informative evidence found; using uninformative prior.
+  - Parameter validity: LIKELY INVALID — No literature found across 5 refined queries
+- **Elapsed:** 46.9s
+
+### `biome_bgcmuso_carbon_pool_calibration_weight_v3` — theoretical
+- **Description:** Internal calibration weight from Biome-BGCMuSo model version 3.x; purely a model-internal tuning parameter
+- **Domain:** Biome-BGCMuSo crop modeling
+- **Expected:** `suspicious`
+- **Verdict:** `suspicious` — MATCHES expected
+- **Reason:** The parameter is a model-internal calibration weight specific to a particular model version and is not an empirically measured scientific quantity.
+- **Empirical:** False
+- **Pipeline:** 8 papers, 0 values, prior `truncated_normal` (confidence: none, informative: False)
+- **LLM recognized:** False (confidence: none)
+- **LLM empirically measured:** False
+- **Terminology:** calibration parameter, tuning weight, scaling factor, empirical adjustment factor, model coefficient
+- **Signals:**
+  ```json
+  {
+    "papers_found": 8,
+    "values_extracted": 0,
+    "n_queries_tried": 15,
+    "is_recognized_parameter": false,
+    "recognition_confidence": "none",
+    "empirically_measured": false,
+    "n_terminology": 5,
+    "prior_is_informative": false,
+    "prior_confidence": "none"
+  }
+  ```
+- **LLM probe:** verdict=`suspicious`, reason='The parameter is a model-internal calibration weight specific to a particular model version and is not an empirically measured scientific quantity.'
+- **Warnings:**
+  - None of the abstracts explicitly mention the exact parameter 'biome_bgcmuso_carbon_pool_calibration_weight_v3' or its numerical value.
+  - The parameter is likely an internal tuning weight that may only be found in the supplementary materials, model code, or detailed methodology sections of papers [1], [2], [3], and [4].
+  - Search refinement round 1: generated 5 new queries.
+  - The specific parameter 'biome_bgcmuso_carbon_pool_calibration_weight_v3' may be an obsolete or highly specific internal tuning weight from version 3.x, whereas most recent literature covers versions 4.0 to 6.2 ([1], [5]).
+  - None of the abstracts explicitly report numerical values for this specific v3.x calibration weight, so full-text review of the model description papers (e.g., [5]) will be necessary.
+  - Search refinement round 2: generated 5 new queries.
+  - None of the abstracts explicitly mention the specific 'biome_bgcmuso_carbon_pool_calibration_weight_v3' parameter or its numerical value.
+  - The parameter is likely an internal tuning weight that may only be found in the supplementary materials, model code, or detailed methodology sections of papers like [2], [3], or [6].
+  - No informative evidence found; using uninformative prior.
+  - Parameter validity: suspicious — The parameter is a model-internal calibration weight specific to a particular model version and is not an empirically measured scientific quantity.
+- **Elapsed:** 560.2s
+
+### `specific_leaf_area` — real
+- **Description:** Leaf area per unit dry mass of leaves
+- **Domain:** maize crop modeling
+- **Expected:** `valid`
+- **Verdict:** `valid` — MATCHES expected
+- **Reason:** Recognized parameter with literature-backed prior
+- **Empirical:** True
+- **Pipeline:** 80 papers, 4 values, prior `truncated_normal` (confidence: medium, informative: True)
+- **LLM recognized:** True (confidence: high)
+- **LLM empirically measured:** True
+- **Terminology:** Specific leaf area (SLA), Leaf mass per area (LMA), Specific leaf weight (SLW), Leaf area-to-mass ratio
+- **Signals:**
+  ```json
+  {
+    "papers_found": 80,
+    "values_extracted": 3,
+    "n_queries_tried": 15,
+    "is_recognized_parameter": true,
+    "recognition_confidence": "high",
+    "empirically_measured": true,
+    "n_terminology": 4,
+    "prior_is_informative": true,
+    "prior_confidence": "medium"
+  }
+  ```
+- **Warnings:**
+  - Crop modeling papers like [3], [4], and [5] might use default specific leaf area values from model documentation rather than measuring them directly in the field.
+  - Papers [2] and [6] might report specific leaf area as an intermediate variable rather than the main focus, requiring careful extraction.
+  - Search refinement round 1: generated 5 new queries.
+  - Papers [3] and [4] lack DOIs and abstracts, which may make full-text retrieval difficult, though their titles are highly relevant.
+  - None of the provided abstracts contain explicit numerical values for SLA, meaning full-text review will be required to extract the actual parameter values.
+  - Used web-assisted extraction to look up paper content online.
+  - Search refinement round 2: generated 5 new queries.
+  - Most selected papers do not explicitly state numerical SLA values in their abstracts, requiring full-text review.
+  - Some papers (like [3] and [4]) may report Leaf Mass per Area (LMA) or Leaf Dry Matter Content instead of SLA; LMA is the inverse of SLA and will require conversion.
+  - Paper [5] is a global vegetation model, so its maize SLA parameter might be a generic crop functional type default rather than a specifically calibrated value for a local context.
+- **Elapsed:** 1954.2s

examples/bullshitbench_run.py

@@ -0,0 +1,240 @@
+"""Real-LLM BullshitBench runner.
+
+Runs a curated mix of fake, theoretical, and real parameters through the full
+Distribird pipeline (with real LLM + Semantic Scholar / OpenAlex). Saves a
+markdown report comparing the validity verdicts.
+
+Usage:
+    python examples/bullshitbench_run.py
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import time
+from dataclasses import dataclass
+from pathlib import Path
+
+from distribird.agent.pipeline import run_parameter
+from distribird.config import get_settings
+from distribird.models import (
+    ConstraintSpec,
+    ParameterInput,
+    ParameterValidity,
+    PipelineResult,
+)
+
+logging.basicConfig(level=logging.WARNING, format="%(message)s")
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class TestCase:
+    name: str
+    description: str
+    domain_context: str
+    expected: ParameterValidity
+    category: str  # "nonsense", "theoretical", "real"
+    constraints: ConstraintSpec | None = None
+
+
+CASES: list[TestCase] = [
+    # ── Pure nonsense ──
+    TestCase(
+        name="mumblesnort_factor",
+        description="A fabricated coefficient that does not exist in any literature",
+        domain_context="general scientific testing",
+        expected=ParameterValidity.LIKELY_INVALID,
+        category="nonsense",
+        constraints=ConstraintSpec(lower_bound=0, upper_bound=10),
+    ),
+    TestCase(
+        name="fake_quantum_correction_xyz",
+        description="A made-up quantum correction term with no scientific basis",
+        domain_context="theoretical physics",
+        expected=ParameterValidity.LIKELY_INVALID,
+        category="nonsense",
+        constraints=ConstraintSpec(lower_bound=0, upper_bound=1),
+    ),
+    # ── Theoretical / non-empirical ──
+    TestCase(
+        name="biome_bgcmuso_carbon_pool_calibration_weight_v3",
+        description=(
+            "Internal calibration weight from Biome-BGCMuSo model version 3.x; "
+            "purely a model-internal tuning parameter"
+        ),
+        domain_context="Biome-BGCMuSo crop modeling",
+        expected=ParameterValidity.SUSPICIOUS,
+        category="theoretical",
+        constraints=ConstraintSpec(lower_bound=0, upper_bound=10),
+    ),
+    # ── Real parameter (control) ──
+    TestCase(
+        name="specific_leaf_area",
+        description="Leaf area per unit dry mass of leaves",
+        domain_context="maize crop modeling",
+        expected=ParameterValidity.VALID,
+        category="real",
+        constraints=ConstraintSpec(lower_bound=5, upper_bound=50),
+    ),
+]
+
+
+async def run_one(case: TestCase, settings) -> tuple[TestCase, PipelineResult, float]:
+    """Run a single test case through the real pipeline."""
+    param = ParameterInput(
+        name=case.name,
+        description=case.description,
+        unit="",
+        domain_context=case.domain_context,
+        constraints=case.constraints or ConstraintSpec(),
+    )
+    print(f"\n{'=' * 70}", flush=True)
+    print(f"Running: {case.name}  (expect: {case.expected.value})", flush=True)
+    print(f"{'=' * 70}", flush=True)
+    t0 = time.monotonic()
+    try:
+        result = await run_parameter(param, settings)
+    except Exception as e:
+        logger.exception("Pipeline crashed for %s", case.name)
+        # Return a synthetic failure result
+        from distribird.distributions.uninformative import wide_normal_prior
+
+        result = PipelineResult(
+            parameter=param,
+            prior=wide_normal_prior(
+                param.name,
+                param.constraints.lower_bound,
+                param.constraints.upper_bound,
+            ),
+            warnings=[f"Crash: {e}"],
+        )
+    elapsed = time.monotonic() - t0
+    print(
+        f"  → verdict: {result.parameter_validity.value}  "
+        f"(papers={result.papers_found}, values={result.values_extracted}, "
+        f"elapsed={elapsed:.1f}s)",
+        flush=True,
+    )
+    return case, result, elapsed
+
+
+def render_markdown(
+    runs: list[tuple[TestCase, PipelineResult, float]],
+    output_path: Path,
+) -> None:
+    lines: list[str] = []
+    lines.append("# BullshitBench — Real-LLM Run\n")
+    lines.append(
+        "Real end-to-end pipeline runs (LLM enrichment + Semantic Scholar + extraction + validity check)\n"
+    )
+
+    # ── Summary table ──
+    correct = sum(1 for c, r, _ in runs if r.parameter_validity == c.expected)
+    lines.append(f"## Summary — {correct}/{len(runs)} verdicts match expected\n")
+    lines.append(
+        "| Parameter | Category | Expected | Verdict | Match | Papers | Values | Time |"
+    )
+    lines.append("|---|---|---|---|---|---|---|---|")
+    for case, result, elapsed in runs:
+        match = "OK" if result.parameter_validity == case.expected else "MISMATCH"
+        lines.append(
+            f"| `{case.name}` | {case.category} | {case.expected.value} | "
+            f"{result.parameter_validity.value} | {match} | "
+            f"{result.papers_found} | {result.values_extracted} | {elapsed:.1f}s |"
+        )
+    lines.append("")
+
+    # ── Per-case detail ──
+    lines.append("## Per-case details\n")
+    for case, result, elapsed in runs:
+        match = (
+            "MATCHES expected"
+            if result.parameter_validity == case.expected
+            else "DOES NOT match expected"
+        )
+        lines.append(f"### `{case.name}` — {case.category}")
+        lines.append(f"- **Description:** {case.description}")
+        lines.append(f"- **Domain:** {case.domain_context}")
+        lines.append(f"- **Expected:** `{case.expected.value}`")
+        lines.append(
+            f"- **Verdict:** `{result.parameter_validity.value}` — {match}"
+        )
+        lines.append(f"- **Reason:** {result.validity_reason or '(none)'}")
+        lines.append(f"- **Empirical:** {result.is_empirical}")
+        lines.append(
+            f"- **Pipeline:** {result.papers_found} papers, "
+            f"{result.values_extracted} values, "
+            f"prior `{result.prior.family.value}` "
+            f"(confidence: {result.prior.confidence.value}, "
+            f"informative: {result.prior.is_informative})"
+        )
+        if result.enrichment is not None:
+            lines.append(
+                f"- **LLM recognized:** {result.enrichment.is_recognized_parameter} "
+                f"(confidence: {result.enrichment.recognition_confidence})"
+            )
+            lines.append(
+                f"- **LLM empirically measured:** {result.enrichment.empirically_measured}"
+            )
+            if result.enrichment.common_terminology:
+                lines.append(
+                    f"- **Terminology:** {', '.join(result.enrichment.common_terminology[:5])}"
+                )
+        if result.validity_signals:
+            sig = result.validity_signals
+            sig_short = {
+                k: v
+                for k, v in sig.items()
+                if k != "probe_result"
+            }
+            lines.append("- **Signals:**")
+            lines.append("  ```json")
+            lines.append("  " + json.dumps(sig_short, indent=2).replace("\n", "\n  "))
+            lines.append("  ```")
+            if "probe_result" in sig and sig["probe_result"]:
+                lines.append(
+                    f"- **LLM probe:** verdict=`{sig['probe_result'].get('verdict')}`, "
+                    f"reason={sig['probe_result'].get('reason')!r}"
+                )
+        if result.warnings:
+            lines.append("- **Warnings:**")
+            for w in result.warnings:
+                lines.append(f"  - {w}")
+        lines.append(f"- **Elapsed:** {elapsed:.1f}s\n")
+
+    output_path.write_text("\n".join(lines))
+    print(f"\nMarkdown report saved: {output_path}")
+
+
+async def main() -> None:
+    settings = get_settings()
+    if not settings.llm_base_url or not settings.llm_api_key:
+        print("ERROR: DISTRIBIRD_LLM_BASE_URL and DISTRIBIRD_LLM_API_KEY required")
+        return
+
+    print(f"LLM: {settings.llm_model} via {settings.llm_base_url}")
+    print(f"Validity check enabled: {settings.enable_validity_check}")
+    print(f"Validity probe enabled: {settings.enable_validity_probe}")
+    print(f"Running {len(CASES)} test cases...")
+
+    # Run sequentially to avoid hammering the LLM/search APIs
+    runs = []
+    for case in CASES:
+        result_tuple = await run_one(case, settings)
+        runs.append(result_tuple)
+
+    output_path = Path(__file__).parent.parent / "bullshitbench_real_llm_results.md"
+    render_markdown(runs, output_path)
+
+    # Brief stdout summary
+    correct = sum(1 for c, r, _ in runs if r.parameter_validity == c.expected)
+    print(f"\n{'=' * 70}")
+    print(f"FINAL: {correct}/{len(runs)} verdicts match expected")
+    print(f"{'=' * 70}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())