GitHub - HUN-REN-AI1Science/Distribird: Automated Bayesian prior construction from scientific literature

Automated Bayesian prior construction from scientific literature

Distribird turns a parameter name and description into a fully cited, publication-ready prior distribution. It searches Semantic Scholar, OpenAlex, and LLM deep-research agents in parallel, extracts numerical values from papers, and fits the best-matching scipy.stats distribution via AIC selection.

"I need a prior for maximum leaf area index of maize."

➜ truncated_normal(mu=5.2, sigma=1.5, a=0, b=12) - fitted from 6 peer-reviewed sources with full citations.

Why Distribird?

Bayesian calibration requires informative priors, but building them from literature is tedious. Researchers default to flat priors, losing valuable domain knowledge. Distribird closes that gap: describe your parameter, get a defensible prior in seconds.

Architecture

START
  └─ enrich ───────────────(route_after_enrich)──► validity_check   # unrecognised → short-circuit
       │ recognised
       ▼
   search_strategy ─► query_gen ─► search ─► relevance_judge
                                                   │
                              (route_after_deliberation)
                              ├─ cross_enrich ─┐
                              └────────────────┴─► fetch_fulltext ─► extract ─► quality_gate
                                                                                     │
                                              (route_after_quality_gate)
                                              ├─ broaden_search ─► query_gen   (Loop B: domain)
                                              ├─ refine_search ──► search       (Loop A)
                                              ├─ refine_extraction ─► quality_gate (Loop C)
                                              └─ synthesize ─► validity_check ─► END

Multi-agent search - Semantic Scholar, OpenAlex, and LLM deep-research agents run concurrently; a moderator LLM selects the best papers via deliberation.

Relevance scoring - An LLM-based relevance judge scores each paper before extraction. When multiple high-relevance papers are found, the pipeline routes through cross-enrichment (citation snowballing + follow-up queries) to discover additional sources.

Feedback loops - A quality gate inspects extraction results and can trigger search refinement (new queries) or extraction refinement (web-assisted re-extraction) before falling through to synthesis.

Full-text retrieval: FetchFulltext downloads each paper's PDF and extracts its text. When a download is blocked (for example a publisher returns 403), it falls back to open-access mirrors via Unpaywall, and optionally to a headless stealth browser for sites behind JavaScript bot walls. See "Full-text PDF fallback" under Install.

Full-paper reading (page-turning): Extraction reads the entire paper rather than only its opening pages - calibration values and parameter tables often sit deep in Methods, Results, or appendices. A paper that fits the model's context window is read in one call; a longer one is split into overlapping "pages" sized to the configured window, each read separately, and the extracted values are merged and de-duplicated. Sized to your model's context via DISTRIBIRD_LLM_MAX_CONTEXT_TOKENS (see Configure).

Validity defense - Every request is classified as VALID, SUSPICIOUS, LIKELY_INVALID, or UNKNOWN. When the enrichment LLM does not recognise the parameter, the pipeline short-circuits past search, extraction, and synthesis straight to the validity node, saving roughly 80–95% of wall-clock time and LLM tokens on out-of-scope requests. Ambiguous (SUSPICIOUS) cases trigger a single second-opinion LLM probe.

Budget-bounded - IterationBudget caps every loop to guarantee termination.

Live progress - The pipeline streams node-by-node updates to the UI, showing which step is running, paper/value counts, and per-parameter progress bars.

Quickstart

Install

pip install distribird

Development install

git clone https://github.com/HUN-REN-AI1Science/Distribird.git
cd distribird
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest                     # 386 tests, all passing (5 network tests deselected by default)

Optional: full-text PDF fallback (stealth browser)

Some publishers (MDPI is one) serve PDFs behind a JavaScript bot wall that returns 403 to plain HTTP clients. Distribird can get past these with a headless stealth browser (Camoufox). It is off by default and kept as a separate extra because it downloads a large browser binary.

pip install "distribird[stealth]"
python -m camoufox fetch          # one-time browser download, about 1.3 GB

Then turn it on (see Configure):

DISTRIBIRD_ENABLE_STEALTH_FETCH=true

This needs a host that can run a real browser, so it does not work on Streamlit Community Cloud, where it is skipped automatically. The lighter Unpaywall mirror fallback still runs everywhere.

Configure

Distribird reads configuration from environment variables (prefix DISTRIBIRD_) or a .env file in the project root.

# .env (or export these in your shell)
DISTRIBIRD_LLM_BASE_URL="http://localhost:4000"   # any OpenAI-compatible endpoint
DISTRIBIRD_LLM_API_KEY="your-key"
DISTRIBIRD_LLM_MODEL="gpt-4o"
DISTRIBIRD_LLM_MAX_CONTEXT_TOKENS="255000"        # your model's context window; sizes page-turning
DISTRIBIRD_SEMANTIC_SCHOLAR_API_KEY=""            # optional, increases rate limits
DISTRIBIRD_OPENALEX_EMAIL=""                       # your email; enables the OA mirror fallback
DISTRIBIRD_ENABLE_OA_MIRROR_FALLBACK="true"        # default; Unpaywall mirrors over plain HTTP
DISTRIBIRD_ENABLE_STEALTH_FETCH="false"            # opt-in stealth browser (see Install)
DISTRIBIRD_ENABLE_HTML_FULLTEXT="true"             # default; extract text from HTML full-text pages
DISTRIBIRD_ENABLE_MARKDOWN_FULLTEXT="true"         # default; read PDFs as Markdown (keeps tables)
DISTRIBIRD_FULLTEXT_MARKDOWN_OCR="false"           # default off; OCR scanned pages (heavy, not on Streamlit)

Full-text PDF fallback. When a paper's PDF URL is blocked, Distribird tries the direct URL first, then open-access mirrors through Unpaywall, then the stealth browser if it is enabled. The Unpaywall step needs DISTRIBIRD_OPENALEX_EMAIL set to a real address and is skipped without one. The stealth step needs the stealth extra and a host that can run a browser; it is skipped on Streamlit Community Cloud. With both off, only the direct URL is used.

When a URL serves HTML instead of a PDF, Distribird extracts the article text from the HTML (for example PubMed Central full-text pages and many repository pages). A quality check rejects bot-challenge pages and thin abstract-only pages so they do not pollute extraction. This is on by default; set DISTRIBIRD_ENABLE_HTML_FULLTEXT=false to turn it off, or raise DISTRIBIRD_HTML_FULLTEXT_MIN_CHARS to be stricter about what counts as an article.

Structure-preserving PDF reading. PDFs are read as Markdown (via pymupdf4llm) rather than flattened plain text, so tables become pipe-tables (| param | value | unit |), headings are kept, and page boundaries are marked [page N]. This matters because the calibration and species-parameter tables where prior-relevant numbers live are exactly what plain text extraction garbles - Markdown keeps each value tied to its row label, column header, and unit. On by default; set DISTRIBIRD_ENABLE_MARKDOWN_FULLTEXT=false to fall back to plain extraction. OCR for scanned/image-only pages is off by default (DISTRIBIRD_FULLTEXT_MARKDOWN_OCR=true to enable) because it loads a heavy layout/onnxruntime stack that memory-limited hosts such as Streamlit Community Cloud cannot run; the fetched open-access papers are digital-born with real text layers, so OCR is rarely needed.

Full-paper reading (page-turning). Distribird reads each paper in full instead of truncating it. Set DISTRIBIRD_LLM_MAX_CONTEXT_TOKENS to your model's context window so the per-call page size is computed correctly: the usable character budget is (max_context_tokens − reserved_answer_tokens) × chars_per_token − prompt_overhead. A paper within that budget is read in a single call; a longer one is split into overlapping pages (DISTRIBIRD_EXTRACTION_CHUNK_OVERLAP_CHARS), each extracted separately, then the values are merged and de-duplicated. DISTRIBIRD_EXTRACTION_MAX_CHUNKS caps pages per paper - when exceeded, Methods/Results pages are kept first and a warning is logged. Lower the context setting for a small local model (paging activates automatically); raise it on a large-context model to read most papers in one call. Each extra page is one more LLM call, so it counts against the per-parameter call budget. Defaults: 255k context, 4k reserved, 3.5 chars/token, 8 pages, 1500-char overlap. DISTRIBIRD_FULLTEXT_STORAGE_MAX_CHARS (default 400k) only caps how much text is stored per paper at fetch time.

The stealth browser follows DOI and handle redirects to the real publisher before clearing the bot challenge, so publishers reached through a doi.org link (MDPI and similar) are recovered. Some publishers run enterprise bot protection that a headless browser cannot pass: ScienceDirect and Elsevier (PerimeterX), Wiley and Hindawi (Cloudflare managed challenge), and some institutional repositories that block direct file access. PDFs behind these are expected misses; the paper still contributes its title and abstract.

Sidebar behaviour in the Streamlit UI:

Settings provided in .env are used automatically - no manual input needed.
Settings not provided in .env appear as required fields in the sidebar; the user must fill them in before generation can start.
An "Override configured settings" toggle lets users temporarily replace .env values without editing the file.
Literature source toggles (Semantic Scholar, OpenAlex, LLM Web Search, LLM Deep Research) are always visible and control which connection fields are required.

Use

Python

import asyncio
from distribird import run_parameter, ParameterInput, ConstraintSpec

result = asyncio.run(run_parameter(
    ParameterInput(
        name="max_lai",
        description="Maximum leaf area index of maize",
        unit="m2/m2",
        domain_context="Biome-BGCMuSo maize crop modeling",
        constraints=ConstraintSpec(lower_bound=0, upper_bound=12),
    )
))

print(result.prior.display_name())   # truncated_normal(mu=5.2, sigma=1.5, a=0, b=12)
print(result.prior.n_sources)        # 6
print(result.prior.confidence.value) # high

REST API

distribird-api                          # starts on :8000

curl -u demo:changeme -X POST http://localhost:8000/api/v1/parameter \
  -H "Content-Type: application/json" \
  -d '{"name":"max_lai","description":"Maximum leaf area index of maize","unit":"m2/m2"}'

Streamlit UI

Try the hosted version at distribird.streamlit.app, or run locally:

streamlit run src/distribird/ui/app.py

Using Distribird as a library

Configure programmatically

Distribird is configured through a single Settings object. Values are resolved in increasing order of precedence, so you can mix all three:

Built-in defaults - sensible values for every option.
A .env file in the working directory (copy .env.example to .env).
Environment variables prefixed with DISTRIBIRD_ (e.g. DISTRIBIRD_LLM_MODEL).
Keyword arguments passed to Settings(...) in code - highest precedence.

import asyncio
from distribird import Settings, run_parameter, ParameterInput, ConstraintSpec

# Anything you don't set falls back to env / .env / defaults.
settings = Settings(
    llm_base_url="http://localhost:4000",   # any OpenAI-compatible endpoint
    llm_api_key="sk-...",
    llm_model="gpt-4o",
)

param = ParameterInput(
    name="max_lai",
    description="Maximum leaf area index of maize",
    unit="m2/m2",
    domain_context="Biome-BGCMuSo maize crop modeling",
    constraints=ConstraintSpec(lower_bound=0, upper_bound=12),
)

result = asyncio.run(run_parameter(param, settings=settings))

Call run_parameter(param) with no settings to use the ambient environment / .env.

Inspect and export the result

prior = result.prior
prior.family.value       # "truncated_normal"
prior.params             # {"mu": 5.2, "sigma": 1.5, "a": 0.0, "b": 12.0}
prior.confidence.value   # "high" | "medium" | "low" | "none"
prior.n_sources          # 6
prior.display_name()     # "truncated_normal(mu=5.2, sigma=1.5, a=0, b=12)"
for ev in prior.evidence:
    print(ev.year, ev.title, ev.doi)

# Ready-to-run sampling code for a single result:
from distribird.export.python_export import export_single_python
from distribird.export.r_export import export_single_r

print(export_single_python(result))   # scipy.stats
print(export_single_r(result))        # R

Run several parameters at once

from distribird import run_batch, export_json, export_python

batch = asyncio.run(run_batch([param_a, param_b, param_c], settings=settings))
print(export_json(batch))     # every prior + citations, as JSON
print(export_python(batch))   # one scipy.stats block per parameter

run_batch evaluates parameters concurrently, bounded by max_parallel_parameters.

Configuration recipes

Common setups - set these as Settings(...) kwargs, DISTRIBIRD_* env vars, or .env lines.

Small local model (LM Studio / llama.cpp / Ollama via LiteLLM). Point at the local endpoint and lower the context window so full-paper page-turning activates automatically:

Settings(
    llm_base_url="http://localhost:1234/v1",
    llm_model="qwen2.5-7b-instruct",
    llm_max_context_tokens=8000,   # papers larger than this are read in overlapping pages
)

Large hosted model. Raise the context window so most papers are read in a single call:

Settings(llm_model="gpt-4o", llm_max_context_tokens=128000)

Reproducible / deterministic runs. Pin the sampling temperatures and seed:

Settings(
    llm_temperature_precise=0.0,
    llm_temperature_creative=0.0,
    llm_temperature_deliberation=0.0,
    llm_seed=42,
)

Reach more paywalled PDFs. Enable the open-access mirror fallback (needs a real email), the stealth browser (needs the stealth extra), and/or an institutional proxy:

Settings(
    openalex_email="you@uni.example",
    enable_oa_mirror_fallback=True,
    enable_stealth_fetch=True,                       # requires: pip install "distribird[stealth]"
    fulltext_proxy_url="http://user:pass@proxy:8080",
)

Faster / cheaper runs. Shrink the corpus and cap the work the pipeline may do:

Settings(
    max_papers_per_query=10,
    max_papers_total=20,
    max_search_queries=3,
    enable_llm_deep_research=False,
    total_llm_calls_max=15,        # hard ceiling on LLM calls per parameter
)

Schema-only / offline. Importing data models never pulls in the heavy orchestration stack (LangGraph, OpenAI, PyMuPDF), so this is dependency-light and needs no LLM:

from distribird import ParameterInput, ConstraintSpec, FittedPrior  # no network, no LLM

Configuration reference

All configuration options (env var = DISTRIBIRD_ + the upper-cased name)

LLM

Setting	Default	Purpose
`llm_base_url`	`""`	OpenAI-compatible endpoint (required)
`llm_api_key`	`""`	API key for the endpoint
`llm_model`	`gemini-3-pro`	Model id
`llm_timeout`	`120.0`	Per-request timeout (seconds)
`llm_max_retries`	`3`	SDK retries on transient errors
`llm_seed`	`None`	Sampling seed for reproducibility
`llm_temperature_precise`	`0.0`	Extraction / relevance / validity / query-gen
`llm_temperature_creative`	`0.3`	Enrichment / refinement
`llm_temperature_deliberation`	`0.1`	Multi-agent moderator

Context window & full-paper reading

Setting	Default	Purpose
`llm_max_context_tokens`	`255000`	Your model's context window; sizes page-turning
`llm_reserved_answer_tokens`	`4000`	Headroom reserved for the model's answer
`llm_chars_per_token`	`3.5`	Chars-per-token estimate for budgeting
`extraction_max_chunks`	`8`	Max pages read per paper
`extraction_chunk_overlap_chars`	`1500`	Overlap between consecutive pages
`fulltext_storage_max_chars`	`400000`	Cap on stored text per paper (memory guard)

Literature sources

Setting	Default	Purpose
`enable_semantic_scholar`	`True`	Use the Semantic Scholar source agent
`semantic_scholar_api_key`	`""`	Optional; raises rate limits
`enable_openalex`	`True`	Use the OpenAlex source agent
`openalex_email`	`""`	Your email; enables the OA mirror fallback
`llm_web_search`	`True`	Use the LLM web-search agent
`enable_llm_deep_research`	`False`	Use the deep-research agent
`deep_research_base_url` / `_api_key` / `_model`	`""` / `""` / `o4-mini-deep-research`	Deep-research endpoint
`enable_deliberation`	`True`	Reconcile agents via a moderator LLM
`deliberation_model`	`None`	Override model for the moderator
`enable_relevance_judgment`	`True`	Score paper relevance before extraction
`enable_snowballing`	`True`	Follow citations of key papers
`snowball_max_seeds` / `snowball_limit_per_seed`	`3` / `10`	Snowball breadth

Corpus sizing & full-text fetch

Setting	Default	Purpose
`max_papers_per_query`	`20`	Papers kept per query
`max_search_queries`	`5`	Queries generated per parameter
`max_papers_total`	`50`	Cap on the aggregated corpus
`enable_oa_mirror_fallback`	`True`	Unpaywall open-access mirrors (HTTP)
`enable_stealth_fetch`	`False`	Headless stealth browser (opt-in extra)
`enable_pmc_resolver` / `enable_mdpi_xml_resolver` / `enable_arxiv_fallback`	`True`	Source-specific open-access resolvers
`enable_html_fulltext`	`True`	Extract article text from HTML pages
`enable_markdown_fulltext`	`True`	Read PDFs as structure-preserving Markdown
`fulltext_markdown_ocr`	`False`	OCR scanned pages (heavy)
`fulltext_user_agent` / `fulltext_proxy_url`	`""`	Override UA / route fetches via a proxy

Synthesis & feedback loops

Setting	Default	Purpose
`enable_fulltext_relevance`	`True`	Relevance-weighted, confidence-capped synthesis
`enable_progressive_search`	`True`	Scope planning + domain broadening
`domain_broadening_max` / `domain_broadening_min_relevant`	`2` / `2`	Broadening escalation control
`min_values_for_synthesis`	`2`	Minimum values before fitting
`search_refinement_max` / `cross_enrichment_max` / `extraction_refinement_max`	`2` / `1` / `1`	Per-loop iteration caps
`total_llm_calls_max`	`30`	Hard ceiling on LLM calls per parameter
`max_parallel_parameters`	`3`	Concurrency for `run_batch`

Validity, server, debugging

Setting	Default	Purpose
`enable_validity_check` / `enable_validity_probe`	`True`	Out-of-scope-parameter detection
`auth_username` / `auth_password`	`demo` / `changeme`	API/UI basic-auth (override before exposing)
`api_host` / `api_port`	`0.0.0.0` / `8000`	API bind address
`debug_trace` / `trace_output_dir`	`False` / `logs/traces`	Structured execution trace

Prior Fitting Strategy

Evidence	Method	Confidence
5+ values	AIC across Normal, Truncated Normal, Gamma, Log-Normal, Beta	High
2 – 4 values	Moment matching with widened σ	Medium
1 value	Wide Normal centered on value	Low
0 values	Jeffreys / wide uninformative prior	None

All fitted distributions respect user-specified physical constraints (bounds).

Export Formats

from distribird import export_json, export_r, export_python

Format	Output
JSON	Parameter name, family, params, citations, confidence
R	Executable R script with distribution calls
Python	`scipy.stats` code ready for MCMC samplers

Demo

A complete worked example using five Biome-BGCMuSo maize parameters:

python examples/maize_bgcmuso/demo.py

Testing

pytest                 # 386 tests (5 network tests deselected)
ruff check src/ tests/ # lint
mypy src/distribird/      # type checking (strict)

Name		Name	Last commit message	Last commit date
.devcontainer		.devcontainer
.github/workflows		.github/workflows
.streamlit		.streamlit
assets		assets
examples		examples
src/distribird		src/distribird
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
CITATION.cff		CITATION.cff
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
streamlit_app.py		streamlit_app.py

Navigation Menu

Uh oh!

Repository files navigation

Why Distribird?

Architecture

Quickstart

Install

Configure

Use

Using Distribird as a library

Configure programmatically

Inspect and export the result

Run several parameters at once

Configuration recipes

Configuration reference

Prior Fitting Strategy

Export Formats

Demo

Testing

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Why Distribird?

Architecture

Quickstart

Install

Configure

Use

Using Distribird as a library

Configure programmatically

Inspect and export the result

Run several parameters at once

Configuration recipes

Configuration reference

Prior Fitting Strategy

Export Formats

Demo

Testing

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages