A reliability-aware trust layer between perturbation screens and experimental decisions.
Current status: Early implementation — validated datasets, splits, baselines, evaluation, connector gateway, measurement/reliability/trust layers, and decision-package contract. No advanced-model benchmark result or production deployment is claimed.
Biological foundation models aim to learn meaningful representations of cellular state. Such representations can inform perturbation prediction, virtual-cell modeling, and cellular response modeling, but representation quality alone does not establish predictive generalization.
Benchmark conclusions depend on dataset construction, train/test leakage, donor and cell-type overlap, perturbation overlap, preprocessing, and metric selection. A model may appear strong on a random cell split while failing on unseen biology. CellForge is designed to make these evaluation choices explicit and reproducible.
Do modern biological foundation and perturbation-response models predict cellular state under unseen perturbations and biological contexts better than strong simple baselines?
The product question is broader and more practical: which perturbations have reliable measurements, does a model add trustworthy information for this setting, and what independent evidence supports or contradicts further review?
The harness provides dataset validation, reproducible biological splits, simple baselines, evaluation contracts, bounded multi-connector execution, and human-reviewed decision-package contracts. Advanced model adapters, a complete end-to-end decision workflow, and benchmark reports remain in progress.
CellForge complements existing biology platforms instead of trying to replace them. Benchling focuses on sequence design, guide scoring, annotation, and laboratory workflow. Cellxgene focuses on discovering and exploring single-cell datasets. CellForge focuses on what happens after a perturbation is represented: validating the data, comparing predictions with measured cellular responses, testing generalization under biological shifts, and preserving evidence for human review.
The CRISPR workflow can import or register a guide, map it to a target and optional Cas9–RNA–DNA structure, connect it to observed single-cell perturbation data, and save a run-linked review. A structure or guide score is evidence for inspection; it is not proof of editing efficacy or a substitute for laboratory validation.
flowchart TD
D[Public perturbation dataset] --> V[Dataset validation]
V --> S[Reproducible splits]
S --> B[Strong simple baselines]
S --> A[Model adapters]
B --> R[Benchmark runner]
A --> R
R --> M[Biological metrics]
M --> G[Generalization analysis]
G --> O[Results and run manifest]
O --> P[Benchmark report]
This diagram describes the intended architecture. See architecture for responsibilities and boundaries.
Random holdout accuracy is a useful reference but can reward familiarity with donors, perturbations, or cellular states shared with training data.
- Seen-context evaluation: a standard reference within represented biological conditions.
- Unseen perturbation: prediction for a perturbation excluded from training and model selection.
- Unseen donor: transfer across individuals, where donor metadata and experimental replication support it.
- Unseen cell type: transfer of responses across cellular identities.
- Unseen context: transfer under another explicitly defined biological shift.
These are candidate evaluation conditions. No dataset has been selected, and no condition is currently supported. Dataset inspection must establish which holdouts are identifiable and meaningful; confounded or missing metadata cannot justify OOD claims.
The planned MVP starts with one public perturbation dataset, an immune-cell focus where feasible, and one prediction task. It will establish one dataset adapter, deterministic preprocessing, reproducible train/validation/test splits, unseen-perturbation evaluation, and at least one additional OOD evaluation if genuinely supported. Strong simple baselines will precede one appropriate advanced perturbation model. Standardized metrics, machine-readable results, and reproducible reports complete the intended MVP.
If the selected dataset cannot support an additional biological holdout, the limitation will be documented and scope revisited before claiming broader generalization. The repository foundation is established; implementation is now proceeding toward a single end-to-end decision workflow.
A common adapter layer will isolate model-specific preparation, fitting/loading, prediction, and metadata. GEARS, scGPT, Geneformer, and STATE are candidate families for later suitability assessment, not installed integrations or promised task support.
Inclusion in CellForge does not imply that every model is directly designed for every perturbation-prediction task. Each integration must demonstrate compatible inputs, outputs, perturbation semantics, and inference conditions. Representation encoders require a justified prediction mechanism before being compared as perturbation predictors.
Simple baselines are first-class participants. Candidate methods include a control-state predictor, training-derived mean responses, linear models, nearest neighbors, and a simple neural baseline where justified. The final set depends on the dataset and task, particularly what information is available for unseen perturbations.
A sophisticated model should demonstrate measurable improvement over meaningful simple methods under the same evaluation contract. Baselines must receive reasonable tuning and cannot use held-out response information. Neither model superiority nor a particular winner is assumed; null and negative findings are valid.
| Family | Planned measurements | Purpose |
|---|---|---|
| Expression agreement | Pearson, Spearman, MSE, MAE | Assess pattern and magnitude agreement on a declared expression scale |
| Differential-expression recovery | Top-k DEG overlap, DEG precision/recall, direction-of-effect agreement, response-gene recovery | Assess recovery of perturbation effects relative to controls |
| Perturbation-level evaluation | Perturbation ranking, response magnitude agreement | Compare responses across interventions |
| Generalization | Separate metrics for seen conditions, unseen perturbations, donors, cell types, and supported contexts | Reveal where performance transfers or fails |
No single aggregate score will define model quality. Definitions, aggregation units, and limitations must accompany results. See planned metrics.
Future runs will retain fixed seeds, dataset versions and fingerprints, preprocessing fingerprints, split manifests, model and adapter identifiers, checkpoint hashes where possible, inference settings, metric settings, configuration snapshots, environment information, timestamps, and run IDs. Raw predictions and standardized derived metrics will be traceable through result artifacts.
Cached preprocessing will be keyed by data and configuration fingerprints, including split-dependent fitted transformations. A stale cache must never silently substitute for a requested transformation. See the future reproducibility contract; no manifest or cache is implemented yet.
Conceptual example only; this is not an executable command or a completed experiment:
Dataset: One public single-cell perturbation dataset, to be selected
Task: Predict post-perturbation gene expression
Split: Unseen perturbation
Models: Strong simple baseline vs appropriate perturbation model
Evaluation: Expression agreement, DEG recovery, direction-of-effect accuracy
Outputs: Predictions, metrics, run manifest, comparison report
The current repository targets Python 3.10+ with minimal package metadata and no runtime dependencies, executable CLI, or scientific implementation.
The planned scientific stack may use AnnData/Scanpy, NumPy, pandas or Polars, SciPy, scikit-learn, PyTorch, and model-specific libraries as needed. Dependencies will be added when actual integrations justify them.
Engineering direction includes typed Python, a Typer CLI, typed experiment configuration, deterministic seeds, cached preprocessing, run manifests, machine-readable results, and automated reporting. Conceptual future operations include validation, preparation, benchmarking, comparison, and reporting; none are available commands today.
CellForge includes a provider-neutral gateway for expressing several approved connector operations as one bounded workflow call. Registered ChatGPT, Claude, MCP, or local adapters can run independent steps concurrently and dependency-linked steps in order. The gateway applies allowlists, step and concurrency limits, timeouts, and separate approval gates for write and destructive operations. It returns normalized per-step results and a SHA-256 manifest identity.
The gateway does not automatically inherit a platform's connector catalog or credentials. The host must expose and authorize each connector, and an adapter must register it with CellForge. Connector execution also does not bypass the human scientific-review requirements described in the tool-use policy.
src/cellforge/validation/: reserved for structural and biological dataset checks.src/cellforge/measure/: perturbation response summaries on an explicit expression scale.src/cellforge/reliability/: configurable measurement reliability classifications.src/cellforge/trust/: per-perturbation model-vs-baseline trust records.src/cellforge/evidence/: normalized, payload-hashed external evidence records.src/cellforge/contradictions/: conservative rules that surface conflicting evidence.src/cellforge/prioritize/: transparent, decomposed candidate triage for human review.src/cellforge/decision/: deterministic, human-reviewable decision packages.src/cellforge/tasks/: reserved for prediction contracts and, initially, task-specific split responsibilities.src/cellforge/models/: reserved for adapters that isolate external model dependencies.src/cellforge/baselines/: reserved for simple methods using the same task and evaluation contract.src/cellforge/metrics/: reserved for metric definitions and computations.src/cellforge/reports/: reserved for reporting logic; rootreports/will hold report artifacts.configs/: guidance for future typed experiment configuration, without fake runnable examples.data/: data-management policy; datasets and caches are excluded from source control.experiments/: run organization and provenance policy.docs/huggingface.md: optional Hugging Face model/artifact access and credential policy.docs/tool-use-policy.md: allowlisted tools, resource limits, network boundaries, and human-approval gates.docs/crispr-scope.md: bounded CRISPR guide, structure, perturbation, and human-review workflow.docs/: architecture, benchmark design, metric rationale, and reproducibility contracts.tests/unit/andtests/integration/: reserved for future meaningful checks; no benchmark tests exist yet.
Empty directories are reserved with .gitkeep; they contain no Python implementation modules. Dataset and runner modules will be introduced only when needed.
The repository began as a Stage 1 foundation and now includes dataset ingestion/validation, deterministic splits, strong baselines, evaluation contracts, bounded connector orchestration, Norman metadata normalization, measurement summaries, reliability classifications, model-trust records, and an auditable decision-package contract. No advanced-model performance claim or measured product recommendation is included yet.
The repository also contains an early local artifact sandbox and Phase 1 GSE90063 source-validation path. These are development foundations, not a completed production service or benchmark result.
The phased roadmap proceeds from dataset validation and reproducible splits to baselines, a model adapter, one advanced model, biological metrics, a local runner, generalization experiments, artifact hardening, and reports. Benchmark before scaling: one dataset, one task, strong baselines, one advanced model, and good evaluation come first.
CellForge is not a new virtual-cell foundation model, an autonomous scientist, a multi-agent system, a generic RAG application, a chatbot, an MCP server, a workflow orchestration platform, a GPU framework, or a clinical decision system. It is also not a generic backend service platform. The core benchmark does not require an LLM.
The primary contribution is rigorous and reproducible evaluation infrastructure for single-cell perturbation models. Generalization, biological evaluation, and scientific honesty take precedence over leaderboard chasing. Unnecessary APIs, databases, queues, cloud infrastructure, and distributed orchestration are outside the initial scope.