Harden Track 1 runtime and release pipeline

.env.example

@@ -4,8 +4,7 @@
 
 FIREWORKS_API_KEY=
 
-# Evaluator/hackathon-provided endpoint ONLY. Never the public Fireworks API
-# URL (compliance addendum), and there is deliberately no default.
+# Evaluator/hackathon-provided endpoint; there is no public fallback.
 FIREWORKS_BASE_URL=
 
 # Enforced strictly when set; an empty value means deny-all.
@@ -15,3 +14,7 @@ ALLOWED_MODELS=
 # FALLTHROUGH_GEMMA_MODEL=
 # FALLTHROUGH_POLICY=hybrid-baseline
 # FALLTHROUGH_FORMAT_RETRIES=1
+# FALLTHROUGH_LOCAL_BACKEND=none
+# FALLTHROUGH_LOCAL_MODEL=/models/fallthrough.gguf
+# FALLTHROUGH_LOCAL_THREADS=2
+# FALLTHROUGH_LOCAL_CONTEXT_SIZE=1024

.gitattributes

@@ -0,0 +1,9 @@
+* text=auto
+*.py text eol=lf
+*.sh text eol=lf
+*.toml text eol=lf
+*.md text eol=lf
+*.json text eol=lf
+*.jsonl text eol=lf
+.env.example text eol=lf
+.gitignore text eol=lf

.gitignore

@@ -149,7 +149,15 @@ activemq-data/
 
 # Environments
 .env
+.env.*
+!.env.example
 .envrc
+.netrc
+.ssh/
+*.key
+*.pem
+credentials*
+secrets*
 .venv
 env/
 venv/

Dockerfile

@@ -1,14 +1,22 @@
-FROM python:3.12.4-slim-bookworm
+FROM python:3.10.20-slim-bookworm
+
+ARG FALLTHROUGH_PACKAGED_ROUTE_TABLE
+ARG FALLTHROUGH_PACKAGED_POLICY_EVIDENCE
 
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
     PIP_DISABLE_PIP_VERSION_CHECK=1 \
-    PIP_NO_CACHE_DIR=1
+    PIP_NO_CACHE_DIR=1 \
+    FALLTHROUGH_PACKAGED_ROUTE_TABLE=${FALLTHROUGH_PACKAGED_ROUTE_TABLE} \
+    FALLTHROUGH_PACKAGED_POLICY_EVIDENCE=${FALLTHROUGH_PACKAGED_POLICY_EVIDENCE}
 
 WORKDIR /app
 
 COPY pyproject.toml requirements.lock README.md LICENSE ./
 COPY src ./src
+# Schema/README are always present. Measured artifacts are added only after
+# compliant evidence exists and are activated through the two non-secret build args.
+COPY policy ./policy
 
 RUN python -m pip install --no-cache-dir --no-deps --requirement requirements.lock \
     && python -m pip install --no-cache-dir --no-build-isolation --no-deps .

docs/amd-pod-environment.md

@@ -13,17 +13,18 @@ Probe date: 2026-07-11.
 | Docker | unavailable |
 | Fireworks environment | not injected on this pod |
 
-The pod is a local-model feasibility and measurement environment. It is not the
-final scoring container, and its GPU must not become a hard routing assumption.
-Use it for vLLM compatibility, latency/VRAM measurements, and shadow-mode route
-traces. Keep candidate selection measurement-driven.
+The pod is training, fine-tuning, export, and quantization infrastructure. It is
+not the final scoring container, and its GPU must not become a routing
+assumption. Accuracy/latency evidence used for final policy decisions must come
+from the exported GGUF through Fallthrough's llama.cpp worker under 4 GB RAM and
+2 vCPU.
 
 Repository workflow:
 
 ```bash
 cd /workspace/fallthrough
 git pull
-python -m pip install -e .
+python -m pip install -e '.[dev]'
 pytest
 python scripts/probe_environment.py
 ```
@@ -37,14 +38,10 @@ git remote set-url origin git@github.com:latch-lab/fallthrough.git
 Do not print, copy, or commit that key. Remove it from GitHub when the pod is
 retired. The final scoring container must not assume the pod SSH configuration.
 
-The optional backend follows vLLM's documented offline `LLM`/`SamplingParams`
-surface and uses `LLM.generate(..., use_tqdm=False)`. It is lazy, defaults off,
-forces Hugging Face/Transformers offline mode, accepts safetensors only, and is
-shadow-only until final runtime compatibility and held-out accuracy are known.
-There is no download opt-in. Pre-stage audited weights outside the repository.
-Local paths are content-hashed before load; cached identifiers require a full
-40-hex immutable revision. Recorded experiment fingerprints include that
-checkpoint identity, the non-secret model label, generation policy, vLLM
-configuration/version, and detected GPU architecture/VRAM so incomparable runs
-do not share an aggregation key. Observed VRAM is global device-0 use and should
-be measured on an otherwise idle pod. See the official [vLLM 0.16 offline LLM API](https://docs.vllm.ai/en/v0.16.0/api/vllm/entrypoints/llm/).
+The Fallthrough local backend uses a content-hashed GGUF file through a pinned
+`llama-cpp-python`/llama.cpp runtime. It is lazy, CPU-only, defaults off, has no
+model identifier or download path, and remains shadow-only until held-out and
+container-constraint gates pass. Its owned worker is killed on a native
+generation timeout. Recorded fingerprints include artifact, generation,
+runtime, prompt-policy, CPU/cgroup, and memory evidence. See the official
+[llama-cpp-python API](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/).

docs/developer-handoff-2026-07-11.md

@@ -1,5 +1,9 @@
 # Fallthrough developer handoff
 
+> Historical handoff: the vLLM plan in this document is superseded by
+> [final-runtime-architecture.md](final-runtime-architecture.md). Preserve this
+> file only as an audit trail of the earlier implementation state.
+
 Date: 2026-07-11  
 Repository: `C:\Users\hepha\PycharmProjects\fallthrough`  
 Remote: `git@github.com:latch-lab/fallthrough.git`  
@@ -21,8 +25,9 @@ Allowed final-answer paths are deterministic in-container computation, a model
 executing inside the scored container, or the evaluator-provided Fireworks
 endpoint. No other remote inference, search, RAG, vector, or information service
 is allowed. Outbound access is deny-by-default except for the injected
-`FIREWORKS_BASE_URL`. Fireworks models must come from `ALLOWED_MODELS` when that
-variable is supplied. Never substitute the public Fireworks API URL.
+`FIREWORKS_BASE_URL`. Fireworks models must come from `ALLOWED_MODELS`. Never
+invent an endpoint fallback; development may explicitly supply the standard
+Fireworks URL, while scoring uses the harness-injected URL unchanged.
 
 Local inference counts as zero scored Fireworks tokens. Gemma must do meaningful
 work through Fireworks to qualify for the additional Gemma award; decorative
@@ -72,7 +77,7 @@ Adopted concepts/components include:
   models;
 - classifier, label, arithmetic, and code-fence improvements.
 
-Rejected components include any behavior that used a public Fireworks URL,
+Rejected components include any behavior that invented a Fireworks URL fallback,
 bypassed an empty allowlist, weakened token accounting or deadlines, introduced
 an evaluator-incompatible service, or otherwise violated the compliance
 addendum. SymPy/mpmath were removed to reduce the runtime footprint.

docs/final-runtime-architecture.md

@@ -0,0 +1,132 @@
+# Final Track 1 runtime architecture
+
+Recorded: 2026-07-11. This decision supersedes the earlier vLLM/local-backend
+plan. The compliance addendum still governs allowed answer-generation paths and
+network access.
+
+## Final grader contract
+
+The scored container is `linux/amd64` with:
+
+- 4 GB RAM;
+- 2 vCPU;
+- startup under 60 seconds;
+- under 30 seconds per task;
+- 10-minute total runtime;
+- compressed image under 10 GB.
+
+It reads a top-level list of `{task_id, prompt}` objects from
+`/input/tasks.json`, writes a top-level list of `{task_id, answer}` objects to
+`/output/results.json`, and exits zero after a successful non-interactive batch.
+Answers are in English. The public image must have a pullable `linux/amd64`
+manifest. The harness injects `FIREWORKS_API_KEY`, `FIREWORKS_BASE_URL`, and
+`ALLOWED_MODELS`.
+
+## One hybrid implementation
+
+The only eligible route types are:
+
+1. deterministic in-container computation;
+2. an optional compact local GGUF model through llama.cpp;
+3. an allowlisted model through the injected Fireworks endpoint.
+
+Routing is measured by task family. Neither local-first nor Fireworks-first is a
+policy. Deterministic routes commit only when their answer can be proved. A
+local call is useful only when its held-out accuracy/token benefit exceeds its
+latency, memory, startup, image-size, and timeout risk.
+
+The local model is optional. If no candidate improves the probability of
+clearing the accuracy gate with fewer Fireworks tokens under the exact final
+constraints, ship deterministic plus Fireworks and omit llama.cpp/model weight
+overhead.
+
+## Local model boundary
+
+Fallthrough has one local generative engine: llama.cpp. vLLM is development-pod
+history, not a Fallthrough runtime backend.
+
+Any shipped local model must be:
+
+- GGUF;
+- approximately 2B-3B parameters;
+- four-bit quantized or smaller;
+- CPU-compatible;
+- baked into the image;
+- proven under 4 GB RAM and 2 vCPU.
+
+The Participant Guide presents 2B-3B four-bit as safe sizing guidance, not an
+official eligibility range. The narrower size/quantization bullets above remain
+Fallthrough's conservative project promotion gate until exact-image evidence
+justifies revisiting them.
+
+Artifact qualification is fail-closed. A bounded little-endian GGUF v2/v3
+inspector rejects shards, counts stored tensor elements, reads the declared
+nominal majority file type, and records tensor-type distribution. The owned
+worker then cross-checks parameter count and file type through llama.cpp's
+loaded-model APIs. Unknown/removed types, inconsistent tensor majorities, or a
+runtime/header mismatch cannot satisfy the promotion gate; header facts alone
+do not prove accuracy or exact-container fitness.
+
+Task timing evidence uses parent end-to-end wall latency, with backend latency
+retained separately. Cold-startup qualification must come from an external
+container wall-clock measurement; an in-process load timer is diagnostic only.
+
+The AMD pod may train or fine-tune with PyTorch/ROCm. Routing evidence must be
+collected after merge/export, GGUF conversion, quantization, and execution via
+the exact llama.cpp integration intended for the image. Pod GPU performance is
+not final-runtime evidence.
+
+Local inference remains disabled unless a valid baked GGUF exists and a measured
+route table explicitly selects it.
+
+## Fireworks boundary
+
+The scored runtime has no endpoint default or public fallback. Every call uses
+the injected `FIREWORKS_BASE_URL`; every selected model must appear in the
+runtime `ALLOWED_MODELS`. Missing or invalid configuration fails closed, and
+actual prompt/completion usage is recorded.
+
+Measured policies use `fw_model:<exact ALLOWED_MODELS entry>` for every remote
+candidate, including Gemma. The late-bound `fw_gemma` alias is reserved for the
+unmeasured baseline and `gemma-only` convenience mode so allowlist reordering
+cannot silently apply measured evidence to a different model.
+
+Development may use local HTTP mocks, an explicitly configured standard
+Fireworks endpoint, or the injected hackathon endpoint. The runtime has no
+endpoint default or fallback, and scoring uses the injected URL unchanged.
+
+Gemma through Fireworks must perform genuine semantic or answer-generation work.
+Other allowed Fireworks models remain eligible when measurements improve the
+main Track 1 strategy.
+
+Local or deterministic final answers and a zero-Fireworks-call batch are valid.
+The undisclosed accuracy gate is primary; qualifying policies are compared by
+total proxy tokens, including added prompt tokens and generated completion
+tokens.
+
+## Evidence required before policy changes
+
+For every task family and route candidate, record:
+
+- accuracy on calibration and untouched holdout cases;
+- Fireworks prompt and completion tokens;
+- latency and startup/load time;
+- invalid-output and runtime-failure rates;
+- cgroup-wide peak memory (worker RSS is diagnostic fallback only);
+- the exact model artifact, runtime, prompt, generation, and hardware identity.
+
+Initial route expectations are hypotheses only: deterministic arithmetic/list
+operations/supported logic, likely Fireworks for factual and unverifiable
+semantic work, and measured comparisons elsewhere.
+
+## Delivery order
+
+1. Keep the current deterministic/Fireworks container healthy.
+2. Obtain a first official evaluator result and Fireworks per-family baselines.
+3. Evaluate compact GGUF candidates using the exact llama.cpp runtime.
+4. Enable a local final-answer route only when constrained held-out measurements
+   justify it.
+5. Build/test the final image with `--platform linux/amd64 --memory=4000000000 --cpus=2`
+   and verify all startup, task, batch, and compressed-image limits.
+6. Push the exact tag, pull it from a clean machine, and verify the public
+   registry manifest contains `linux/amd64` before spending a submission slot.