Status: committed before any training results exist. Changes after this point are logged in the Amendments section at the bottom, not edited silently into the text above.
How does validation loss scale with non-embedding parameters and training compute for small decoder-only transformers, trained on FineWeb-Edu, in the roughly 1M–30M non-embedding parameter range, on free-tier T4 GPUs?
- Architecture: decoder-only transformer, RoPE, RMSNorm, SwiGLU MLP, pre-norm residual blocks, tied embeddings.
- Data: FineWeb-Edu (
sample-10BTsubset), ~1.3B training tokens, a fixed held-out validation set. - Tokenizer: a 16K-vocabulary BPE tokenizer trained from scratch on a sample of the corpus.
- Hardware: Kaggle Notebooks, T4 x2, free tier.
- Model sizes: five sizes spanning roughly 1M–30M non-embedding
parameters (see
configs/).
- No claims about model behavior above ~30M parameters.
- No claims about instruction-following or chat behavior; this is a base-model pretraining study only.
- No multi-node or FSDP claims; this project uses at most 2 GPUs on a single node.
- Benchmark scores (HellaSwag, ARC-Easy) are reported for completeness but are expected to sit near chance at this scale, and are not the headline result. Validation loss and the fitted scaling exponent are.
- Five model sizes trained, 3 seeds each on the three smallest sizes.
- A fitted power law (loss vs. compute, loss vs. params) with reported uncertainty (bootstrapped across seeds).
- My medium-size run within ~5% of a reference nanoGPT run's loss on the same data and token budget (sanity baseline).
- Every reported number reproducible from a single committed YAML config plus a fixed git commit hash.
(Written now, before Phase 4's throughput measurement. Any change made after seeing real throughput numbers is logged below, not merged silently into this section.)
- Token budget: ~20 tokens per non-embedding parameter per model.
- Learning rate: swept on a log grid at the two smallest sizes and the largest size; interpolated for the middle sizes.
- Optimizer: AdamW, linear warmup + cosine decay, gradient clipping at 1.0.
| Risk | Mitigation |
|---|---|
| Kaggle weekly GPU quota runs out mid-sweep | Weekly GPU-hour budget table in docs/gpu_ledger.md; smallest/cheapest runs scheduled first; largest size is explicitly droppable without invalidating the study |
| fp16 training instability (no native bf16 on T4) | RMSNorm/RoPE computed in fp32; GradScaler scale factor logged; fallback to a lower LR if loss spikes |
| Kaggle session dies mid-run (9-hour cap, preemption) | Checkpoints (model, optimizer, scaler, scheduler, RNG state, dataloader offset) pushed to the Hugging Face Hub every ~15 minutes; resume correctness tested explicitly in Phase 3 |
| Data pipeline bug silently invalidates results | Fixed, hashed validation set; decode-and-inspect checks after tokenization; nanoGPT baseline comparison at medium size |
| Scope creep extends the timeline indefinitely | Stretch goals (muP, GQA, MoE, long-context) frozen until the Definition of Done in the main to-do list is fully checked |
- Target: 6 weeks part-time, ~5–8 hours/week including GPU time.
- Hard stop date: set this before Phase 1 and put it here.
(Empty at project start. Each entry below should be dated and explain what changed and why — e.g., "2026-XX-XX: measured T4 throughput came in 30% below the estimate used to build the size table; XL token budget reduced from 630M to 440M tokens to stay inside the weekly GPU quota.")