TinyScale

Research question: How does validation loss scale with non-embedding parameters and training compute for small decoder-only transformers, trained from scratch on FineWeb-Edu, in the ~1M–30M parameter range, on free-tier Kaggle T4 GPUs?

Full scope, success criteria, and risk register: docs/CHARTER.md

Status

• Phase 0 — accounts, repo, CI, config system, smoke test
• Phase 1 — data pipeline
• Phase 2 — model implementation and unit tests
• Phase 3 — training loop (fp16, checkpoint/resume)
• Phase 4 — performance profiling and multi-GPU
• Phase 5 — the scaling study
• Phase 6 — ablations and evaluation
• Phase 7 — packaging and write-up

Headline Result

(to be filled in after Phase 5 — the fitted scaling exponent and the loss-vs-compute plot go here)

Results Table

(to be filled in — model size, params, tokens, final val loss, val bits-per-byte, GPU-hours, for every run)

Reproduce It

git clone https://github.com/YOUR_USERNAME/tinyscale.git
cd tinyscale
pip install -e ".[dev]"
pytest -q                               # unit tests, CPU only
python -m tinyscale.train.run --config configs/xs_seed0.yaml

Hardware and Cost

(to be filled in from docs/gpu_ledger.md — total GPU-hours, hardware used, $0 cost, link to the ledger)

What Didn't Work

(to be filled in — at least one real failure and how it was diagnosed)

Limitations

• Trained at a small scale (≤30M non-embedding parameters) on limited free-tier compute; scaling-law extrapolation beyond this range is not validated.
• fp16 (not bf16) throughout, since Kaggle's T4s lack native bf16 support.
• Zero-shot benchmark scores at this scale are expected to sit near chance and are reported for completeness, not as the primary result.

Project Structure

src/tinyscale/
  model/   # transformer implementation
  data/    # tokenization, shard loading
  train/   # training loop, checkpointing
  eval/    # scaling-law fitting, benchmarks
configs/   # one YAML per run — the single source of truth per experiment
docs/      # charter, GPU ledger