Augur

A personal prediction & calibration tracker, built on Couchbase.

Anyone can say they're right most of the time. Augur makes you prove it. You log predictions as probabilities — "70% chance I ship this before Friday" — resolve them later as true or false, and Augur tells you whether your 70%s actually happen 70% of the time.

It scores you with a Brier score and log loss, and plots a calibration curve: the honest, uncomfortable picture of how well your confidence matches reality. It's the core idea behind superforecasting, shrunk to a single-file dashboard.

---

Architecture

Browser (public/) ──HTTP──> Express API (src/routes.js)
                                  │
                                  ├─ repository.js  ── Couchbase SDK ──> Couchbase Server
                                  │      KV · mutateIn · N1QL              (bucket: augur
                                  │                                         scope:  forecasts
                                  └─ calibration.js (pure scoring math)     collection: predictions)

• No build step. The frontend is plain HTML/CSS/JS; the SVG calibration chart is hand-drawn.
• The math is pure and unit-tested (src/calibration.js, test/), so the interesting logic is verifiable without a database.

---

Quick start

You need Node 18+ and Docker (for a local Couchbase node).

# 1. install deps
npm install

# 2. start a local Couchbase node
docker compose up -d
#    First run only: open http://localhost:8091 and complete the setup wizard,
#    creating an Administrator user. Use the same credentials in your .env.

# 3. configure
cp .env.example .env        # on Windows PowerShell: copy .env.example .env
#    edit .env if your Couchbase user/password differ

# 4. provision bucket, scope, collection and indexes
npm run init-db

# 5. (optional) load sample predictions
npm run seed

# 6. run it
npm start

Open http://localhost:3000.

Windows note: everything above works in PowerShell. docker compose requires Docker Desktop running.

---

API

Base URL: http://localhost:3000/api

Method	Path	Body	Description
POST	/predictions	{ statement, probability, category?, deadline? }	Record a forecast. probability is strictly between 0 and 1.
GET	/predictions	—	List all. Query: ?status=open|resolved, ?category=work.
GET	/predictions/:id	—	Fetch one.
POST	/predictions/:id/resolve	{ outcome: true|false }	Mark what actually happened.
DELETE	/predictions/:id	—	Delete.
GET	/stats	—	Brier score, log loss, base rate, and the calibration curve.

Example:

curl -X POST http://localhost:3000/api/predictions \
  -H 'content-type: application/json' \
  -d '{"statement":"I finish the report by Friday","probability":0.7,"category":"work"}'

# ... later ...
curl -X POST http://localhost:3000/api/predictions/<id>/resolve \
  -H 'content-type: application/json' -d '{"outcome":true}'

curl http://localhost:3000/api/stats

---

How the scoring works

• Brier score = mean of (probability − outcome)², where outcome is 1 for true and 0 for false. 0 is perfect, 0.25 is what you'd get by always saying 50%, 1.0 is confidently wrong every time. Lower is better.
• Log loss punishes confident mistakes harder than Brier (probabilities are clamped away from 0/1 so a single cocky miss doesn't blow up to infinity).
• Calibration curve buckets your predictions by the probability you assigned, then compares the average probability in each bucket to how often those statements actually came true. A perfectly calibrated forecaster's dots sit on the dashed diagonal.

Why is probability forbidden from being exactly 0 or 1? Because proper scoring rules explode on certainty, and "I'm 100% sure" isn't a forecast — it's a claim.

---

Testing

npm test        # runs the pure calibration unit tests (no database needed)

---

Project layout

augur/
├─ docker-compose.yml        # single-node Couchbase for local dev
├─ scripts/
│  ├─ init-couchbase.js       # provisions bucket/scope/collection/indexes (idempotent)
│  └─ seed.js                 # loads sample predictions
├─ src/
│  ├─ config.js               # env-driven config
│  ├─ db.js                   # Couchbase connection singleton
│  ├─ repository.js           # data access: KV, mutateIn, N1QL
│  ├─ calibration.js          # pure scoring math (Brier, log loss, curve)
│  ├─ routes.js               # Express REST routes + validation
│  └─ server.js               # app entry
├─ public/                    # zero-build dashboard (HTML/CSS/JS, SVG chart)
└─ test/                      # unit tests for the scoring math

---

License

MIT — see LICENSE.