bench: the longitudinal harness measures retrieval on synthetic memories, not agent outcome on a real repository

[cdeust]

What exists

benchmarks/lib/longitudinal_runner.py drip-feeds N synthetic memories spread uniformly across the past 365 days, then probes old vs recent facts at wall-clock time. Its own docstring states the object of measurement: "how cleanly the decay-aware retrieval pipeline preserves recall as memories age" (lines 1-7). Ground truth is deterministic by construction — "the chosen number for memory <uid> is <hash(uid) % 10000>" (lines 8-12) — and the run uses a dedicated database that is dropped and recreated at start (lines 23-26).

That is a sound measurement of one component. It is not a measurement of the claim the product makes.

The gap

The claim is behavioural: a session with Cortex reaches a correct result faster, does not repeat an approach already known to fail, and does not re-derive a settled decision. Nothing in the suite measures that. Concretely, the existing harness:

• uses synthetic memories with synthetic probes, so a retrieval win cannot be shown to become a behavioural win;
• scores top-1 retrieval of a seeded fact — a proxy whose correlation with outcome is assumed, never measured;
• runs on a throwaway database, so it never meets the distribution of a real store (tool events, checkpoints, wiki pointers, superseded versions, conflicting facts).

What to build

A longitudinal outcome benchmark on a real repository.

1. Corpus. A repo with genuine Cortex history rather than a seeded one.
2. Tasks with checkable answers, drawn from that history and verifiable independently of Cortex: "which commit introduced behaviour X", "was approach Y tried, and what was the recorded reason it stopped", "what is the decided value of constant Z and its cited source".
3. Two arms: agent with recall vs agent without. Same model, same effort level, same task set, same ordering.
4. Score on outcome, not on retrieval: task correct/incorrect against the independent answer key, plus turns-to-answer and tokens spent.
5. Repeat, and report the delta with a variance estimate. A single paired run is an anecdote, not a result.

Constraints

• Clean database, single process, reproducible across runs — the rule already applied to the other benchmarks.
• No tuned constant in the scoring rubric. Thresholds come from the task definitions or they do not exist.
• If the delta is null or negative, that is the result, and it is published as-is.

Why it is worth the effort

Every positioning statement about this project rests on an outcome claim, and the suite currently measures a component proxy instead. Until this exists, the honest statement is "retrieval degrades gracefully with age", which is a narrower claim than the one being made.