feat(kb): record the roofline on every recipe - #1217
Conversation
A recipe only kept what beat the incumbent, so a session that improved nothing left behind session rows and no measurement. The distance to the roofline is worth keeping either way: it says whether the run was near its ceiling or nowhere close, and which side bound it. Project the latest roofline snapshot onto the recipe through extras, so the write sits outside the has_validated_win gate that guards best_config. Extras merge with the prior row, so a later session without a roofline preserves the earlier one rather than erasing it. Local mode only. The remote publisher still filters roofline actions out of its new-KEEP test, and nothing reads the stored value back yet. Co-authored-by: Cursor <cursoragent@cursor.com>
CI E2E report — ✅ Succeeded
|
|
Reviewed. No blockers — tests pass (7 new + 529 in the related subset), ruff is unchanged vs the merge base, and the extras routing does exactly what the description claims. A few things worth fixing: 1. Every other key comes from 2. build_recipe_roofline([{"ts": "t", "perfmodel_breakdown": {"ops": [{} for _ in range(69)]}}])
# -> {'ts': 't', 'perfmodel_breakdown': {'ops_truncated_from': 69}, 'snapshot_count': 1}If every op projects to empty, 3. No pointer to where the field came from (low)
4. Optional: guard the allowlists against drift The three key tuples restate field names from Things the PR got right that I'd rather not see regress: |
snapshot_count was read off the snapshot list while every other field came from snapshots[-1], and each session overwrote it, so a recipe carried how many snapshots the last writing session happened to take — which reads as if the roofline aggregated them. snapshot_id already names the snapshot that was projected, so drop the count rather than rename it. ops_truncated_from was written whenever the op list exceeded the cap, including when every op projected to nothing and no ops array was stored, leaving a marker that claimed rows the recipe never held. Write it only alongside a non-empty ops, which lets an all-empty breakdown fall out entirely. Name the writer and the local-mode, write-only status in the docstring so the block found in a recipe JSON points back at what produced it, and pin the projection allowlists against the fields build_roofline_snapshot and the PerfModel breakdown dataclasses emit, so a new snapshot field fails a test instead of being silently dropped. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Thanks for the PR — the implementation looks fine. I don't think this requirement is worth merging as-is: roofline is already captured and used within each session, nothing reads recipe["roofline"] back, and the write path is local-only while production uses remote KB. Persisting this adds storage without changing runtime behavior. I'd recommend closing this PR. If there's a concrete read-side use case later, we can revisit with a full design. |
A recipe only kept what beat the incumbent, so a session that improved nothing left behind session rows and no measurement. The distance to the roofline is worth keeping either way: it says whether the run was near its ceiling or nowhere close, and which side bound it.
Project the latest roofline snapshot onto the recipe through extras, so the write sits outside the has_validated_win gate that guards best_config. Extras merge with the prior row, so a later session without a roofline preserves the earlier one rather than erasing it.
Local mode only. The remote publisher still filters roofline actions out of its new-KEEP test, and nothing reads the stored value back yet.