A/B trial: Opus 5.5 at medium thinking vs Sonnet 5 for the project subagents

[gotgenes]

I'd like to find out whether Claude Opus 5.5 at thinking: medium can replace Sonnet 5 (currently at the default high) for the three project subagents: pre-completion-reviewer, tidy-first-assessor, and craftsmanship-scout. The comparison should cover both cost and review quality. The Opus 5.5 announcement claims it is unusually token-efficient, but I haven't found any local evidence either way: none of these agents has run on Opus 5.5 yet.

Break-even

Pi 0.87.1's registry (pi-ai/dist/providers/data/anthropic.json) prices the two models per MTok as follows:

Model	input	output	cacheRead	cacheWrite
claude-sonnet-5	$2	$10	$0.20	$2.50
claude-opus-5-5	$4	$20	$0.20	$5

Opus 5.5 costs twice as much for every token class except cache reads, which cost the same. That makes the break-even depend on each agent's token mix. I matched every subagent transcript under ~/.pi/agent/sessions/*pi-packages*/*/tasks/ to its parent subagent tool call by time window, took the agent type from the call's subagent_type, and recomputed each run's cost under both price tables:

Agent (Sonnet 5, high)	n	median turns	median $ (p90)	mean output tok	cacheRead share of cost	Opus 5.5 at identical tokens
pre-completion-reviewer	263	29	$0.81 ($2.23)	19,257	53%	1.47×
tidy-first-assessor	116	8	$0.36 ($0.65)	11,309	28%	1.72×
craftsmanship-scout	3	15	$0.59 ($0.72)	—	—	—

So Opus 5.5 breaks even only if it uses roughly a third fewer tokens as the reviewer and roughly 40% fewer as the assessor. The reviewer looks like the likelier win. Its median of 29 turns compares with 15 for Sonnet 4.6 on the same agent (n=136), and every turn re-reads the whole context.

Some prior data points:

• Sonnet 4.6 at medium cost a median $0.35 vs $0.50 at high for the reviewer (n=11 vs 136).
• The agent files don't set thinking:, so every run above used high.
• thinking: is a supported frontmatter field (packages/pi-subagents/src/config/custom-agents.ts).
• Neither model nor thinking is locked, so a per-dispatch override works without editing the agent files.

Why a paired trial

Cost varies too much between tasks for an unpaired before/after comparison to mean anything: the reviewer's p90 is 2.75× its median. So each dispatch point should run both arms against the same commit.

Proposed design

• Pairing. At each dispatch point, the dispatching agent issues two subagent calls in the same message with an identical prompt:
  • the control, as today
  • the Opus arm, with model: "anthropic/claude-opus-5-5" and thinking: "medium"
• Concurrency. Both arms run in parallel. The reviewer's deterministic checks (tsc --noEmit, biome check, eslint, rumdl check, vitest run) write nothing, but they do compete for CPU. When the arms disagree on a deterministic check, the parent re-runs that check once and records the losing result as a contention artifact rather than a model difference.
• Acting on the reports. The dispatching agent acts on the union of valid findings from both arms, and the overall verdict is the worse of the two. A real defect caught by only one arm is not thrown away.
• Quality log. For each pair, the dispatching agent records each arm's verdict and its unique findings, each marked valid or invalid. Token counts can't show whether a cheaper reviewer misses things.
• Cost measurement. A committed script derives turns, tokens, and cost per arm from the transcripts. Runs are paired by parent session plus overlapping dispatch windows, and each arm is identified by its transcript's model_change entry. No cost or token figure is written into the log by hand.
• Stopping rule. 10 pairs for the reviewer and for the assessor. At historical dispatch rates (263 reviewer runs in 83 days, 116 assessor runs in 72 days), that's about 3 days and about a week. The scout is paired whenever it happens to be dispatched.
• Extra spend (estimated). About $1.60 per reviewer pair and $0.70 per assessor pair, if Opus uses the same number of tokens as Sonnet.

Touchpoints

• .pi/skills/pre-completion/SKILL.md Step 2, .pi/skills/tidy-first/SKILL.md Step 2, and the craftsmanship-scout dispatch in .pi/prompts/plan-improvements.md: add the paired dispatch, marked as temporary.
• A trial log plus the measurement script (location to be settled in planning).
• After the trial: either switch each agent's frontmatter per the results, or remove the trial instructions. Either way the trial scaffolding comes out.

Acceptance criteria

• The three dispatch points issue a paired Opus 5.5 / medium arm while the trial is open.
• A committed script reports per-arm turns, output tokens, and cost for each pair from the transcripts.
• Each pair has a quality entry: verdicts, unique findings per arm, and valid/invalid calls.
• After 10 pairs per agent (reviewer and assessor), a written decision per agent, and the trial scaffolding is removed.