Skip to content

Latest commit

 

History

History
54 lines (39 loc) · 3.31 KB

README.md

File metadata and controls

54 lines (39 loc) · 3.31 KB

Benchmark: one skill, every target

Kitbash compiles one source skill to every agent's native format. This measures what that output actually costs, in tokens, per target — run on committed fixtures through the real kitbash compile pipeline.

Two costs matter:

  • Loaded — tokens the agent reads when the skill is in play.
  • Standing — tokens sitting in the context window every session, before the skill is even invoked. Lazy targets keep only a stub; eager targets carry the whole body.

One caveat on the eager rows: agentsmd is loaded by the agent automatically, but aider does not read CONVENTIONS.md on its own — it is loaded with aider --read CONVENTIONS.md or a read: entry in .aider.conf.yml. Its standing figure is what the file costs once wired in; unconfigured it costs nothing, and compile says which case a repo is in.

Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager ratio is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with node packages/cli/scripts/benchmark.mjs.

prereview — manifested (budget 1500, lazy)

Target Loaded (tok) Standing (tok) Loading
claude-code 567 40 lazy
cursor 568 40 lazy
agents 567 40 lazy
zed 567 40 lazy
copilot 567 40 lazy
cline 567 40 lazy
windsurf 569 40 lazy
gemini 567 40 lazy
aider 560 560 eager
agentsmd 560 560 eager

Standing tax: ~40 tokens on a lazy target vs ~560 on an eager one — about 14× per session for the identical skill. A team running four agents pays that gap four times over, forever, unless something measures it.

review-checklist — bare / unmanifested (no budget)

Target Loaded (tok) Standing (tok) Loading
claude-code 880 19 lazy
cursor 879 19 lazy
agents 880 19 lazy
zed 880 19 lazy
copilot 880 19 lazy
cline 880 19 lazy
windsurf 881 19 lazy
gemini 880 19 lazy
aider 885 885 eager
agentsmd 885 885 eager

Standing tax: ~19 tokens on a lazy target vs ~885 on an eager one — about 47× per session for the identical skill. A team running four agents pays that gap four times over, forever, unless something measures it.

Why this is the pitch

Kitbash compiles to the cheapest loading mode each target actually supports — Claude Code, Cursor, Devin (ex-Windsurf) and the vendor-neutral .agents/skills/ path all load on demand, so a skill there costs only its stub. The table above is what it costs on the targets whose only mode is eager: the whole body, every session, before the skill is invoked.

No other format has a field for this. Kitbash measures it at compile time, warns when a lazy-authored skill lands on an eager target, and fails the build under --strict. The copy-per-agent status quo pays the eager cost on every target and never sees the number.