Progressive Skill Disclosure Evaluation

Does progressive disclosure actually pay off for agents with large skill catalogs? Read the findings →

TL;DR — Yes. Tokens are the reliable win: total tokens drop ~16% at 10 skills to ~69% at 90. Latency improves ~10% (median 276 ms saved per selector call, 95% CI [155, 417]) — real, but far smaller than a naive mean-based measurement suggests. The bigger finding: index the full skill, disclose only its summary. Doing so beats disclosing the entire catalog on accuracy (98.9% vs 90.0%, p = 0.02) at a third of the tokens.

This benchmark compares three ways to expose 100 synthetic skills to a Microsoft Agent Framework agent:

• Option A: send all 100 skill names and descriptions to the selector model, then load the selected skill's full content.
• Option B: run local BM25 over skill names and descriptions, send only the top three candidates, then load the selected skill's full content.
• Option C: run local BM25 over the entire skill definition (body, instructions, examples, schemas), send only the top three candidates' names and descriptions, then load the selected skill's full content.

Options B and C disclose an identical prompt surface and therefore cost the same. They differ only in what the retriever indexes, so any gap between them is pure retrieval quality.

Each synthetic skill includes a body, detailed instructions, examples, and input/output schemas. Those full contents are hidden during selection and loaded only after selection.

Query regimes

Skill descriptions are templated from skill names, so a query phrased like the description makes BM25 impossible to beat — which is why accuracy sat at 100% in earlier versions of this eval and told us nothing. Two regimes bracket the range:

• --query-style literal — query words appear verbatim in the description. Best case for metadata-only retrieval.
• --query-style paraphrase — query words appear only in the body and examples, never in the name or description. Near-worst case.

Real traffic sits between them. Under paraphrase, Option B's recall@3 collapses to 11% while Option C holds 100%.

Metrics

• Token consumption: model/runtime usage and a tiktoken estimate are recorded side by side for the selection and execution turns.
• Accuracy: retrieval recall@3 (the expected skill is in the candidate set) and exact model selection accuracy.
• Latency: end-to-end, selector, and execution latency, with warm-up requests excluded.

Option A's retrieval recall is expected to be 100% because its candidate set contains all skills. Its model-selection accuracy remains a meaningful comparison.

Setup

cd skill-disclosure-eval
python -m pip install -e ".[dev]"
Copy-Item .env.example .env
az login

Set FOUNDRY_PROJECT_ENDPOINT and FOUNDRY_MODEL in .env. The endpoint uses the same Microsoft Agent Framework FoundryChatClient configuration as the existing Forgebook samples.

Run

Generate and inspect fixtures without credentials:

skill-disclosure-eval --fixtures-only

Run a quick live evaluation:

skill-disclosure-eval --query-limit 10 --repeats 1 --warmups 1 --query-style paraphrase

Run the default evaluation:

skill-disclosure-eval

Outputs are written to results\:

• fixtures\skills.json, fixtures\queries-literal.json, fixtures\queries-paraphrase.json
• raw-results.json and raw-results.csv
• summary.json and summary.md

The complete default run makes two model calls per option, query, and repeat. With 200 queries, three options, and three repeats, that is 3,600 model calls plus warm-ups.

Scale sweep

To find where the benefit starts, sweep across catalog sizes:

python -m skill_eval.scale --queries-per-size 10 --repeats 1 --warmups 1 `
    --query-style literal --output results/scale-v4-literal

python -m skill_eval.scale --queries-per-size 10 --repeats 1 --warmups 1 `
    --query-style paraphrase --output results/scale-v4-paraphrase

Keep --warmups at 1 or higher. Each size builds a fresh client, so with no warm-up the first call pays TLS, credential, and connection-pool setup — a 6–17 s outlier that lands on whichever option runs first and badly skews mean latency.

Results in this repo

• results\scale-v4-literal\ and results\scale-v4-paraphrase\ — the current three-option sweeps across 10–90 skills, plus charts. Use these numbers.
• results\scale-v3\ — superseded. Methodologically sound but Options A/B only and literal queries only, so its accuracy column is uninformative.
• results\scale-v2\ — superseded. Fixed the original five defects but suffered cold-start contamination that inflated its latency figures 3–5×.
• results\scale\ — the original sweep, retained for comparison. Five methodology defects.
• results\ — a single-shot 100-skill run.

All defects are documented in FINDINGS.md. Earlier runs are kept deliberately so the corrections stay auditable — cite only scale-v4.