Move heavy eval fixtures off the skill-consumption critical path #1121

Summary

Consumers install skills by registering this repository as a plugin marketplace
(/plugin marketplace add dotnet/skills, codex plugin marketplace add dotnet/skills).
That operation clones the whole repository. But the shipped product is only
plugins/**. The eval fixtures and references under tests/** are the majority
of the repository, and every consumer downloads them even though they are never
used at skill-consumption time.

This issue proposes to move the heavy eval fixtures off that critical path.

Measurements

Taken on branch abhitejjohn-improve-eval-quality (draft PR #1118), tracked files only.

• tests/** is about 7× the file count and 4× the bytes of the actual
  product (plugins/**).
• Compressed snapshot proxy (git archive --format=tar.gz HEAD): full repo
  3.24 MB, tests/** alone 1.13 MB — the fixtures are about 35% of a
  snapshot clone.

Largest fixture set — tests/dotnet-upgrade: 999 files / 7.62 MB, mostly
954 .cs project fixtures. This one directory is ~80% of all tests/** bytes.

Per-plugin fixture bytes (top):

New artifacts in PR #1118 / branch abhitejjohn-improve-eval-quality
(merge-base → branch tip): 326 files changed, +12,194 / −582.

• 305 new files added, totaling 0.27 MB.
• 290 of the 305 land in tests/dotnet-msbuild (eval fixtures).

The branch is file-count heavy but byte-light: it does not add much size, but
it adds many small fixture files. Each new file adds a Git object and a working-tree
entry that every marketplace consumer clones. This is the pattern the repository
repeats each time an eval is strengthened, so the file count grows steadily on the
critical path.

Why these files are on the critical path

• Skill consumers do not build or test the repo. They only load plugins/**.
• The install path (plugin marketplace / skill-installer) fetches the repository,
  so tests/** comes with it.
• Harnesses that clone the repo only to read skills pay the same cost: more objects
  to transfer, more inodes to write, slower shallow clones and container image layers.
• Fixture growth is unbounded. Every strengthened eval (like PR test(msbuild): strengthen eval quality #1118) adds more
  files that consumers must download but never use.

Proposed options

Option A — Separate fixtures repository (for example dotnet/skills-evals)

Move tests/** fixtures and references to a second repository. This repo keeps
plugins/**, eng/**, and thin eval specs (eval.yaml). CI in the fixtures repo,
or a submodule/checkout step, provides fixtures when evals run.

• Pros: removes ~61% of bytes and ~81% of files from the consumer clone;
  clean product boundary.
• Cons: loses atomic "skill + fixture" changes in one PR; two-repo coordination;
  cross-repo versioning needed to keep evals reproducible.

Option B — Selectively fetched artifact package

Package fixtures as a versioned artifact (NuGet, GitHub Release asset, or a git
archive pinned by tag/SHA). Eval runs download and unpack the pinned version.

• Pros: consumers never fetch fixtures; fixtures are versioned and immutable;
  works well for CI caching.
• Cons: publish step to maintain; offline/air-gapped evals need a local cache
  or vendored copy; a new supply-chain surface to secure (checksums, provenance).

Option C — Sparse-checkout / partial clone guidance

Keep one repo, but document and default consumers to git clone --filter=blob:none
or sparse-checkout that excludes tests/**. Also confirm the marketplace/installer
uses a shallow or sparse fetch.

• Pros: no repository split; atomic changes preserved.
• Cons: does not shrink history; depends on client support for filtered clone;
  the marketplace fetch behavior may not honor it.

Option D — Git LFS for large fixture blobs

Track large fixture files (for example the dotnet-upgrade .cs set) with Git LFS.

• Pros: working tree stays small unless LFS objects are pulled.
• Cons: LFS adds bandwidth quota and tooling requirements; poor fit for many
  tiny files (the smart-pointer overhead can exceed the savings); still on the path
  unless clients skip LFS.

Tradeoff summary

Recommendation

Adopt Option B (selectively fetched, version-pinned fixture artifact) as the
primary fix, because it fully removes fixtures from the consumer clone while keeping
evals reproducible through an immutable pinned version. Start by relocating the
largest offender, tests/dotnet-upgrade (999 files / 7.62 MB), which alone removes
~80% of tests/** bytes for a small, well-scoped first step.

As a low-risk immediate mitigation, also apply Option C: document and default
consumers to a partial/sparse clone that excludes tests/**, and confirm the
marketplace install path fetches shallow or filtered.

Keep thin eval specs (eval.yaml) in this repository so evals remain discoverable
and CI can still map a skill to its eval; only the heavy fixtures move.

Filed from an analysis session. Measurements are reproducible with
git ls-files tests, git archive --format=tar.gz HEAD [tests], and
git diff --diff-filter=A <merge-base> abhitejjohn-improve-eval-quality.