Reduce generated dashboard history downloaded by plugin installations #1091

Problem

This repository is distributed as an agent plugin source, so many users will not clone it manually. Their plugin installer will fetch the repository automatically using whatever Git strategy or archive transport that installer implements. As a result, contributor documentation alone cannot ensure that installations avoid the repository's generated dashboard history.

The repository contains three append-only generated snapshot branches:

• dashboard-token-data
• dashboard-eval-data
• gh-pages

Current workflow consumers read only the current files/current branch tips, but the branches retain every historical snapshot. A normal Git clone can fetch objects reachable from all remote branches even though it checks out only main. An automated plugin installer may therefore transfer and retain generated history that is unrelated to the plugin content being installed.

The installer-specific impact still needs to be measured. Installers may use a full clone, single-branch clone, shallow/partial clone, sparse checkout, or GitHub archive download; these have materially different costs. We should not assume every installation produces the same local size.

Evidence

GitHub repository metadata reports approximately 139,194 KiB (~136 MiB). That reflects a well delta-compressed server-side pack and is not the same as the amount of uncompressed historical content reachable from repository refs.

git-sizer reports approximately 10.2 GiB of total uncompressed blob content across reachable history.

Measurements from a fresh bare clone:

These branch totals overlap because branches can reach shared objects. They must not be added together and presented as unique repository size.

One concrete contributor is data/token-usage.json: the analysis found 258 historical gh-pages versions, with individual historical blobs reaching roughly 20–40 MiB.

Fetching disconnected generated branches without --depth=1 can download and retain unnecessary ancestry. Fetch order, automatic maintenance/repacking, retained refs/reflogs, and duplicate or poorly packed local packs can make an ephemeral checkout substantially larger than GitHub's optimized pack. This explains reports of .git directories over 2 GiB, but does not imply every clone or plugin installation will reach that size.

Why plugin installation changes the priority

For manual contributors, we can recommend an efficient command. Plugin users generally do not control the installer's clone flags, and may not even see that a Git operation occurs. We therefore need a repository- or installer-level solution rather than relying exclusively on user guidance.

The preferred installation behavior should fetch only main and only the plugin content required for installation. Depending on installer capabilities, suitable approaches include:

• a single-branch partial clone (--filter=blob:none --single-branch)
• a shallow single-branch clone when history is unnecessary (--depth=1 --single-branch)
• sparse checkout after a partial clone when only one plugin directory is needed
• a GitHub source archive or packaged plugin artifact that does not include Git history

Archive/package installation is the strongest option when the installer does not need Git history at all.

Work completed so far

#1089 — shallow generated-branch operations in Actions

PR #1089 updates all seven relevant sites in .github/workflows/evaluation.yml:

• four fetches of dashboard-token-data and dashboard-eval-data, covering both data generation and deploy-dashboard
• three clones of dashboard-token-data, dashboard-eval-data, and gh-pages before publishing

Each operation now uses --depth=1. This is behavior-preserving because consumers only check out/show/copy tip snapshot files, and publishers create a normal fast-forward child commit from the shallow-cloned current tip.

The PR does not force-push, rewrite remote history, alter retention, or stop future branch growth. It safely reduces Actions runner transfer and local object retention. actionlint v1.7.7 passed.

#1090 — efficient clone guidance for contributors

PR #1090 documents:

git clone --filter=blob:none --single-branch https://github.com/dotnet/skills.git

for contributors who need main history, and:

git clone --depth=1 --single-branch https://github.com/dotnet/skills.git

for CI/read-only use that needs only the latest snapshot. It explains that the shallow option omits older main history.

This helps manual users, but it does not control automated agent plugin installers and is therefore only one part of the solution.

Proposed next steps

1. Measure actual plugin installer behavior

For each supported agent/plugin installation path:

• identify whether it uses Git clone, GitHub archives, or packaged artifacts
• record clone/fetch flags and refs requested
• measure network transfer, resulting checkout size, and .git size from a clean installation
• verify whether generated dashboard branches are downloaded
• determine whether the installer can use single-branch, shallow, partial, sparse, or archive-based installation without breaking updates

2. Optimize automated installation

Where this repository or its marketplace metadata controls installation behavior, prefer an archive/package containing only required plugin files. Otherwise, update the relevant installer to avoid fetching generated branches and history.

The result should not depend on users manually applying clone flags.

3. Perform a one-time generated-branch compaction

After #1089 is merged, consider a separately approved maintenance operation:

1. Pause dashboard publishing workflows.
2. Record all three generated-branch tips and create an external backup bundle.
3. Create a parentless commit for each generated branch whose tree exactly matches its current tip.
4. Verify old and new tip trees are identical.
5. Update each branch using --force-with-lease pinned to the recorded old SHA.
6. Validate dashboard data, Pages output, and the next normal fast-forward publishing run.
7. Resume workflows.
8. Allow GitHub's server-side unreachable-object cleanup to run; contact GitHub Support if old unreachable objects remain material to repository size.

Replacing each whole generated branch with an equivalent root snapshot is simpler than selectively rewriting it with git filter-repo. Existing local clones will not shrink automatically; a fresh clone is safest.

This operation requires explicit maintainer approval because it rewrites the three generated branch histories. It should not rewrite main, tags, or unrelated refs.

4. Stop renewed growth at the source

One-time compaction resets the generated branches, but append-only publishing will start growing them again. The durable architecture is to move generated dashboard data out of this source repository and deploy Pages without a history-bearing gh-pages branch, for example:

• store generated datasets in a dedicated data repository or artifact/object store
• deploy GitHub Pages through Actions upload/deploy artifacts
• remove the generated branches from this repository once migration and rollback validation are complete

Recurring force-push compaction is possible but is not preferred because it introduces workflow races, branch-protection exceptions, and recovery complexity.

Safety and limitations

• Do not add force pushes to the normal publishing workflow as an incidental optimization.
• Do not compact branches until Use shallow fetches for generated dashboard branches #1089 is merged and workflow behavior is verified.
• Keep a backup outside this repository; a backup tag in the repository would keep the old objects reachable.
• Use --force-with-lease, not an unguarded force push, for an approved one-time rewrite.
• Compaction may not immediately change GitHub's reported size because hidden refs, caches, and server garbage-collection timing can retain objects temporarily.
• Existing clones may retain old tips in refs or reflogs and will not automatically reclaim space.
• Measurements should distinguish raw reachable blob content, optimized pack size, network transfer, working-tree size, and .git size.

Completion criteria

• Use shallow fetches for generated dashboard branches #1089 is merged and generated-branch workflow operations remain functional.
• Document efficient repository cloning #1090 is merged so manual contributors have efficient clone guidance.
• Supported agent/plugin installers are inventoried and measured from clean installs.
• Automated installation avoids fetching generated dashboard history.
• Maintainers decide whether to perform the one-time generated-branch compaction.
• If approved, compaction is backed up, tree-equivalence checked, performed with leases, and validated end to end.
• Maintainers decide on a durable data/Pages migration that prevents renewed repository-history growth.