[rafalgolarz]

TT-18683: Prune the Go build cache before saving it (tyk-analytics master only)

Changes

Adds an opt-in feature go-cache-prune to the generated release.yml and enables it only for tyk-analytics master (config/config.yaml). Every other repo and branch renders exactly the same release.yml as before; the only changed golden snapshot is tyk-analytics/master/.github/workflows/release.yml (3 files in the PR).

With the feature on, a new step runs before "Save Go build cache": it deletes files in ~/.cache/go-build that were untouched for 2+ days (Go bumps the mtime of entries it uses) and logs du -sh before and after. It runs only off-PR and only when a save will actually happen (cache-hit != 'true'). The build container writes as root, so it uses sudo when the runner user is not root. The module cache (~/go/pkg/mod) is not pruned: Go does not bump its mtimes, so age says nothing about use.

Why: a partial restore (restore-keys) is saved under a new key on every go.sum change, so the cache only grows (60.9 GB / 153k files in the master entry).

Measured (tyk-analytics, job 1.26-bullseye, 8x runner)

The pruned cache was seeded once from the master cache; then the same job was re-run on the same commit, alternating between the unpruned master cache (control) and the pruned one. n = 4 per arm, no invalid runs.

scenario	restore (median)	Build (median)	job total (median)
code change in config (real rebuild), run 40118194165	312.5 -> 183 s	136.5 -> 137 s	710.5 -> 568.5 s
unchanged code, run 40086564857	-104.5 s	-2.5 s	-127 s
dependency change (golang.org/x/sys v0.48.0 -> v0.49.0), run 40138314180	277 -> 170 s (one pruned point at 297 s)	191 -> 183.5 s (169-215 s in both arms)	714 -> 602.5 s

The pruned cache is 31.2 GB / 66k files. Build did not get slower in either rebuild scenario.

What this does not show

• The real master path was not exercised: the experiments ran the prune and the save in a PR build; the template step only runs off-PR and when cache-hit != 'true'.
• Only tyk-analytics was measured; other repos are unchanged by this PR.
• It saves runner time, not PR wall-clock: in the last 14 PR runs the "Coordinated MCP qualification" jobs (median 36 min) were always the last job to finish, while the goreleaser job takes ~12 min. Expected gain: ~2 min of runner time per release run.
• Nothing changes until a cache save happens (after the next go.sum change on master, e.g. a bot go.mod update). Until then the existing large entry stays (inferred from the step condition).
• Go itself trims build-cache entries unused for 5 days (inferred; the 60.9 GB restore shrank to 31.6 GB during Build in one run), but the trimmed cache is never saved back unless a save is triggered. This PR makes the saved entry small.

Risks

• The step deletes cache files off-PR on master only; if it turns out to hurt, remove the flag. sudo worked on the 8x runner in the experiments.
• A pruned cache could miss entries needed after a very large dependency change; two scenarios were tested (code change, one dependency bump).

After merging: how to decide whether to keep going

1. Merge this PR; gromit opens a sync PR in tyk-analytics (master). Nothing changes until that sync PR is merged.
2. Wait for the first master push run of release.yml where the save happens: the step "Save Go build cache" runs (it is skipped on an exact cache hit). Master updates from bot go.mod commits usually trigger it.
3. In that run check the job 1.26-bullseye:
   • "Prune Go build cache" ran (not skipped); the log shows before:/after: sizes (expect the cache to drop from ~60 GB to ~30 GB);
   • "Save Go build cache" succeeded (log line Cache saved with key: ...); note its duration.
4. Then compare 5+ PR runs after that save with the PR runs before it (same job, gh api repos/TykTechnologies/tyk-analytics/actions/runs/<id>/jobs step durations):
   • restore step should fall by ~100-130 s (target restore ~170-185 s against ~280-310 s);
   • Build should stay within the previous spread (experiments: 132-147 s with a code change, 169-215 s with a dependency change). Revert if the median Build is more than 30 s slower than before.
5. Decision:
   • good: extend the flag to other repos with the same job (add go-cache-prune to their branches in config/config.yaml, one repo at a time);
   • bad or no effect: remove the flag line from config/config.yaml, merge the resulting sync PR in tyk-analytics. That is the whole rollback.

Related

• TT-18683 skip BuildKit cache export on PR builds #578: skip the BuildKit cache export (cache-to) on PR builds, tyk-analytics only (image push steps ~98 s shorter in an experiment, n = 5). Both PRs change the tyk-analytics master snapshot: after one is merged, rebase the other and run make update-golden.

Refs: TT-18683

Ticket Details

TT-18683

Status	In Code Review
Summary	Speed up Go builds and stop GitHub Actions cache exhaustion in CI (tyk-analytics, gromit, github-actions)

Generated at: 2026-10-28 22:00:14