Skip to content

TT-18683 skip BuildKit cache export on PR builds - #578

Merged
rafalgolarz merged 2 commits into
masterfrom
TT-18683_pr-no-buildx-cache-export
Oct 28, 2026
Merged

rafalgolarz merged 2 commits into
masterfrom
TT-18683_pr-no-buildx-cache-export

Conversation

@rafalgolarz

@rafalgolarz rafalgolarz commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem

In the goreleaser job, the "push image to CI" steps export the BuildKit cache
(cache-to: type=gha,mode=max) on every PR build. On tyk-analytics this has two costs:

  • Time: in a run inspected in detail the cache export took 52 s of 72 s (fips) and 55 s of
    75 s (std). Together the two push steps took a median of 159 s per PR job (156-199 s).
  • Cache space: every PR build leaves buildkit-blob-* entries (~245 MB per image) on its PR
    ref, and they pile up with each push. Across open PR refs this is ~7 GB of tyk-analytics's
    10 GB Actions cache budget (tyk: ~10 GB, one PR ref holds 13 GB; tyk-pump: ~6 GB), which gets
    useful entries evicted (e.g. the Go test cache on master).

The export brings almost nothing back: ci/Dockerfile.distroless starts with a
COPY <package>.deb that changes on every build, which invalidates every layer after it.
In 3 sampled PR runs the image builds reported CACHED=0 for the first image and CACHED=1 for
the second.

Change

New feature flag pr-no-cache-export. With the flag, in the "push image to CI" step:

cache-to: ${{ github.event_name != 'pull_request' && 'type=gha,mode=max' || '' }}

so PR builds do not export the cache; master and tag builds export as before. Without the
flag the rendered output is unchanged. cache-from is untouched.
Enabled for tyk-analytics master only; the golden diff is one file
(tyk-analytics/master/.github/workflows/release.yml, two lines).

Evidence (tyk-analytics PR #6388, job 1.26-bullseye on the 8x runner)

Baseline = last 4 PR runs from the Actions API; experiment = 5 runs with the same change
applied by hand to release.yml.

baseline experiment
push fips + std image to CI, median 159 s (156-199) 61 s (60-64)
job total, median 759 s 564 s (548-660)

Only about 98 s of the ~195 s total drop is attributable to this change; the rest is
run-to-run variance in other steps (restore ranged 277-368 s within the series).
Mechanism, from the logs of one run: no "preparing build cache for export" / "exporting
cache" lines, and no buildkit-blob-* entries on the PR ref.

What is not proven

  • API/UI tests on the images built this way passed in one completed run; in the other runs
    they were cancelled by the next push. They should be re-checked over several runs after
    this lands.
  • The CACHED counts after the change (2 and 3) were taken with a different counting method
    than the baseline (0 and 1), so they are only indicative.
  • Only tyk-analytics was measured. Other repos have different Dockerfiles (tyk builds
    std/fips/ee and uses sbom: true), and their image builds may benefit from the cache. This
    is why the flag is off elsewhere.

Rollout and rollback

Merging gromit opens a sync PR in tyk-analytics for master only; nothing changes anywhere
else. Roll back by removing the flag from config/config.yaml.
Enabling the flag for another repo should follow the same measurement (5 runs, push steps and
tests). Wall-clock time of the whole PR is not expected to change: it is set by the MCP
qualification jobs. The gain is ~1.6 min per PR job and much less pressure on the cache limit.

Refs: TT-18683

Ticket Details

TT-18683
Status In Code Review
Summary Speed up Go builds and stop GitHub Actions cache exhaustion in CI (tyk-analytics, gromit, github-actions)

Generated at: 2026-10-28 20:42:01

@rafalgolarz
rafalgolarz merged commit cdbaaa8 into master Oct 28, 2026
16 checks passed
@rafalgolarz
rafalgolarz deleted the TT-18683_pr-no-buildx-cache-export branch October 28, 2026 21:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants