Skip to content

perf(ci): make docs.yml skip runs it doesn't need and cost less when it does - #367

Merged
berntpopp merged 10 commits into
mainfrom
perf/docs-workflow-screenshot-cache
Aug 7, 2026
Merged

berntpopp merged 10 commits into
mainfrom
perf/docs-workflow-screenshot-cache

Conversation

@berntpopp

Copy link
Copy Markdown
Owner

Implements .planning/specs/2026-08-06-docs-workflow-performance.md.

docs.yml ran on every push to main at a measured 276 s and had never been optimised.

What changed

  • paths filter — 64 of the last 200 first-parent pushes to main changed nothing the published site depends on. Workflow-level rather than build.yml's job-level dorny/paths-filter: that file's pending-required-check hazard doesn't apply here (no pull_request trigger, not a required check), and a workflow-level skip costs 0 s where a job-level one still spins a runner.
  • Screenshot manifest — fixes a live bug. test('02 - import menu') writes its PNG inside a conditional; if that selector stopped matching, the test passed without writing the file and Upload screenshots published the stale committed copy while CI stayed green. The suite now declares all 23 screenshots and fails if any wasn't written. Because Upload screenshots has no if: always(), that failure now blocks publication entirely.
  • Playwright browser install removed — the suite drives Electron via _electron.launch and never opens a Playwright browser.
  • apt gated on a native-cache miss — libsqlite3-dev/build-essential exist only to compile the native SQLite module.
  • 5 redundant sleeps deleted (65,600 → 60,800 ms), each adjacent to a real Playwright wait already guaranteeing the same condition.
  • timeout-minutes on both jobs (spec item 4.5 of the build-CI-performance spec, applied to the workflow being touched).

Measured

Run 31120283316 vs baseline 31109514729. Full data: .planning/artifacts/perf/build/docs-yml-before-after.md.

Step Before After
Install system dependencies 9 s 0 s (skipped)
Restore Playwright browsers 8 s removed
Install Playwright 29 s removed
Generate screenshots 131 s 120 s
Build-job step total 234 s 172 s (−26.5%)

Artifact inspected, not just exit-code checked: 34 PNGs, none blank, footer reads the current v0.70.5, highlight boxes correctly aligned.

What is NOT proven

Stated plainly because the numbers above are easy to over-read:

  • N=1. Of Generate screenshots' −11 s, only ~4.8 s is the deterministic sleep removal; the rest is variance.
  • The after-run's job total of 418 s is unusable — GitHub Actions was in a major outage and the job suffered runner-acquisition stalls. Only step timings are comparable.
  • ~214 s wall clock and 47.3% over 200 pushes are estimates, not measurements.
  • Task 5's cold native-cache path is unverified — apt was skipped in the measured run, so the path where it is actually needed has never executed.
  • The Pages deploy is unverified on this branch — the deploy job is stuck waiting; GitHub Pages has been in major_outage.
  • The paths filter's skip behaviour has never been observed — workflow_dispatch bypasses path filters, so 64/200 is derived from git history.

Design decision recorded

An earlier revision added a content-keyed screenshot cache. It was cut after adversarial review (codex gpt-5.6-terra, xhigh): measured over 200 pushes it would hit 7 times — ~2.2 percentage points — while carrying the entire staleness surface, and the review found two CRITICAL holes in it. Because the app version must stay current and AppFooter.vue:9 renders it in every screenshot, every release bump legitimately changes all 23 images, which is what destroys the cache's value. Full record in the spec's "Adversarial review record". It is deferred behind the manifest validation this PR adds, which is its prerequisite.

Overlap with tracked work

Two items of .planning/specs/2026-08-05-build-ci-performance.md: item 4.5 (timeout-minutes) and item 4.10 (Playwright browser caching, which this PR removes from docs.yml).

🤖 Generated with Claude Code

https://claude.ai/code/session_01PohiEWFT9CkCK4XNuhm9fR

berntpopp and others added 10 commits August 6, 2026 16:59
Measured baseline 276 s per push, on every push to main. Spec is revision 3:
adversarial review (codex gpt-5.6-terra, xhigh) cut the content-keyed
screenshot cache -- 7 hits per 200 pushes, ~2.2 percentage points, carrying
the whole staleness surface -- and surfaced a live bug where a silently
skipped test republishes a five-month-old screenshot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PohiEWFT9CkCK4XNuhm9fR
… non-destructive

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PohiEWFT9CkCK4XNuhm9fR
test 02 writes import-menu.png inside a conditional; if its selector stops
matching, the test passes without writing the file and CI publishes the
stale committed copy -- currently a March 2026 image showing v0.30.0.
Adds an explicit manifest of the 23 screenshots and asserts it.
Measured over 200 first-parent pushes to main: 64 changed nothing the
published site depends on, yet each cost a full 276 s rebuild and
redeploy. Also adds timeout-minutes to both jobs (spec item 4.5).
Each removed waitForTimeout sat immediately beside a real Playwright wait
that already guaranteed the same condition. The 33 'replaceable' sleeps
are deliberately left alone: the suite is one unbroken causal chain and
rewriting them risks flake in the only pipeline that publishes the docs.
The screenshot suite drives Electron directly via _electron.launch and
never opens a Playwright browser, so the browser download and its cache
are dead weight. Whether Playwright's --with-deps system libraries are
still needed for Electron under xvfb is being verified in CI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PohiEWFT9CkCK4XNuhm9fR
libsqlite3-dev and build-essential exist to compile the native SQLite
module; the ABI-keyed cache means that rarely happens. Ordering is
load-bearing: the step must sit after the cache restore so it can read
cache-hit, and before npm ci, whose postinstall may compile.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PohiEWFT9CkCK4XNuhm9fR
…anch

- paths filter: add resources/** (gene_reference.db reaches the renderer
  via PanelFilterSection inside FilterDrawer, which is on-screen in two
  captured screenshots; harmless today, closed permanently)
- apt gate: correct the comment's premise -- rebuild-native.mjs can purge a
  bad restore and compile while cache-hit is still 'true'
- screenshot manifest: replace a stale line number with a name anchor
- spec: record that build.yml already proves Electron runs under xvfb with
  no playwright install, so Phase 2a is a confirmation not an experiment

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PohiEWFT9CkCK4XNuhm9fR
Build-job step time 234 s -> 172 s (-26.5%) on run 31120283316. Playwright
browser install and apt both confirmed removable; screenshots generate with
no Playwright browser present. Cold native-cache path and the Pages deploy
remain unverified -- GitHub Actions and Pages were both in major_outage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PohiEWFT9CkCK4XNuhm9fR
@berntpopp
berntpopp merged commit 330d736 into main Aug 7, 2026
20 of 26 checks passed
@berntpopp
berntpopp deleted the perf/docs-workflow-screenshot-cache branch August 7, 2026 06:21

This branch had an error being deployed

1 failed deployment
github-pages — fa9704d8 Deployed Aug 7, 2026 by berntpopp via Deploy to GitHub Pages #299
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant