feat(storybook): add aria surface regression check - #5217
Conversation
Roles and accessible names are a public API — consumers query them with
getByRole/findByRole — but nothing in CI watched them for *change*:
- the public API check diffs rolled-up .d.ts files, and an aria-label
written inline in a component body is a value, not a type;
- axe asks whether a name exists, never whether it is the same name;
- Chromatic diffs pixels, not semantics.
Capture each story's role + accessible-name pairs in the existing
Storybook test-runner pass and diff them against the baseline published
on main. No second Storybook build: main already visits every story, so
the baseline costs one artifact merge.
Advisory for now — it posts a PR comment and never fails a build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🔍 Review policy: FeatureThe PR title starts with Required approvals
How this was decided
Policy source: |
✅ No New Circular DependenciesNo new circular dependencies detected. Current count: 0 |
🔍 Visual review for your branch is published 🔍Here are the links to: |
📦 Alpha Package Version PublishedUse Use |
✅ No breaking public API changesNo public exports were removed, renamed, or had existing props/types changed in a breaking way compared to Comparing
|
🔎 Accessible name & role changesℹ️ No baseline found — the Compares the role + accessible-name pairs every story renders against the baseline from the latest |
♿ Accessibility (axe) — components changed in this PR✅ No a11y issues in the stories this PR changed. Scope: only stories in the files/component folders this PR changed. It can't yet flag downstream ripple from shared-code/token changes, or diff against |
Coverage Report for packages/react
File Coverage
|
||||||||||||||||||||||||||||||||||||||
1 participant
Description
Roles and accessible names are a public API — downstream suites find elements with
getByRole("button", { name: "Clear" })andcy.findByRole(...)— but nothing in CI watched them for change. This adds an aria surface check that captures every story's role + accessible-name pairs in the existing Storybook test-runner pass and diffs them against a baseline published onmain, reporting the result as a PR comment.Type of change
Screenshots (if applicable)
No visual changes — this is CI tooling. What it posts on a PR that renames an accessible name:
Implementation details
Three existing gates all miss this class of regression, by construction:
check-api-surface.tsrunsts.createProgramover the rolled-up.d.tsfiles. Declarations hold types, not values, so anaria-labelwritten inline in a component body changes with a byte-identical API diff. There are dozens of these —F0InputField's"Clear",Arrows'"Increase"/"Decrease",pagination's"Go to next page",breadcrumb's landmark name."Clear"→"Clear input"keeps every rule green.Verified end to end: renaming
aria-label="Clear"inF0InputField.tsxleft all 23 of its stories passing axe, and produced exactly the comment above. Then reverted.feat: capture each story's role + accessible-name pairs in the test-runner's
postVisit, written toaria-snapshots.jsonlalongside the existing a11y artifactWhy scope to `body` rather than `#storybook-root`?
In the test-runner
pageis the preview iframe, sobodypicks up the portaled content axe currently cannot see — dropdowns, dialogs, tooltips, the Select listbox — which is the part consumers' Cypress suites struggle with most. Storybook's own wrappers are plaindivs and map togeneric, which Playwright omits, so none of the harness leaks in.feat: add
check-aria-surface.ts, which diffs base vs head per story, pairs same-role removals/additions into renames, and renders the PR commentfeat: publish the aria surface of
mainas a 90-dayaria-baselineartifact, and diff PRs against itWhy this needs no second Storybook build
The workflow already runs on
push: mainand visits all ~2.3k stories there, so the baseline costs one artifact merge. The PR side builds the merge commit (PR + current main), which lines up with a baseline taken from main. The alternative — building both sides per PR, asapi-surface.yamldoes — would double an 8-shard, ~30-minute job.test: 33 unit tests, including a fixture pinned to verbatim Playwright 1.57
ariaSnapshot()outputWhy a real fixture?
The parser is only as good as its grip on the snapshot format, and three things in the real output were not obvious: link targets are emitted as their own
- /url: "#"entries and must not parse as a role; a trailing: xis the element's text content, not part of the name; and names are always double-quoted with backslash escapes. The fixture was captured from an actual chromium run, not hand-written.chore: document the check in
packages/react/AGENTS.md, including its coverage limitNoise control
Counts rather than sets, so "three buttons named Remove became one" — a
getAllByRolebreak — is caught. Named nodes are always tracked; unnamed ones only for roles tests query bare, withlist/listitem/paragraphdeliberately excluded as the noisiest structural roles. Ambiguous multi-renames stay split across removed/added rather than guessed at. A story with nondeterministic content can opt out withparameters: { ariaSnapshot: { skip: true } }.Two degraded modes are handled explicitly rather than reported as PR breakage: a failed shard passes
--partialso unrun stories aren't blamed on the PR, and an empty head side reports an infrastructure problem instead of "you deleted every story".Known limits
breakingTotalis already returned for when it's flipped to blocking.F0InputField's clear button only mounts once the field has a value, so that rename surfaced in the one story with a filled input. A conditional element with no story covering its state is invisible here.mainruns once and publishes one. The comment says so explicitly.