Accessibility testing workflow with Chromatic, Storybook/Vitest, and the Datadog inventory.
Automated accessibility (a11y) testing for the design system, starting with @prepared911/ui. Everything runs on the same axe-coreopens in a new tab engine via @storybook/addon-a11y — there is no second test stack to maintain.
Quick Reference
The pipeline has three layers, each with a distinct role:
Layer
Tool
Purpose
Story-level testing
@storybook/addon-a11y + Vitest
Run axe per story in a real browser; surface violations in dev, gate CI on opted-in stories
Observability
a11y:inventory:ui-core + Datadog
Batch-audit all ui-core stories; ship violation counts as Datadog metrics for trend tracking
Visual regression
Chromatic
Snapshot stories, block PRs on unreviewed visual changes, and report a11y for opted-in stories
Which do I care about?
Writing a component → check the A11y panel in Storybook locally.
Opening a PR → Chromatic is the visual gate for stories opted in with WithSnapshot (unreviewed visual diffs fail; axe debt does not).
Tracking debt over time → the Datadog dashboard below.
Environment Flags
Three flags control a11y behavior. All are inlined into the Vite bundle at build time (via define in apps/storybook/.storybook/main.ts and vitest.config.ts), so they must be set before Vitest/Storybook starts. They resolve through resolveA11yTestMode() in @prepared911/util-storybook-helpers.
Flag
Purpose
Used in
STORYBOOK_A11Y_TODO=1
Downgrades every mode to "todo" — violations show as warnings but never fail tests
test:storybook:ci (the standard CI job), so existing debt never blocks PRs
STORYBOOK_A11Y_AUDIT=1
Forces the global default from "off" to "error" — every loaded story runs axe and fails on any violation
a11y:inventory:ui-core audit script
STORYBOOK_TEST_SCOPE=ui-core
Narrows the loaded story set to packages/ui-core/src/ only, keeping audit runs fast
a11y:inventory:ui-core audit script
Mode precedence (highest → lowest):
STORYBOOK_A11Y_TODO=1 — forces everything to "todo", so nothing errors or blocks.
Story-level withA11yTestingError — enables "error" for the wrapped story/file.
STORYBOOK_A11Y_AUDIT=1 — forces the global default to "error" (used with STORYBOOK_TEST_SCOPE to constrain scope).
Global default "off" (set in preview.ts).
Workflows
Local development
A11y tests are off by default globally. Opt in per story or per file with the helpers below, then run Storybook normally and check the A11y panel in the sidebar:
CI — Storybook Vitest
Runs in .github/workflows/ci.yaml (storybook-test aggregate over four shards). Executes all stories in browser mode (Chromium via Playwright) with STORYBOOK_A11Y_TODO=1, so any story opted in with withA11yTesting* — even in "error" mode — never blocks a merge:
Full audit — ui-core inventory
Runs axe across all ui-core stories in "error" mode and produces a human-readable report plus a Datadog metrics CSV:
Internally this:
Runs Vitest with STORYBOOK_A11Y_AUDIT=1 STORYBOOK_TEST_SCOPE=ui-core --reporter=json --outputFile=a11y-inventory/vitest-results.json.
Writes raw results to a11y-inventory/vitest-results.json.
Vitest 5's json reporter writes to a file (.vitest/json/output.json by default) instead of stdout, so piping its output no longer works — always pass --outputFile, or set reporters: [["json", { stdout: true }]] to print it.
3. Calls scripts/format-a11y-inventory.mjs → produces report.md + summary.json.
Open a11y-inventory/report.md to review findings grouped by component, rule, and impact. To format the results for Datadog locally:
CI — merge_group inventory
Runs automatically via .github/workflows/storybook-a11y-inventory.yaml:
Trigger:merge_group (only when turbo/packages/ui-core/src/** changed), or manual workflow_dispatch.
Non-blocking:continue-on-error: true and excluded from merge-gatekeeper's required checks — a failure here never blocks a merge.
What it does:a11y:inventory:ui-core → a11y:format-datadog-metrics → uploads datadog-metrics.csv to Datadog via int128/send-datadog-action.
Chromatic
Runs on relevant non-draft PRs via .github/workflows/chromatic-storybook-ci.yaml. Opt stories in with WithSnapshot. Local publishes: pnpm chromatic.
Visual regression: captures opted-in story snapshots and blocks the PR until a reviewer accepts visual diffs. TurboSnap (onlyChanged: true) only re-snapshots affected stories; a full run is forced when design-token SCSS (packages/ui-core/src/styles/) or apps/storybook/.storybook/globals.scss changes. Snapshots auto-accept on trunk.
Accessibility tests: the build's Accessibility tab reports axe violations for snapshotted stories. The required GitHub status check is the Actions aggregate named Chromatic, not Chromatic's external UI Review status.
Chromatic is complementary to the axe audit: it catches visual regressions (layout, color, contrast) that axe misses, while axe catches semantic/structural issues (missing labels, ARIA roles, keyboard navigation) that visual diffing misses.
Datadog dashboard
Prepared / ui-core a11yopens in a new tab tracks violation trends from the merge_group inventory: total violations over time, severity mix, top components, top axe rules, and a per-PR/SHA table.
A few things to know when reading it:
Metrics are gauges — each run overwrites, so widgets use max/last per bucket, never sum.
Data is sparse by design: it only lands on merge_group runs that touch turbo/packages/ui-core/src/**.
These ui-core-only counts intentionally differ from Chromatic's totals — Chromatic spans all snapshotted ui-* packages across light and dark modes.
Metrics
Namespace: prepared.ui_core.a11y (all gauges).
Metric
Extra tags
Description
violations.total
—
Total axe violation count across ui-core stories
failed_stories
—
Stories with at least one violation
passed_stories
—
Stories that passed axe cleanly
unattributed_failures
—
Stories that failed for non-a11y reasons (axe never ran)
errored_stories
—
Stories where the axe runner itself errored
violations_by_rule
rule:<axe-rule-id>
Count per axe rule (e.g. color-contrast, button-name)
violations_by_component
component:<folder>, component_path:<path>
Count per ui-core component folder
violations_by_impact
impact:critical|serious|moderate|minor
Count per axe impact level
Every row is also tagged with environment:github-actions, service:storybook-a11y, package:ui-core, repository:prepared911/prepared911, plus pr:, sha:, and ref: per run.
Note: ref currently ships empty; filter and group by pr/sha instead.
Output files
All audit output is written to turbo/apps/storybook/a11y-inventory/ (gitignored):
File
Produced by
Contents
vitest-results.json
a11y:inventory:ui-core
Raw Vitest JSON output
summary.json
format-a11y-inventory.mjs
Parsed counts: total, by rule, by component, by impact
report.md
format-a11y-inventory.mjs
Human-readable tables — open this to review findings
datadog-metrics.csv
format-datadog-metrics.mjs
CSV for int128/send-datadog-action
Opt-in helpers
Located in turbo/packages/util-storybook-helpers/src/with-a11y-testing.ts, imported from @prepared911/util-storybook-helpers:
Mode summary:
Mode
Storybook UI
Vitest CLI
CI
"off"
No a11y results
Passes
No effect
"todo"
Violations shown as warnings
Passes
Never blocks
"error"
Violations shown as errors
Fails
Blocks unless STORYBOOK_A11Y_TODO is set
Burn-down (a11y debt)
STORYBOOK_A11Y_TODO=1 is currently set in CI so no story blocks PRs, giving the team time to fix existing violations incrementally. Per component:
Run pnpm a11y:inventory:ui-core and open a11y-inventory/report.md to find the component's violations.
Fix the violations in the component or its stories.
Upgrade the story from withA11yTesting → withA11yTestingError to lock in the clean state and get CI protection going forward.
Repeat until every ui-core component is on withA11yTestingError.
Once all ui-core stories are on "error" mode, remove STORYBOOK_A11Y_TODO from test:storybook:ci to fully gate CI. Watch the Datadog dashboardopens in a new tab to confirm the trend is falling.
Troubleshooting
Stale results / cached bundle — clear the Vite dep cache: pnpm storybook:clean (rm -rf node_modules/.vite), then re-run.
Audit reports 0 violations on a first run — treat a suspiciously clean result as a signal to inspect the raw vitest-results.json, not a result to trust. Violations in "error" mode throw before task.meta.reports is populated, so the parser recovers them from Vitest failure messages; a parser regression can silently under-count.
A story fails for a non-a11y reason — it shows up under unattributed_failures, not the violation counts. Check the story's play function.