Visual Regression Testing
How to check that a change moves no pixel it shouldn't, and how to show the pixels it does move. Read this before refactoring CSS, design tokens, a shared component's markup or legacy stylesheets, and before writing "no visual change" in a merge request.
Unit tests and lint do not see rendering. A class Tailwind does not generate, an unlayered
stylesheet that now outranks a utility, a cascade-layer !important that used to win: each
passes every other check and still changes the page.
Which tool answers which question
| Question | Tool | Needs |
|---|---|---|
Did my branch change how any story renders, compared with develop? |
Storybook comparison | Two Storybook builds |
| Does the new component render like the old one it replaces? | Parity specs (a recipe) | Storybook on :6006 |
| Is a migrated layout still the size it was measured to be? | Geometry checks | Nothing (play function) |
| How do the busy screens look before and after, with real data? | Integration-testing screenshots | The MR's CI pipeline |
| Does a keyboard focus ring stay visible? | Foundations/Focus ring (src/theme/FocusRing.stories.tsx) |
Nothing (play function) |
| Did a SCSS refactor change a resolved colour, or what legacy styles emit? | Stylesheet checks | Nothing |
For a refactor that should change nothing (tokens, SCSS moves, component internals), run the Storybook comparison: it covers every story in both themes and needs no backend. For a change that moves pixels on purpose (spacing, padding, a new token value), run it too, and attach the largest differences to the merge request.
Ground rules
- Storybook renders with the app's stylesheet.
.storybook/preview.tsximportssrc/tailwind.cssand thensrc/styles/index.scss, in the same order assrc/index.tsx, and its theme decorator callsapplyTheme()to setdata-theme. A story therefore shows the same cascade the app does. The stylesheets load before the story renders, so every tool here waits only fordata-themebefore capturing. - Freeze before capturing. Wait for
document.fonts.ready(a capture in the fallback font differs on every text pixel), and inject*, *::before, *::after { transition: none !important; animation: none !important; }.gotoParity()andtheme-snapshots.mjsdo both. - Both themes. Capture light and dark: a token change is often right in one and wrong in the
other. Storybook takes the theme as a URL global:
iframe.html?id=<story-id>&viewMode=story&globals=theme:dark. - Pixels for colour, boxes for geometry. A pixel diff is the right check for a colour or a
missing element. For spacing and size, measure
boundingBox()instead: a 1px shift changes every pixel after it, so a pixel ratio cannot tell "everything moved 1px" from "one thing moved 20px". A parity spec checks dimensions first for this reason. - The root font size is 13px (12px below
md). rem values differ between the two, so measure at both viewports when a layout is rem-based (theAppHeaderLayout*stories do). - Storybook is not the app for focus and hover.
storybook-addon-pseudo-statesrewrites:focus-visibleand:hoverselectors. Confirm focus and hover results against the dev server on :8001. - Never bulk-accept a baseline. Look at every changed image, and state in the merge request what changed and by how much.
Storybook comparison: every story, base vs head
scripts/theme-snapshots.mjs screenshots every story of a static Storybook build in light and
dark, then pixel-diffs two such captures. Build the base branch and your branch, capture each, and
compare. It serves the build itself; nothing else needs to be running.
1 2 3 4 5 6 7 8 9 10 | |
capture <storybook-static> <out-dir> [--filter <regex>] [--font <family>] [--widths <px,…>] [--themes <light,dark>] [--styles]writes<story-id>.<theme>.png, full page, at a 1280×900 viewport.--filtermatches story ids (--filter '^actions-').--fontforces one font family everywhere, for comparing two builds that load different web fonts: the text then cannot differ, so colour changes stand out.--widthscaptures each story once per viewport width instead, as<story-id>.<theme>.w<px>.png. Use it for any layout change:576,768,992,1200are thesm,md,lgandxlbreakpoints, and a width belowsm(375) is a phone. A layout that breaks only at one breakpoint (a column that stops wrapping, a row squeezed on a phone) looks fine at 1280px.--themes lightskips the dark captures when the change cannot touch colour.compare <base-dir> <head-dir> <report-dir>writesreport.json(every pair:same,changedwithchangedPixels,ratioandsizeChanged, ormissing in head/new in head) and, for each changed pair, abase | head | changed pixels in redPNG. It prints the changed pairs sorted by ratio and exits 1 when anything differs.- The per-pixel threshold is 0.05: antialiasing passes, a one-step colour ramp change does not.
capture … --stylesalso writes<story-id>.<theme>.styles.json: the computed style of every element the story renders (portals included), keyed by its DOM path (tags and sibling positions, no class names). When both captures have them,comparelists every property that changed, asdiv:4>div:0>… padding-top 11px → 9px, understylesinreport.jsonand on the console, and exits 1 for them too. Use it for a refactor that should change no style at all, such as moving a stylesheet's rules into utilities: it names the element and the property, and it catches a change too small for the pixel threshold. Check that the stories you filter on actually render the changed markup (grep the head captures for the new value).capture … --root <px>forceshtml's font size (the rem root) before each story loads, androot-invariance <storybook-static> <out-dir> [--roots 13,20]captures at both roots into<out-dir>/root-13and<out-dir>/root-20and compares them into<out-dir>/report(Storybook's own 1rem canvas padding is pinned to 16px in both). A story that differs still sizes something in rem; see "Root font size" in design-tokens.md.comparepairs any two directories of PNGs by file name, so it also diffs the integration-testing screenshots.
Reading the report. For a change that should be invisible, the expected result is
0 changed in both themes; open every changed sheet. For an intended change, start with the
largest ratios and every size changed entry (the page got taller or shorter), check that each is
the change you meant, and name the stories you checked in the merge request. A ratio says how much
of the page changed, not by how much: a 1px shift near the top of a story changes most of it.
Parity specs: side by side comparison
A parity spec renders two implementations on one Storybook page and diffs the two live renders against each other, in the same browser and run, so there is no committed baseline to go stale.
This pattern is a recipe for migrations that replace a component in place:
- The story renders each case twice, wrapped in
data-pair="<case>" data-role="variant-a"anddata-role="variant-b", at the same fixed width. - The spec creates a harness and runs each case in both themes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | |
- The checks, in order: dimensions (
boundingBox(), withinmaxDimensionSlackPx), the pixelmatch ratio (diffRatioThreshold), and dominant-colour chromaticity (chromaticityTolerance), which catches a wrong hue on a small, text-heavy element that a ratio would pass. A failing case writes<name>-a.png,-b.pngand-diff.pngtodiffDir. - Thresholds are measured, not borrowed. Run the spec, record the largest ratio and
chromaticity across all cases and both themes, and set each threshold a little above that
noise floor.
expectBufferParity()takes two captured buffers, for cases that need an interaction (hover, focus) between the two captures.
Geometry checks: pinned measurements
When components stand alone, play functions pin the measurements instead. The AppHeaderLayout stories in
packages/ui/src/TopBar/TopBar.stories.tsx render the header at a desktop and a mobile viewport,
in both themes, and assert each element's getBoundingClientRect() against reference values, within 0.3px. yarn test:storybook runs them with every other story, in CI too.
Use this pattern for a layout whose measurements must survive a refactor: find elements by
data-testid, record where each expected number came from, and make each viewport and theme its
own story. The Vitest runner sizes the page from globals.viewport, which takes an object naming
one of the presets in .storybook/preview.tsx: globals: { viewport: { value: 'mobile' } }. A bare
string (viewport: 'mobile') is ignored, and the story renders at 1200×900.
Integration-testing screenshots
waldur-integration-testing drives a full Waldur stack in Playwright. Its tests call
take_screenshot("<name>") on a page or component, which saves docs/screenshots/<name>.png, and
every E2E job keeps docs/screenshots/ as an artifact for a week. These are the only screenshots
of real pages with data that need no local backend.
- The head: a HomePort merge request that changes runtime files triggers the suite downstream
("Trigger integration tests") with the MR's image.
Select E2E suites(scripts/select-e2e-suites.mjs) narrows it to the areas the change can affect, so an MR may run fewer jobs thandevelopdoes. - The base: the downstream pipeline triggered from
develop, which runs every suite against:latest. Use one at or after your branch's merge base. - Comparing: download each job's artifacts from both downstream pipelines, put each side's
PNGs in one directory, and run
node scripts/theme-snapshots.mjs compare <base-dir> <head-dir> <report-dir>. Pairs exist only for jobs that ran on both sides; the rest show asmissing in headornew in head. - Expect noise. Data differs between runs (dates, generated names, row counts), so a pixel ratio ranks the screenshots rather than passing or failing them. Look at the side-by-side sheets.
tests/test_docs_screenshots.pydefinesgenerate_documentation_screenshots, which is not namedtest_*, so pytest does not collect it and its screenshots are not produced in CI.
Stylesheet checks (no browser)
node scripts/color-diff.mjs save|compare <snapshot.json>records the colour every declaration of the compiled legacy stylesheet resolves to, in light and dark, and reports each one that changed. Use it for SCSS colour-variable refactors; see design-tokens.md → "Verifying a refactor changes nothing".yarn styles:usage compare(scripts/bootstrap-css-usage.mjs, runs in CI) fails when the compiled legacy stylesheet gains an unused class or its selectors are added or reordered, and prints the selector diff. After an intended change, runyarn styles:usage saveand commit the baseline. See development-setup.md.
What CI runs
| Job | What it catches |
|---|---|
| Run Storybook tests | A story that fails to render or whose play function fails (yarn test:storybook), including the focus-ring and geometry checks. No pixel comparison; allow_failure on merge requests. |
yarn styles:usage compare |
Added or reordered selectors and new unused classes in the legacy stylesheet. |
| Trigger integration tests | The downstream E2E suite, with its docs/screenshots/ artifacts. |
theme-snapshots.mjs does not run in CI. Run it locally when the change can move pixels and put the
result in the merge request.
Reporting a visual change in a merge request
- Which tools ran, against which base commit, and in which themes and viewports.
- For an intended change: the largest difference observed (in pixels for geometry, as a ratio for pixel diffs), and side-by-side images of the busiest affected screens.
- For a change meant to be invisible: "0 changed" from the Storybook comparison, or the stories that did change and why that is acceptable.
- What was not checked, and why (no backend, a suite that did not run).
Commands
| Command | What it does |
|---|---|
yarn build-storybook |
Static Storybook in storybook-static/, the input of theme-snapshots.mjs |
node scripts/theme-snapshots.mjs capture … / compare … |
Storybook comparison |
node scripts/theme-snapshots.mjs root-invariance … |
Stories that render differently at a 13px and a 20px root |
yarn test:storybook |
Every story and its play function, the focus-ring and geometry checks too |
yarn vitest --project=storybook run <path/to/X.stories.tsx> |
One story file |
yarn test:storybook runs each story in headless Chromium through @storybook/addon-vitest; it
needs no running Storybook or app.