Skip to content

Visual Regression Testing

How to check that a change moves no pixel it shouldn't, and how to show the pixels it does move. Read this before refactoring CSS, design tokens, a shared component's markup or legacy stylesheets, and before writing "no visual change" in a merge request.

Unit tests and lint do not see rendering. A class Tailwind does not generate, an unlayered stylesheet that now outranks a utility, a cascade-layer !important that used to win: each passes every other check and still changes the page.

Which tool answers which question

Question Tool Needs
Did my branch change how any story renders, compared with develop? Storybook comparison Two Storybook builds
Does the new component render like the old one it replaces? Parity specs (a recipe) Storybook on :6006
Is a migrated layout still the size it was measured to be? Geometry checks Nothing (play function)
How do the busy screens look before and after, with real data? Integration-testing screenshots The MR's CI pipeline
Does a keyboard focus ring stay visible? Foundations/Focus ring (src/theme/FocusRing.stories.tsx) Nothing (play function)
Did a SCSS refactor change a resolved colour, or what legacy styles emit? Stylesheet checks Nothing

For a refactor that should change nothing (tokens, SCSS moves, component internals), run the Storybook comparison: it covers every story in both themes and needs no backend. For a change that moves pixels on purpose (spacing, padding, a new token value), run it too, and attach the largest differences to the merge request.

Ground rules

  • Storybook renders with the app's stylesheet. .storybook/preview.tsx imports src/tailwind.css and then src/styles/index.scss, in the same order as src/index.tsx, and its theme decorator calls applyTheme() to set data-theme. A story therefore shows the same cascade the app does. The stylesheets load before the story renders, so every tool here waits only for data-theme before capturing.
  • Freeze before capturing. Wait for document.fonts.ready (a capture in the fallback font differs on every text pixel), and inject *, *::before, *::after { transition: none !important; animation: none !important; }. gotoParity() and theme-snapshots.mjs do both.
  • Both themes. Capture light and dark: a token change is often right in one and wrong in the other. Storybook takes the theme as a URL global: iframe.html?id=<story-id>&viewMode=story&globals=theme:dark.
  • Pixels for colour, boxes for geometry. A pixel diff is the right check for a colour or a missing element. For spacing and size, measure boundingBox() instead: a 1px shift changes every pixel after it, so a pixel ratio cannot tell "everything moved 1px" from "one thing moved 20px". A parity spec checks dimensions first for this reason.
  • The root font size is 13px (12px below md). rem values differ between the two, so measure at both viewports when a layout is rem-based (the AppHeaderLayout* stories do).
  • Storybook is not the app for focus and hover. storybook-addon-pseudo-states rewrites :focus-visible and :hover selectors. Confirm focus and hover results against the dev server on :8001.
  • Never bulk-accept a baseline. Look at every changed image, and state in the merge request what changed and by how much.

Storybook comparison: every story, base vs head

scripts/theme-snapshots.mjs screenshots every story of a static Storybook build in light and dark, then pixel-diffs two such captures. Build the base branch and your branch, capture each, and compare. It serves the build itself; nothing else needs to be running.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Base build, in a separate worktree so your checkout stays on your branch
git worktree add ../homeport-base origin/develop
(cd ../homeport-base && yarn install --immutable && yarn build-storybook)

# Head build
yarn build-storybook

node scripts/theme-snapshots.mjs capture ../homeport-base/storybook-static /tmp/vt/base
node scripts/theme-snapshots.mjs capture storybook-static /tmp/vt/head
node scripts/theme-snapshots.mjs compare /tmp/vt/base /tmp/vt/head /tmp/vt/report
  • capture <storybook-static> <out-dir> [--filter <regex>] [--font <family>] [--widths <px,…>] [--themes <light,dark>] [--styles] writes <story-id>.<theme>.png, full page, at a 1280×900 viewport. --filter matches story ids (--filter '^actions-'). --font forces one font family everywhere, for comparing two builds that load different web fonts: the text then cannot differ, so colour changes stand out.
  • --widths captures each story once per viewport width instead, as <story-id>.<theme>.w<px>.png. Use it for any layout change: 576,768,992,1200 are the sm, md, lg and xl breakpoints, and a width below sm (375) is a phone. A layout that breaks only at one breakpoint (a column that stops wrapping, a row squeezed on a phone) looks fine at 1280px. --themes light skips the dark captures when the change cannot touch colour.
  • compare <base-dir> <head-dir> <report-dir> writes report.json (every pair: same, changed with changedPixels, ratio and sizeChanged, or missing in head / new in head) and, for each changed pair, a base | head | changed pixels in red PNG. It prints the changed pairs sorted by ratio and exits 1 when anything differs.
  • The per-pixel threshold is 0.05: antialiasing passes, a one-step colour ramp change does not.
  • capture … --styles also writes <story-id>.<theme>.styles.json: the computed style of every element the story renders (portals included), keyed by its DOM path (tags and sibling positions, no class names). When both captures have them, compare lists every property that changed, as div:4>div:0>… padding-top 11px → 9px, under styles in report.json and on the console, and exits 1 for them too. Use it for a refactor that should change no style at all, such as moving a stylesheet's rules into utilities: it names the element and the property, and it catches a change too small for the pixel threshold. Check that the stories you filter on actually render the changed markup (grep the head captures for the new value).
  • capture … --root <px> forces html's font size (the rem root) before each story loads, and root-invariance <storybook-static> <out-dir> [--roots 13,20] captures at both roots into <out-dir>/root-13 and <out-dir>/root-20 and compares them into <out-dir>/report (Storybook's own 1rem canvas padding is pinned to 16px in both). A story that differs still sizes something in rem; see "Root font size" in design-tokens.md.
  • compare pairs any two directories of PNGs by file name, so it also diffs the integration-testing screenshots.

Reading the report. For a change that should be invisible, the expected result is 0 changed in both themes; open every changed sheet. For an intended change, start with the largest ratios and every size changed entry (the page got taller or shorter), check that each is the change you meant, and name the stories you checked in the merge request. A ratio says how much of the page changed, not by how much: a 1px shift near the top of a story changes most of it.

Parity specs: side by side comparison

A parity spec renders two implementations on one Storybook page and diffs the two live renders against each other, in the same browser and run, so there is no committed baseline to go stale.

This pattern is a recipe for migrations that replace a component in place:

  1. The story renders each case twice, wrapped in data-pair="<case>" data-role="variant-a" and data-role="variant-b", at the same fixed width.
  2. The spec creates a harness and runs each case in both themes:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
const { gotoParity, expectPixelParity } = createParityHarness({
  storyId: 'component-parity--default',
  diffDir: path.join(__dirname, '../test-results/parity-diffs'),
  thresholds: {
    maxDimensionSlackPx: 0.5,
    diffRatioThreshold: 0.06,
    pixelmatchThreshold: 0.25,
    foregroundDistanceThreshold: 30,
    chromaticityTolerance: 3,
  },
});

// In each test, after `await gotoParity(page, theme)`:
await expectPixelParity(
  page.locator(`[data-pair="${caseId}"][data-role="variant-a"]`),
  page.locator(`[data-pair="${caseId}"][data-role="variant-b"]`),
  `${theme}-${caseId}`,
);
  1. The checks, in order: dimensions (boundingBox(), within maxDimensionSlackPx), the pixelmatch ratio (diffRatioThreshold), and dominant-colour chromaticity (chromaticityTolerance), which catches a wrong hue on a small, text-heavy element that a ratio would pass. A failing case writes <name>-a.png, -b.png and -diff.png to diffDir.
  2. Thresholds are measured, not borrowed. Run the spec, record the largest ratio and chromaticity across all cases and both themes, and set each threshold a little above that noise floor. expectBufferParity() takes two captured buffers, for cases that need an interaction (hover, focus) between the two captures.

Geometry checks: pinned measurements

When components stand alone, play functions pin the measurements instead. The AppHeaderLayout stories in packages/ui/src/TopBar/TopBar.stories.tsx render the header at a desktop and a mobile viewport, in both themes, and assert each element's getBoundingClientRect() against reference values, within 0.3px. yarn test:storybook runs them with every other story, in CI too.

Use this pattern for a layout whose measurements must survive a refactor: find elements by data-testid, record where each expected number came from, and make each viewport and theme its own story. The Vitest runner sizes the page from globals.viewport, which takes an object naming one of the presets in .storybook/preview.tsx: globals: { viewport: { value: 'mobile' } }. A bare string (viewport: 'mobile') is ignored, and the story renders at 1200×900.

Integration-testing screenshots

waldur-integration-testing drives a full Waldur stack in Playwright. Its tests call take_screenshot("<name>") on a page or component, which saves docs/screenshots/<name>.png, and every E2E job keeps docs/screenshots/ as an artifact for a week. These are the only screenshots of real pages with data that need no local backend.

  • The head: a HomePort merge request that changes runtime files triggers the suite downstream ("Trigger integration tests") with the MR's image. Select E2E suites (scripts/select-e2e-suites.mjs) narrows it to the areas the change can affect, so an MR may run fewer jobs than develop does.
  • The base: the downstream pipeline triggered from develop, which runs every suite against :latest. Use one at or after your branch's merge base.
  • Comparing: download each job's artifacts from both downstream pipelines, put each side's PNGs in one directory, and run node scripts/theme-snapshots.mjs compare <base-dir> <head-dir> <report-dir>. Pairs exist only for jobs that ran on both sides; the rest show as missing in head or new in head.
  • Expect noise. Data differs between runs (dates, generated names, row counts), so a pixel ratio ranks the screenshots rather than passing or failing them. Look at the side-by-side sheets.
  • tests/test_docs_screenshots.py defines generate_documentation_screenshots, which is not named test_*, so pytest does not collect it and its screenshots are not produced in CI.

Stylesheet checks (no browser)

  • node scripts/color-diff.mjs save|compare <snapshot.json> records the colour every declaration of the compiled legacy stylesheet resolves to, in light and dark, and reports each one that changed. Use it for SCSS colour-variable refactors; see design-tokens.md → "Verifying a refactor changes nothing".
  • yarn styles:usage compare (scripts/bootstrap-css-usage.mjs, runs in CI) fails when the compiled legacy stylesheet gains an unused class or its selectors are added or reordered, and prints the selector diff. After an intended change, run yarn styles:usage save and commit the baseline. See development-setup.md.

What CI runs

Job What it catches
Run Storybook tests A story that fails to render or whose play function fails (yarn test:storybook), including the focus-ring and geometry checks. No pixel comparison; allow_failure on merge requests.
yarn styles:usage compare Added or reordered selectors and new unused classes in the legacy stylesheet.
Trigger integration tests The downstream E2E suite, with its docs/screenshots/ artifacts.

theme-snapshots.mjs does not run in CI. Run it locally when the change can move pixels and put the result in the merge request.

Reporting a visual change in a merge request

  • Which tools ran, against which base commit, and in which themes and viewports.
  • For an intended change: the largest difference observed (in pixels for geometry, as a ratio for pixel diffs), and side-by-side images of the busiest affected screens.
  • For a change meant to be invisible: "0 changed" from the Storybook comparison, or the stories that did change and why that is acceptable.
  • What was not checked, and why (no backend, a suite that did not run).

Commands

Command What it does
yarn build-storybook Static Storybook in storybook-static/, the input of theme-snapshots.mjs
node scripts/theme-snapshots.mjs capture … / compare … Storybook comparison
node scripts/theme-snapshots.mjs root-invariance … Stories that render differently at a 13px and a 20px root
yarn test:storybook Every story and its play function, the focus-ring and geometry checks too
yarn vitest --project=storybook run <path/to/X.stories.tsx> One story file

yarn test:storybook runs each story in headless Chromium through @storybook/addon-vitest; it needs no running Storybook or app.