Five Stale Baselines at Once: Regeneration Is a Decision
Five hard failures at once on the public pages, all visual baselines. When --update-snapshots is safe, and when it silently blesses a broken render.
TL;DR
Five visual failures on the public pages looked like stale baselines after recent UI changes, but blind regeneration would lock in any shipped bugs. Regenerate in the same environment as CI with an explicit mode, then review changed PNGs like a code diff to catch unintended layout shifts. A visual baseline is an executable spec, so unreviewed updates remove oversight rather than fix tests.
The full client portal suite run took 44.8 minutes on 2 workers. The summary: 103 tests passed, 9 flaky (passed on retry), and 5 hard failures. All five failures sat exactly on the public pages: the home page and four sector pages. All visual tests.
Five failures of the same kind, right on the most visible pages. My first reflex had formed before I finished reading the log: these are obviously stale baselines, just regenerate.
Five failures stuck on the public pages
Playwright stores reference screenshots as PNG files in a <spec-file>-snapshots/ folder, named after the test title, browser, and platform — for example home-chromium-linux.png. The visual comparison itself uses await expect(page).toHaveScreenshot(): on the first execution Playwright creates the reference, and later runs compare the render against that reference [1].
When the tests fail, all five messages follow the same pattern: the actual screenshot differs from the expectation. Understandable, since legitimate frontend changes were shipped earlier by the same program: a new header menu, new breadcrumbs, and one new page. The five failing pages happened to use those components.
But this is where I stopped before typing any command. “The UI change has shipped” is not proof that the change is correct.
The wrong guess: a UI change makes baselines automatically stale
The common wrong guess here: once you know there is a UI change, visual failures are automatically considered stale. Yet --update-snapshots validates nothing. The flag writes the current render as the new truth [1]. If the shipped change is broken — a cut-off header, breadcrumbs covering content, a jumping layout — the regen enshrines that broken render as the baseline. The suite goes green, and the damage ships to production without a trace.
Playwright’s own documentation states that the snapshot folder should be committed to version control and every change reviewed [1]. The sentence “review any changes” applies to regens too, not just code changes.
There is one more trap: browser rendering varies by OS, version, hardware, power source (battery versus adapter), headless mode, and other factors [1]. If the baselines were made in one environment and the regen runs in another, the diff can be a pure rendering artifact. Reviewing PNGs that were compared in the wrong environment from the start is pointless.
A safe regen is a reviewed regen
The procedure I use, in order:
- Make sure the regen runs in the same environment the baselines live in. In this repo, that is the same headless Linux as CI. Do not regenerate from a laptop with a different OS.
- Run the regen with an explicit mode, not relying on the default:
npx playwright test --update-snapshots=changed- Review every changed PNG like a code diff. Open the old and new files side by side, and ask one thing: is the diff only in the areas that did change (header, breadcrumbs), or is there movement you never asked for? If the layout shifts outside the change area, that is not a stale baseline. That is a bug.
- Commit the tests and the snapshot folder in the same commit. Snapshots are part of the specification, not a build artifact that may drift apart.
- Verify with a full run without the flag. In this case the second run turned the visual tests green in 38 seconds, then 27 seconds on the run after.
The --update-snapshots flag accepts five mode values: all, changed, missing, none, default. Without the flag, the regen behavior is default; once the flag is written without a value, the value is changed [3]. I always write the mode value explicitly so the intent is readable in the command history.
Mechanically: toHaveScreenshot() waits for two consecutive screenshots to yield the same result before comparing, so renders that are still moving or flaky are filtered out before the comparison happens [2]. If failures persist after two stable runs, the difference is real, not noise.
Why do these rules matter? Because a visual baseline is an executable design specification. Running --update-snapshots is changing the specification. A specification change that is not reviewed is not regeneration; it is the removal of oversight.
Sources
[1] Visual comparisons - Playwright