The Artifacts Folder Was Empty After a CI Failure

A test failed in the CI run for the KotaPortal admin CMS. I opened the artifacts folder at test-results/ and found nothing. No video, no trace, just an empty directory waiting for files that never arrived.

The config had trace: "on-first-retry", and retries stayed at its default of 0. A trace is only recorded when a test runs again after failing, and with zero retries a failure is final on the spot. The test lands straight in the "failed" category instead of "flaky", because flaky means it failed first and passed on a retry [3]. No retry, no second run, no recording. The evidence I needed only existed on a path the run never took.

Not an app bug. An evidence setup bug.

My first fix: record everything

My first move was to flip recording to always-on for every test. If every test records, I told myself, a failure can never leave me empty-handed again.

The problem is what stays behind. Successful runs keep their videos too, and videos of every test in a large suite add up fast. The docs describe exactly this trade-off when they define retain-on-failure for video: record each test, then remove all videos from successful runs [1]. Deleting on success is a first-class feature, not something I should build by hand.

Two recordings, two jobs

Once I separated the two jobs, the config stopped fighting itself. One suite, two kinds of recordings.

Failure evidence lives at the config level: trace: "retain-on-failure" and video: "retain-on-failure" in the use block. Videos are saved when the browser context closes at the end of a test [1], so the file exists the moment the test ends. Retention no longer depends on retries at all. Any final failure keeps its artifacts, on the first attempt or the fifth.

One honest disclaimer about the trace half. The trace-viewer docs recommend trace: "on-first-retry" for CI debugging and enumerate on-first-retry, on-all-retries, off, and on [2]. They do not list retain-on-failure for trace. In this config the same string is accepted, and the probe below proved a trace.zip is kept on final failure. That decoupling argument is my reasoning, not a doc quote: on-first-retry ties evidence to the retry policy, and with the default retries value that policy produces nothing [3].

Documentation tours are a different job. The tour spec for the KotaPortal admin CMS uses test.use({ video: { mode: "on", size: { width: 1280, height: 720 } } }), the per-file override form [4]. Here the video records even when the test passes, on purpose, because a tour that only survives a failure is useless. I added a waitForTimeout of 1200 ms between beats as breathing room for whoever narrates it later. Passing runs keep these files, and that is the point.

Both live in test-results/ at the repo root, which is gitignored. Artifacts never reach the repo.

Break a test on purpose

Do not trust the config until retention is proven. Deliberately make one test fail, run the suite, and open test-results/. You should find a .webm and a trace.zip for that test, and nothing from the passing ones. Turn retries up and confirm the same holds when the test fails on a later attempt too.

That probe is what separated a config I believe from a config I copied. Five minutes of intentional breakage replaces hope with files.

Sources

1. [Video options - Playwright docs](https://playwright.dev/docs/videos) 2. [Trace Viewer - Playwright docs](https://playwright.dev/docs/trace-viewer) 3. [Test retries - Playwright docs](https://playwright.dev/docs/test-retries) 4. [test.use() options - Playwright API](https://playwright.dev/docs/api/class-testoptions)