The E2E Suite Went Green While Every Route Reported MISS

The routing crawl finished, and I got what I wanted: a green run, 26 routes checked. Then I read the log. Every single route in PortalKota's admin CMS printed MISS. The marker data-testid="page-shell" never showed up anywhere, according to the crawl, and the suite passed anyway.

My first guess was the timeout. The crawl runs against a slow staging build, and I assumed the assertion was timing out silently before the page settled. I raised the per-route timeout, ran it again, same result. Then I guessed the marker was genuinely missing and the catch block was swallowing the failure. So I put err.message inside the catch.

No message ever printed. Not once across 26 routes. That meant the code path I thought was failing never failed at all.

The assertion that asserted nothing

Here is the line that ran inside the loop:

// broken: resolves instantly, never asserts
await page.getByTestId('page-shell').toBeAttached();

locator.toBeAttached() is a method on the Locator object, the piece Playwright itself calls the central piece of its auto-waiting and retry-ability [1]. Called directly, it returns a promise that resolves right away. It is not an assertion. It does not retry, it does not throw, and it does not care whether the element exists. The only thing that wraps it in a real assertion, with retry-until-timeout semantics, is expect().

So every route "checked" in microseconds, found nothing, resolved happily, and my catch block had nothing to catch. The MISS labels came from my own instrumentation, which printed them whenever the shell lookup returned nothing. The test had no opinion about it whatsoever.

The fix is one wrapper and an explicit budget:

await expect(page.getByTestId('page-shell').first())
  .toBeAttached({ timeout: 45_000 });

Once wrapped, expect re-fetches the element and re-checks until the condition holds or the timeout expires, which defaults to 5 seconds [2]. Passing 45_000 gives the slow staging build room to settle instead of racing it.

This failure mode is worth naming precisely: the raw method gives you a promise-shaped object where an assertion should be, and TypeScript will not complain, because the method legitimately exists on Locator. Nothing looks wrong in review.

The dropdown that exists only while open

Fixing the crawl unblocked the rest of the suite, and the rest of the suite had opinions. The species filter on the tourism listing page was the next one to break.

It looks like a native in one part of the CMS and a Radix UI Select in another. For the native one, clicking the option text does nothing. Use selectOption with the visible label, the documented way to select an option in a native [3]:

await page.getByLabel('Filter jenis').selectOption({ label: 'Cagar alam' });

For the Radix one, the trap is different. Radix renders its items in a portal, so the SelectItem for a given option only exists in the DOM while the dropdown is open — the component is, as Radix documents it, a list of options triggered by a button [4]. A selector targeting the item text fails the moment you check it at the wrong time, and it will keep failing no matter how long you wait. The stable anchor is the trigger button, identified by its aria-label, for example "Filter jenis". Assert on the trigger, then click it, then query the item. Never make the item your primary handle.

What the adversarial loop kept finding

After the crawl was honest, I started breaking things on purpose. I deleted a row before the test checked it. I renamed a toast string. I submitted a form with an empty status field. Each break that the suite ignored became a new assertion, and it caught a spread of small factual errors.

MediaPicker in single-mode confirms immediately, without a separate confirm step, so a test that waits for one hangs forever. The toast copy is asserted exactly: "Wisata dibuat", "Halaman dibuat", "Slide ditambahkan". The service status enum expects aktif, not active, and sending the English word silently stores an invalid value rather than throwing. One form refuses to save without an explicit status value, and one POST payload needs an explicit is_active boolean instead of relying on the backend default.

None of these are exotic. They are ordinary UI facts that a suite can encode wrongly and still pass, because passing only means the selectors matched.

What I now do before trusting a green run

The rule I took from this: a green run proves your selectors matched, not that the page works. Before I believe a test, I break its assumption on purpose and check that the run turns red. Rename the toast. Empty the field. Point the marker at a component that never renders. If the run stays green, the test is decorative.

I now instrument every silent catch with the error message before the first real run, and I refuse to merge an assertion that is not wrapped in expect. The crawl that printed 26 MISS lines and passed taught me the expensive version of that lesson. The cheap version is one adversarial run before you trust the color.

Sources

[1] Locator - Playwright

[2] Assertions - Playwright

[3] Actions - Playwright

[4] Select - Radix UI