Why My Login Assertion Never Failed

I was reading through frontend/e2e/support/fixtures.ts, the admin login helper used across our entire E2E test suite, and stopped at the very last line: an assertion checking only the URL. At first glance, the safeguard looked neat. If the login failed, the page would not navigate, and the test would immediately complain.

What I missed: the /admincms route serves a dual purpose, acting as both the pre-login screen and the post-login dashboard, so the condition evaluates to true even when the development admin credentials are rejected. An assertion that cannot fail is a dangerous illusion: this one never failed because it literally could not.

I initially assumed this helper was rushed. In reality, the entire spec depended on it, and the assertion that looked the safest was actually the most dangerous, as it never raised an alarm when a login failed. I replaced that line with a state marker that only appears after a valid session is established. This change was small, but the principle behind it applies to any assertion: an assertion is only as valuable as its ability to fail.

Auto-retry is not a safety net

Playwright provides the PageAssertions class containing assertions for page states [5], and toHaveURL is indeed one of them. Its documentation uses URL assertions after a click that genuinely changes the route, not for waiting on a login result in a route that never shifts.

Assertions in Playwright automatically retry: they are re-evaluated until the condition is met or the timeout is reached [1]. The timeout never comes into play here because the condition is already true from the very first millisecond. Auto-retry only helps if the condition can change from false to true. If the route is exactly the same on both success and failure paths, no matter how many times it retries, the result remains true for a failing test.

Official best practices emphasize the same direction: automated tests should verify the code as the end user experiences it, avoiding implementation details invisible to the user [2]. Users do not judge a login by the URL in the address bar. They judge it by the elements that appear after a successful entry.

There is a quick way to check for this kind of suspicion without writing a new test: run the failure scenario and see which assertions still pass. If the session is cleared or credentials are rejected, and an assertion remains green, that assertion is not verifying anything. On the failure path, the URL in this fixture still points to /admincms, exactly matching the success path.

Markers that only appear after a valid session

The fix is in the fixture diff, where four lines became one:

// Before: assertion against a route that never changes
await expect(page).toHaveURL(/admincms/);

// After: waiting for a post-login marker
await expect(page.getByTestId('admin-user-chip')).toBeVisible();

The admin-user-chip is rendered by the admin layout only if the session is valid. If the login fails, the element never appears, and the test stops red as it should, instead of passing because the route always matches.

The same variation of failure often appears in other fixtures: asserting against placeholder text filled by the framework, or against an element that is always present on the page. The pattern is identical: a condition whose truth is guaranteed from the start.

data-testid is not a shortcut

Testing Library guiding principles famously state that tests resembling how the software is used provide the most confidence [3]. Therefore, the query priority matters: search by role or text visible to the user first, and use data-testid only when neither has a stable equivalent [4]. I still use it here, as it is far better than tying the test to a DOM structure or CSS class name that could change at any moment.

The practical measure applies to assertions anywhere: before placing an assertion, ask yourself what it looks like on the failure path. An assertion that yields identical results on both paths can simply be deleted. It adds test lines without adding confidence, and its cost is only felt when a real failure slips through silently.

A previous fixture article discussed why the login helper is kept separate from the spec: Fixture First: Writing the Login Helper Before Any Spec. The line I replaced in that article was the one that previously looked the safest, and in my opinion, it is the most honest part of this case: the most expensive failure is not a noisy test, but a silent test that can never fail.

Sources

[1] Playwright docs: Test assertions
[2] Playwright docs: Best practices
[3] Testing Library: Guiding principles
[4] Testing Library: getByTestId
[5] Playwright docs: PageAssertions