The test that could not fail

Every indicator was green for a new feature in a trace CLI: a commit-hash seed widens itself by pulling task IDs from commit subjects, then rescans the board, decisions, and logs exactly once. The assertion guarding it was modest: assert run(...) == 0. The logic felt airtight, if the process returns zero, the expansion must have run.

Review broke the assumption with one sentence: the assertion is vacuous. Inside the fixture, the commits door always matches the fixture's own commit hash. So run(...) returned zero not because the expansion worked, but because the no-expansion path already satisfied the same condition. The proof ritual was brutal: strip the expansion block from a copy of the source file, rerun the suite, and the test must fail. It stayed green. The feature could be deleted without a single test objecting.

A test that cannot fail is not a test

This is manual mutation testing survival. The Stryker documentation phrases the principle: mutation testing introduces changes to code and runs the tests against the changed code, and "If your tests fail then the mutant is killed . If your tests passed, the mutant survived ." [4]. The mutant here was the deletion of the expansion feature, and the mutant survived. Survival means a test is missing, not that the code is correct.

The trap is specific: an exit code only proves the program did not crash. For a feature that produces an observable change, "ran without error" is not "did its job". Overly loose success criteria let another path steal the pass.

Pin the assertion to specific evidence

The fix shifts validation from the exit code to the widened state. Standard output is captured through contextlib.redirect_stdout, which the Python documentation defines as a "Context manager for temporarily redirecting sys.stdout to another file object ." [6], pointed at io.StringIO. The first line of output is then asserted: the hash seed must have widened into a task ID that previously existed only in the fixture board. In pytest the equivalent pattern ships ready-made through its capture fixtures; readouterr() returns "a namedtuple with two attributes, out and err ." [5], and for pure-stdout CLIs the fd-level equivalent reads straight from the file descriptors. Which tool you use does not matter; what matters is you test output, not the status code.

The new assertion was checked the same way the old one was convicted: without the expansion block, the suite must and does fail. Both directions are now covered: feature present, test passes; feature gone, test objects.

The feature under test is itself bounded by design so its failures stay easy to diagnose. The expansion runs at most one pass: the second scan never becomes a new seed, the chain stays O(2x) and cannot get intoxicated. The task-ID regex is deliberately scoped to two card prefixes; letting any other token in lets the hash direction wander. A seed cap keeps the output readable.

The general lesson left standing: assert on observable effects. Exit code for crashes, output for behavior changes. A green suite may be telling a story about another path nobody noticed.

Sources

4. Stryker Mutator documentation, "What is mutation testing?" (accessed 2026-10-12): "Mutation testing introduces changes to your code, then runs your unit tests against the changed code."; "If your tests fail then the mutant is killed. If your tests passed, the mutant survived."
5. pytest documentation, "How to capture stdout/stderr output" (accessed 2026-10-12): "The return value of readouterr() is a namedtuple with two attributes, out and err."
6. Python documentation, contextlib (accessed 2026-10-12): "Context manager for temporarily redirecting sys.stdout to another file object."