Fake implementations must fail: the selftest that tests itself
A green suite proves nothing if a fake can pass it. I fed three fake implementations into my 32-test selftest to prove the grader works.
TL;DR
After my 32-test suite went green, I worried it might be vacuous, so I built three fake validators (always pass, always fail, always emit every error code) that must each score zero passes. This is manual mutation testing, a known technique. I also pinned the selftest's sha256 so silent grader edits get caught.
The green suite that started bothering me
The terminal showed 32 green tests in ingest-blueprint-selftest.sh. PASS=32, FAIL=0. A day earlier I had shipped a different version of that same suite: 32 deliberately red tests, frozen as a CLI contract. Now the implementation caught up and everything passed. Instead of relaxing, I caught myself asking an uncomfortable question: what if all 32 tests pass and the whole thing proves nothing?
My working assumption was the classic one: green means good. Tests pass, code works. Then I replayed a scenario in my head. What if the validator under test is just exit 0 in every branch? Every test still goes green. A suite like that has a name: vacuous.
This is not just paranoia. The official coverage.py FAQ points out that def and class lines execute on import, so a module can show moderate coverage while zero real tests actually ran [3]. Ned Batchelder, the maintainer, says it himself: coverage is good, but it isn't perfect [3]. Green on a dashboard is not the same as proven.
Three fakes that must all fail
Commit c609608 in my internal repo is the answer. The 32-test selftest got re-frozen with three new guards. I built three fake implementations with extreme behaviors, then fed each one to the suite:
- FAKE-OK. Always exits 0, pretending every validation passed.
- FAKE-FAIL. Always fails, processes nothing.
- FAKE-ALLREASONS. The sneakiest one: always emits every failure reason code the validator knows.
The acceptance bar is brutal: each fake must score exactly PASS=0 and FAIL=32. One fake sneaking through means the suite is vacuous. FAKE-ALLREASONS matters because of a lazy-validator pattern I have seen too often: checking "an error string appeared" instead of "the right error string appeared". So the tests now assert exact reason codes. Failing with R75 means something different than failing with R80b, and my tests must tell those apart.
One more guard. The selftest file pins its own sha256 in the commit message, starting with 8100450c. Worst case: someone edits the grader to always pass so the verification step finishes faster. Without a locked hash, the suite stays green and nobody notices the guard is dead from the inside. With the hash, every silent edit shows up immediately. Baseline after the re-freeze: STUB=32 against the stub, then the kernel suite at PASS=74, FAIL=0, STUB=0.
The official name is mutation testing
What I built is the manual version of a known idea. The Stryker Mutator docs describe the principle: introduce changes (mutants) into your code, run the tests, and expect them to fail. Tests that don't fail mean your tests do not sufficiently cover the code [1]. Surviving mutants are missing tests [1].
Automated tools like Stryker for JavaScript or mutmut for Python do this beautifully at scale, flipping operators and re-running suites [1][2]. For a small bash CLI project, I chose the manual lane: three fake scripts, a PASS=0 x3 bar, one locked hash. I would rather know for certain that my grader cannot be fooled by a script that always succeeds than chase a fake 100 percent coverage number. Proof that a system can reject the wrong thing beats proof that it accepts the right one.
If you want to try this yourself: write a one-line bash script containing only exit 0, feed it to your suite, and run it. A healthy suite must reject it. If yours stays green, you just found a bigger problem than any bug.
Sources
[1] Stryker Mutator, What is mutation testing?
[2] mutmut, Python mutation tester documentation
[3] Ned Batchelder, coverage.py FAQ