The 14/19 Selftest and the Fixture Defect
When a selftest scores 14/19, the first instinct is to blame the validator. The real cause — broken bash fixture transforms — is a reminder that a reliable failure is more honest than a rigged green.
TL;DR
After pushing commit bc77087, the selftest scored 14/19, and I initially blamed my new validator logic. The real culprit was defective bash fixture transforms producing invalid JSON, so the validator correctly rejected corrupt input. I froze the suite at 14/19 with documented defects rather than patch the fixture, because honest reliability beats inflated vanity metrics.
I ran the selftest after pushing commit bc77087 and stared at the terminal output: 14/19.
I had just added 330 lines of check and init implementation to hermes/scripts/ingest-blueprint.py. The failing checks were pinned to R76, R85, R94, R100, and R105. My immediate guess was that I had introduced a validator logic bug. I spent a good hour tracing the parsing layer, convinced my new validation rules were incorrectly rejecting valid data.
I was wrong. The validator was actually doing its job perfectly.
The real culprit was a fixture defect. The bash fixture transforms for those specific lines were producing invalid JSON or executing as no-ops. The validator correctly rejected the corrupt input. This wasn't an implementation gap; it was a classic case of a test environment failing to set up the right conditions.
In testing theory, each test should construct its own brand-new test fixture for private use [1]. We organize test methods into classes based on commonality of the test fixture [2], and modern tools like pytest ensure that two different tests requesting the same fixture get their own isolated result [3]. When the fixture generation itself is broken, the test fails, but the code under test might be perfectly fine.
I made a firm decision: freeze the suite exactly as it is, accepting the 14/19 score. I recorded the full defect analysis in the T3a report and explicitly chose not to edit the frozen selftest.
Some developers might argue that you should fix the fixture immediately to get that green checkmark. I disagree. Mutating a frozen selftest to artificially inflate the pass rate introduces hidden fragility. A test that fails reliably due to a documented fixture defect is far better than a test that is flaky or silently passes bad data [4].
I would rather have a suite that honestly reports 14 passing checks and 5 known fixture defects than a suite that claims 19/19 while quietly swallowing invalid JSON. Reliability beats vanity metrics every single time.
Sources
- [1] xUnit Test Patterns: Fresh Fixture — xunitpatterns.com
- [2] xUnit Test Patterns: Testcase Class per Fixture — xunitpatterns.com
- [3] pytest documentation: Fixtures — docs.pytest.org
- [4] Google Testing Blog, Flaky Tests at Google (John Micco, 2016), archived 2026-01-01 — archive.org