The Validator Vanished, CI Stayed Green
A mirror sync deleted a validation script and CI stayed green. Three validator failure classes and two cheap fences: mandatory anchors plus a frozen checksum.
TL;DR
A git mirror sync silently deleted our validation script, yet CI stayed green because nothing verified the checker's existence. The postmortem surfaced three silent failure modes: missing, broken, and environment-brittle validators. The fix adds two cheap fences, a missing-anchor rule and a frozen checksum, so green finally means something.
That morning I opened the CI dashboard and every check was green. On a whim, I looked into the scripts folder and check-close-actions-drift.sh was gone. A git mirror sync had silently deleted it. The annoying part: the pipeline still passed, because not a single step verified whether the checker itself still existed. The watchdog disappeared, yet the health report stayed clean.
My first guess was wrong. I assumed green meant healthy. In reality, the suite stayed green precisely because the validator had vanished. After digging through the fix commit's diff, I found not one but three classes of validator failure at once, and all of them operate silently.
Three classes of validator failure
The first class is the most obvious: the missing validator. Nothing verified the existence of the verification tool itself, so its removal left no trace on the dashboard.
The second class is subtler: the silently broken validator. A bug wrapped the row iterator _iter_rows twice, and as a result the unclosed-action rule (R105) never fired. Meanwhile there was a translation file that genuinely lost its closing action. This is my favorite kind of failure to hate: the tool exists, its report says healthy, but it is mute.
The third class: the brittle validator. Selftest fixtures flaked because of escaped-brace substitution in bash function context on bash 5.2.37. String substitution that relies on shell parsing quirks makes test results vary between environments. The fix moved to path-based mutation through a mutate helper, which is more deterministic because it does not depend on version-specific shell behavior.
Drift is normal, not an incident
Once those three classes were visible, one small conclusion emerged: parallel copies are normal. In distributed workflows, every party works at its own pace and the second developer must merge before pushing to avoid overwriting the first developer's work. Sync is manual by design, so drift is always lurking [3].
Translated documents follow a similar pattern. Sphinx treats message catalogs as material compiled into locale directories, for example ./locale/es/LC_MESSAGES/usage.mo, and the build output lives a life of its own, separate from the original source [2]. Every copy with its own lifecycle needs an alarm, not just the hope that everything stays in sync.
Two cheap fences
The first fence: mandatory anchors. The new translated-anchor-missing rule (R89) makes any translated section without anchor_sections fail validation immediately. No more translated sections standing alone without a trace back to their source section. When the source document changes, those hanging anchors become visible automatically, instead of being discovered months later by a reader.
The second fence: the frozen checksum. The selftest was re-frozen with the value eb512923, and any change to the test script breaks the hash match. Keep the limit in mind: checksums reliably detect accidental corruption, because the chance of two files sharing a digest is vanishingly small, but they are no protection against deliberate tampering [4]. For internal drift alarms that never face an attacker, that is more than enough.
Meanwhile, reading constants like STATE_ACTIONS and CLOSE_ACTIONS out of Python source still goes through ast.literal_eval, because it only evaluates literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None, and Ellipsis [1]. Validation that reads code stays free of hidden code-execution gaps.
These two fences are cheap because they add zero infrastructure. What changed is the contract: if the validator is missing or the hash does not match, the pipeline must go red. Green has to mean something.