testing

A horizontal rule that vanished without a trace exposed a healthy two-way converter design: canonical push, tolerant pull that still refuses loudly.

TL;DR: My horizontal rule vanished on pull because the parser only handled <hr /> and silently ignored plain <hr>. The fix now throws a clear error for the bare tag instead of quietly dropping it. The tool now pushes conservative canonical HTML but only tolerates its known dialect on pull, backed by round-trip property tests.

The audit re-hashed after render ran, and the selftest failed. Derived artifacts must stay out of the canonical digest, and execution order defines what a hash means.

TL;DR: R97 failed because the audit hashed the canonical digest after render ran, and render's output mutated the very directory being hashed. Fix: hash only sources of truth, excluding derived artifacts like nav.json and pages.json, with render now check-first and stdout silenced. Flaky fixture string substitution was replaced with path-based mutation and a re-frozen checksum.

My automation browser profile leaked an id-ID locale, the UI rendered Tanya Qwen instead of Ask Qwen, and every exact-label selector quietly broke. Here is the two-layer fix.

TL;DR: An automation suite broke because the browser profile's id-ID locale made the target site render UI text in Indonesian, so exact-match selectors like "Ask Qwen" failed. The fix: regex alternation accepting both languages, plus pinning the locale via context emulation. A small assert-based test guards against regressions.

When a selftest scores 14/19, the first instinct is to blame the validator. The real cause — broken bash fixture transforms — is a reminder that a reliable failure is more honest than a rigged green.

TL;DR: After pushing commit bc77087, the selftest scored 14/19, and I initially blamed my new validator logic. The real culprit was defective bash fixture transforms producing invalid JSON, so the validator correctly rejected corrupt input. I froze the suite at 14/19 with documented defects rather than patch the fixture, because honest reliability beats inflated vanity metrics.

A data validator flagging false positives isn't just noisy — it destroys developer trust. Here is how I added a negative-control corpus and exact reason-code assertions to turn a paranoid validator into a precise one.

TL;DR: The validator flagged a blank source as unverified because it treated missing facts as failures rather than negative controls. A new 20-sentence negative-control corpus now fails the test if false positives exceed 10 percent. Seven exact reject reason-codes plus a positive control ensure bad data gets rejected for the right reasons.

A green suite proves nothing if a fake can pass it. I fed three fake implementations into my 32-test selftest to prove the grader works.

TL;DR: After my 32-test suite went green, I worried it might be vacuous, so I built three fake validators (always pass, always fail, always emit every error code) that must each score zero passes. This is manual mutation testing, a known technique. I also pinned the selftest's sha256 so silent grader edits get caught.

How commit 6f5ae82 turned a deliberately failing selftest into a frozen CLI contract: red tests first, implementation later, no silent drift.

TL;DR: The author's selftest showed 32 failures, all intentional: that's the red step of TDD, freezing the CLI contract before any implementation exists. It's a batch-freeze variant rather than Canon TDD, accepting rework risk since a stable spec beats iteration rhythm for a CLI. Synthetic fixtures keep tests honest, and next comes implementing until green.

The transcript must carry the hijack sentence verbatim as data. That is exactly how you prove a media file never acted as an instruction.

TL;DR: The author deliberately embedded a prompt-injection sentence into an audio fixture to test a media pipeline's defenses. Counterintuitively, the test requires the hostile transcript to stay verbatim and labeled untrusted, since injection is about text switching from data to instructions. The real defense is a contract proving content never becomes commands, not filtering text or trusting the model's intelligence.

A 70/0/0 acceptance run made me think my agent pipeline was deterministic. It was the runner that deserved the credit; temperature 0 was no guarantee.

TL;DR: A flawless 70/0/0 acceptance run looked like proof of a deterministic agent pipeline, but it really validated the harness, not the model. Temperature zero doesn't guarantee reproducibility either; batch invariance and LLM-as-judge studies show outputs and verdicts still flip. Trustworthy results come from test isolation plus byte-level diffs against golden files, never from an agent grading its own work.