Skip to content
HomeAboutActivityPortfolioBlogAsk AI

The Validator That Cried Wolf: Measuring Precision in Data Fact Checking

Adityo Guni Waluyo

A data validator flagging false positives isn't just noisy — it destroys developer trust. Here is how I added a negative-control corpus and exact reason-code assertions to turn a paranoid validator into a precise one.

#testing#bash#quality-assurance

TL;DR

The validator flagged a blank source as unverified because it treated missing facts as failures rather than negative controls. A new 20-sentence negative-control corpus now fails the test if false positives exceed 10 percent. Seven exact reject reason-codes plus a positive control ensure bad data gets rejected for the right reasons.

I ran ingest-blueprint-selftest.sh on a quiet Tuesday morning, expecting a clean green build. Instead, the terminal spat out a wall of warnings: WARN data-fact-unverified. Twenty corpus sentences were flagged, all citing src001.

Here is the catch. src001 is a blank slate. It intentionally contains absolutely none of the corpus fact tokens. The validator was accusing the system of failing to verify facts that were never there in the first place.

My first guess was a sloppy regex matching bug. I figured the pattern matcher was getting confused by empty strings or edge-case whitespace. I spent an hour tracing the regular expressions, only to find they were perfectly fine.

The real issue was a precision problem. The validator treated the absence of a fact as an unverified fact, rather than recognizing it as a negative control. It lacked the nuance to say, "This source has nothing to verify, so passing is the correct behavior."

Building a Negative Control

I had to teach the validator how to handle silence. In commit c690354, I introduced a 20-sentence negative-control corpus under rule R80b. Each sentence runs as a standalone data page citing src001.

The test now explicitly counts the WARN data-fact-unverified flags. If false positives exceed 2 out of 20 (more than 10%), the entire test fails hard. This forces the system to prove it can ignore empty noise.

Using negative controls is a proven method to estimate bias and variance when no effect should be observed. When unexpected associations spike, it reveals systematic bias in the dataset [2].

Hardening the Reject Logic

Catching the false positives was only half the battle. I also needed to ensure the validator rejected actual bad data for the right reasons.

Under rule R97, I defined 7 exact reject sub-cases with strict reason-code assertions: digest-mismatch, stale-acc-content, quote-too-short, hash-not-in-quote, warn-undisposed, stage-not-final, and render-drift.

I also added one positive control ensuring a valid audit exits with a 0 status code. We now pull the canonical 64-hex digest directly from the check output field instead of recomputing it. This eliminates redundant processing and guarantees we are evaluating the exact payload the system produced.

Standardizing these error codes aligns with RFC 9457, which advocates for machine-readable problem details where a type URI serves as the primary identifier [3].

Precision is not just about catching bad data. It is about not punishing good data. As scikit-learn notes, precision measures a classifier's ability to not label a negative sample as positive [1]. Just like CDC laboratory quality assurance programs improve measurement reliability [4], adding strict negative controls to our ingest pipeline stops the validator from crying wolf.

I strongly believe that a validator throwing false positives is worse than one that is slightly lenient. A noisy validator gets ignored by developers. A quiet, precise one gets trusted.

Sources

Related articles

testing

The 14/19 Selftest and the Fixture Defect

When a selftest scores 14/19, the first instinct is to blame the validator. The real cause — broken bash fixture transforms — is a reminder that a reliable failure is more honest than a rigged green.

testing

Fake implementations must fail: the selftest that tests itself

A green suite proves nothing if a fake can pass it. I fed three fake implementations into my 32-test selftest to prove the grader works.

testing

Freezing a CLI contract with 32 red tests

How commit 6f5ae82 turned a deliberately failing selftest into a frozen CLI contract: red tests first, implementation later, no silent drift.

ai

Hidden instructions in media files: prove they stay data

The transcript must carry the hijack sentence verbatim as data. That is exactly how you prove a media file never acted as an instruction.

ai

Golden-File Verification Runners: No Self-Grading

A 70/0/0 acceptance run made me think my agent pipeline was deterministic. It was the runner that deserved the credit; temperature 0 was no guarantee.

backend

Lock stolen mid-run: what fencing tokens taught me

A lock checked once at startup guards the door, not the data. Fencing tokens validated at every state write, plus a heartbeat lease, are what guard the data.

frontend

TypeScript Types Never Reach Your JSON, So Write Contract Tests

One missing badge image broke typecheck. A story about type erasure, optional properties, and parametrized contract tests guarding per-locale JSON content.

devops

Verify Said 0 Fail. Bugs Still Shipped.

Auditing my blog article pipeline: a chopped mermaid token, paragraphs that shattered despite clean verify, and three gate gaps found only by a human question.