Checksummed Evidence: Test Numbers Are Claims Until Verified
Raw verification output wrapped in a self-checksum makes agent numbers tamper-evident, while a fail-closed gate rejects slices without anchored review verdicts.
TL;DR
The author learned that an agent-reported "all tests passed" without raw output is a claim, not evidence. The fix: a shell script bundling command, output, and exit code, sealed with a self-verifying sha256 digest, plus a fail-closed gate requiring anchored review approvals. Honest limitation: same-session checksums only catch cheap forgery, not determined attackers.
The terminal glowed green: "All tests passed, coverage 42.0%." The coding agent reported it confidently, complete with a list of files it called done. I believed it. The feature looked right in the UI at a glance, so why be suspicious?
Doubt arrived later, when I tried to reproduce the number. No execution log was left behind. All I had was an agent-written summary claiming success. At that point it landed: a number without raw output is not evidence, it is a claim.
Evidence Bundles That Checksum Themselves
The fix is a 144-line shell script in the KotaPortal work repo (project worklog). Its first mode captures evidence: run the verification command, write the command line plus the full raw output plus the exit code into one per-slice bundle file, then close it with a single sha256: <digest> line. The digest is computed over the file body without the checksum line itself.
An evidence-check command recomputes the digest and compares. Someone edits a number in the bundle? The match breaks. The selftest exercises exactly this: one coverage figure changed from 42.0% to 99.9%, and the check must fail (project worklog). The pattern is the same one sha256sum has offered for decades, "print or check SHA256 (256-bit) checksums", with -c to verify [6]; the difference is that the checksum lives attached to the bundle body itself.
The Fail-Closed Done Gate
A checksum alone is not enough if the review step can be skipped. The script's second mode is a slice completion gate: a slice may only be marked done if brief, report, and review files all exist. The review must carry two verdicts that both read PASS, one for spec compliance and one for task quality. Any FAIL or BLOCKED anywhere fails everything.
The most important detail is how approval is parsed. The final verdict line is anchored: it must start with Overall: followed by approve. The word approve buried mid-paragraph does not pass. The gate also requires a raw-output block (lines starting with $ ) and a finding-count line, Finding counts: Critical n, Major n. Any Critical or Major finding needs an explicit resolved, fixed, or parked note. The word unresolved acts as an automatic veto.
None of this was designed on paper. The first version of the gate received REQUEST-CHANGES plus medium security findings from dual review (project worklog). The fix round produced three things: an anchored approve check so prose cannot sneak through, the unresolved veto, and a checksum trigger that fires when the bundle exists or the report mentions coverage. Selftest cases grew from 13 to 16.
The Honest Limit
A checksum computed and stored in the same file by the same session only catches cheap forgery: accidental edits, lazily retyped numbers, summaries rewritten by hand. A determined attacker simply recomputes the hash. The honest claim is tamper-evidence for low-effort forgery, not tamper-proofing. The upgrade path is clear: cryptographic signatures or external anchoring, in line with in-toto's notion of attestation, authenticated metadata about artifacts consumed by automated policy engines, with cryptographic digests recommended for immutability [7].
OWASP files assumptions made without verifying integrity under A08, and one of its mitigations reads like this post's thesis: verify the data is from the expected source and has not been altered [5]. Having the tamper test in the selftest from day one, including the checksum-mismatch case, is the same posture [5].
One tradeoff I left in deliberately: the checksum trigger is advisory, firing when a bundle exists or the report mentions coverage. A report with neither passes unchecked. If bypassing becomes a habit, a structured report format is the next upgrade.
For anyone copying this: start from the selftest, not the gate. The cases "valid checksum passes", "tamper is detected", "fake verdict is rejected" are the specification in executable form. A fail-closed gate without a selftest is just a confident script, and we already know how much a confident script without evidence is worth.