Last night I opened two acceptance-test transcripts, `runner-b4a-1.txt` and `runner-b4a-2.txt`. The first lines held a `{"error": "copy-mismatch", "attempts": 3}` retry note, then a wall of PASS lines from R1 through R70. Final score: 70/0/0. Seventy passed, zero failed, zero skipped. My first reaction was pure optimism: my agent pipeline is deterministic, this task is done. I was wrong, and the reason I was wrong is the interesting part.
A perfect score proves nothing about the pipeline. It proves the verification runner around it works the right way: verdicts come from re-assembled artifacts diffed on the kernel side, never from the agent grading its own homework. Two similar-looking things with very different consequences.
The temperature-zero illusion
The common habit: set temperature to 0 and treat the output as frozen. For language models, the field says otherwise. The Thinking Machines team sampled 1000 completions at temperature 0 with an identical prompt and got 80 unique outputs
[6]. All of them stayed identical through token 102, then branched right there. The root cause is not sampling; it is batch invariance. Serving infrastructure packs requests into batches whose size shifts with load, and the kernels do not guarantee bit-identical results for different positions in a batch.
A similar story comes from a Japanese AI-safety lab studying harnesses that use an LLM as a judge. Pinning temperature to 0 reduces pass/fail flips but does not eliminate them: across 690 API calls, 1 to 2 of 7 borderline items stayed non-reproducible
[7]. Two extra traps showed up along the way. A harness that forgets to set temperature lets the provider silently apply its default of 1.0. And on newer model generations the parameter is deprecated outright, so the old mitigation is history.
So my conclusion shrunk to one line: never let the agent judge its own work. The verdict has to come from something it cannot talk its way around.
What makes that score trustworthy
Which brings me back to the 70/0/0. Two pillars hold it up: isolation, and an external comparator.
Isolation is about giving every test a clean room. The canonical pattern looks like pytest's `tmp_path` fixture: each test function gets its own temporary directory
[1], so no state leaks between tests. I once lost an afternoon to a stale test cache squatting in a shared directory. Since then I treat isolation as a requirement, not a nice-to-have.
The comparator is even blunter: `git diff --no-index`, which compares two files on disk without needing a repository
[4]. No model gets asked for its opinion. PASS or FAIL is a byte-level comparison against committed golden files. If a golden file is missing, the test fails outright rather than registering a mere difference; the Syrupy snapshot library is built on exactly that principle
[5]. Golden updates never happen automatically either. They require an explicit flag, so every change to "the right answer" shows up in review.
You can see the pattern in the oracles from that transcript. R36 validated chunking: 4 chunks, with 2 table chunks re-emitting their row headers. R40 checked that paragraph joins stayed verbatim. R40b asserted that embedding requests carried the correct prefix. All deterministic artifacts, diffable down to the byte, re-assembled from kernel state rather than agent memory.
The model-evals world lands on the same shape. OpenAI calls building high-quality evals one of the most impactful things an LLM builder can do
[2], and their recipe separates two components: the test-data schema on one side, the graders on the other
[3]. The split itself is the point. Pass or fail gets decided by frozen data and criteria from before the run, not by a model's opinion after it.
That copy-mismatch line at the top of the transcript is a fitting reminder. Even copying output from one stage to the next missed three times before it stuck. If raw copying can fail quietly, an unverifiable "all green" can fail even more quietly.
Let the file diff do the talking. If a verdict can be regenerated from artifacts, it means something. If it can only be spoken by a model, it has not passed anything.
Sources
- pytest docs, tmp_path [1]
- OpenAI Evals, README [2]
- OpenAI platform docs, Working with evals [3]
- git-scm docs, git diff [4]
- Syrupy, README [5]
- Thinking Machines, Defeating Nondeterminism in LLM Inference [6]
- arXiv 2606.26185, LLM-as-judge reproducibility [7]