evaluation

A 70/0/0 acceptance run made me think my agent pipeline was deterministic. It was the runner that deserved the credit; temperature 0 was no guarantee.

TL;DR: A flawless 70/0/0 acceptance run looked like proof of a deterministic agent pipeline, but it really validated the harness, not the model. Temperature zero doesn't guarantee reproducibility either; batch invariance and LLM-as-judge studies show outputs and verdicts still flip. Trustworthy results come from test isolation plus byte-level diffs against golden files, never from an agent grading its own work.

Search snippets claimed 95-96% on SWE-bench Verified. The official board says 79.2%. Four rules to keep benchmark numbers honest.

TL;DR: The author checked the actual SWE-bench Verified leaderboard and found the top score is 79.20%, not the 95%+ figures circulating in search snippets. The gap exposed how easily benchmark numbers get cited without opening primary sources. The fix: log access dates, trust only opened primaries, snippets lose conflicts, report tied scores as-is.