The Search Snippet Said 96%, the Board Said 79.2%

I opened swebench.com in a browser tab expecting confirmation. Search snippets for "SWE-bench Verified leaderboard" had been telling me the top score was 95%, some said 96%. I'd used those numbers already: cited them in research notes, believed them without checking the source directly. The page loaded and I parsed the leaderboard table row by row.

79.20%. The highest score. Twice.

Not 95. Not 96. Seventy-nine point two, held by Sonar Foundation Agent and live-SWE-agent, both running Claude 4.5 Opus, both dated December 2025 [8]. My first thought: the search snippets must be from a newer version of the board. Maybe swebench.com hadn't been updated since December. I refreshed the page, checked the page source, looked at the dates next to every single entry. Every row had a timestamp. The board wasn't a video that was behind, it was a photo. Each entry was a discrete submission with a date. The snippets weren't fresher than the board. The snippets were wrong, or referencing a different benchmark variant entirely.

That's when the real problem landed: I had been consuming benchmark numbers from search result summaries without opening the primary source.

The Real Numbers

The SWE-bench Verified board, parsed directly, tells a different story than what circulates in blog posts and AI tweets [8]. The leaderboard has a "Verified" subset of 500 human-filtered instances, validated by OpenAI when they announced the benchmark in late 2024 [11]. The top entries cluster around 79%, not mid-90s. The 95% figures floating around likely conflate different SWE-bench variants or cherry-pick single-run results from non-standard evaluation setups.

The entry that caught my attention was mini-SWE-agent: 76.80%, about $0.75 per run, entry dated February 17, 2026 [8]. I opened its README on GitHub and found the vendor's own claim was ">74%", more conservative than the leaderboard number, which is honest in a way most benchmark claims aren't [9]. The agent has no tools other than bash. It's roughly 100 lines of Python. The README asks: "What if our agent was 100x simpler, and still worked nearly as well?"

That framing stuck with me. Here's an agent that sits within 2.4 percentage points of the absolute top on the most rigorous SWE-bench variant, costs seventy-five cents per run, and does it with nothing but a shell. Meanwhile, the complex multi-tool workflows with ten integrated services don't even appear on this specific board.

I initially wrote in my research archive that mini-SWE-agent was "the only non-Claude agent in the top 5." I was wrong. Looking at the raw data I'd already scraped: TRAE (Doubao-Seed-Code) sits at 78.80%, also not Claude, and live-SWE-agent+Gemini scores 77.4%, also not Claude [8]. The corrected data goes into the archive. The wrong claim stays in this article because the mistake is the point.

Four Rules

That verification incident became the basis of a hard protocol. Every time I pull benchmark numbers for research, four rules apply, no exceptions.

Record the access date. Benchboards change. A number from January might not match March. If you don't log when you looked, the number is untethered.

Numbers only from opened primaries. If you haven't loaded the actual page and read the table yourself, the number doesn't exist. Search snippets, tweet screenshots, blog citations — none of these count. Open the source.

Snippets lose on conflict. When the search summary says one thing and the primary says another, the primary wins. Always. The snippet was generated by a model that may have hallucinated, aggregated across variants, or simply parsed the page wrong.

Report tied scores as-is. Two entries at 79.20% means two entries at 79.20%. Don't pick one and call it the winner. Don't fudge the tie.

These rules sound obvious until you realize how rarely they're followed, including by me, before this incident. The convenience of snippet numbers is real. They arrive in your context window without effort. But a benchmark number you haven't verified is a claim you're borrowing from a machine that might be wrong.

Anthropic's guidance on building effective agents distinguishes between workflows (structured, predetermined paths) and agents (model-driven, autonomous loops) [10]. Their advice applies beyond agent architecture: "find the simplest solution possible, and only increasing complexity when needed." mini-SWE-agent is the embodiment of that, a bash-only agent outperforming elaborate multi-tool setups on the hardest benchmark subset.

I packaged this entire research arc, including the verification protocol and the broader skills-analysis methodology, into a structured archive in the AI agent skills research stack article. The commit that finalized it (b04b594) converted raw research notes into a reusable three-chapter skill. But the protocol is the part that travels beyond this specific research. Any benchmark number you cite should survive these four rules. Most won't.

Sources:

• SWE-bench Leaderboards [8]
• mini-swe-agent README (raw) [9]
• Anthropic, Building Effective Agents [10]
• OpenAI, Introducing SWE-bench Verified [11]