benchmark
Technical notes on web development, DevOps, and AI integration.
2 articles
- 21:07ai
The Most-Used Coding Model Is Not the Benchmark King
OpenCode's usage board is led by cheap open models, not benchmark kings. How I read usage, retention, and price before switching models.
TL;DR: Benchmark leaders like GPT-5.6 top SWE-Bench, yet deepseek-v4-flash leads OpenCode's real-world usage, suggesting cost, speed, and context length matter more than peak scores. The author recommends checking usage volume, momentum, and weekly retention as satisfaction signals. Treat official model lists as a starting point, then validate against usage data and a small trial in your own repo before committing.
#llm#benchmark#deepseek - 23:00ai
The Search Snippet Said 96%, the Board Said 79.2%
Search snippets claimed 95-96% on SWE-bench Verified. The official board says 79.2%. Four rules to keep benchmark numbers honest.
TL;DR: The author checked the actual SWE-bench Verified leaderboard and found the top score is 79.20%, not the 95%+ figures circulating in search snippets. The gap exposed how easily benchmark numbers get cited without opening primary sources. The fix: log access dates, trust only opened primaries, snippets lose conflicts, report tied scores as-is.
#ai-agents#swe-bench#benchmark