A new arXiv benchmark of long software engineering tasks, each a chain of dependent steps on a single codebase, found that across nine frontier models the strongest reached a solve rate of only 18.5%, and 38 of its 69 tasks were solved by no model
The solve rate of 18.5% and the fact that 38 of 69 tasks were not solved by any model were measured by Xiangkai Zeng, Zheng Liu, and Zihao Lin in their paper "Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?" submitted on 4 Nov 2026 to arXiv:2611.07166. These numbers represent the performance of the strongest of nine frontier models on the Multi-step SWE benchmark, which consists of 69 software engineering tasks.
Verified 2026-11-09
Track with InsiderWhy it matters
The low success rate on multi-step engineering tasks reveals that even frontier models still struggle with the kind of sustained reasoning real software projects demand.
More on Models
- AI models can now win gold at the International Mathematical Olympiad, yet Stanford's 2026 AI Index found they still fumble a task most children master early. Across ClockBench's 180 clock designs and 720 questions, the top model, GPT-5.4 High, read analog clocks correctly only 50.6% of the time, versus 90.1% for humans
- After briefly nearly closing in 2024, the performance gap between the best closed-weight and open-weight AI models widened again on the Arena leaderboard. As of March 2026, Stanford's 2026 AI Index finds the top closed model leads the top open model by 3.3%, up from a 0.5% lead in August 2024
- Stanford's 2026 AI Index tracks how AI models perform on GPQA Diamond, a set of PhD-level chemistry and physics questions designed to resist Google searches. Mean model accuracy on the benchmark reached 93% in 2025, compared with an expert human validator baseline of 81.2% -- AI models now beat PhD-level experts by 12 points on GPQA Diamond
FREE · NO CARD
Facts like this, six days a week
The Daily Receipts on weekdays, The Weekly Brief on Saturday — six sourced facts a day by email, two more than the free site shows. Every one traced to the filing it came from. Free.
Get the free daily briefFree · No card · Unsubscribe anytime
Join now and you start with 100 Receipts on The Receipts Ledger — your first step toward a free year of Insider or Pro. How it works →
See today's issue →