AI benchmarks typically score just one model on a single run, but a new study testing 21 LLMs across 16 benchmarks spanning coding, reasoning, medicine and agentic tasks found that correcting for both single-model and single-run bias reveals real achievable performance is 82% better than standard single-model benchmark scores

A study published on arXiv by Fowler et al. examined 21 large language models (LLMs) across 16 benchmarks covering tasks like coding, reasoning, and medicine. The study introduced the "Capability Frontier" to correct for underestimation from single-model evaluation and overestimation from noisy samples. It found that correcting for both single-model and single-run biases reveals an 82% improvement in achievable performance compared to standard single-model benchmark scores.