Benchmark sources
What each source measures, how it updates and whether it contributes to ranking.
21Scoring this round69Reviewed sources and benchmarks
Ranking methodBroad benchmarks and experience
17Cross-capability tests and human blind preference.
- Counts toward rankingAA IndexReruns multiple current benchmarks in one environment to produce an overall capability score. Prices are reference information and do not affect the score.Artificial AnalysisEvidence budget30%
- Cross-referenceLiveBenchContinuously refreshed tasks with automatically verifiable answers test whether models hold up when the questions change.LiveBench
- Counts toward rankingArena TextPeople compare anonymous pairs of answers and choose which they prefer to read, adding overall experience that multiple-choice tests miss.LMArenaEvidence budget5%
- Counts toward rankingArena WebDevHuman blind preference tests whether websites are usable and ready to deliver.LMArenaEvidence budget2.4%
- Counts toward rankingLiveBench · Language and instruction followingAverage of Language + IF, weighted 50% each.LiveBenchEvidence budget3%
- Counts toward rankingArena · Creative writing blind preferencePeople choose between anonymous stories, poems and other creative writing to assess whether the work engages readers.LMArenaEvidence budget5%
- Under observationDesign Arena · Visual designHuman blind preference for website, interface, chart and slide designs.Design Arena / Intelligence
- Under observationEQ-Bench 4 · Emotional and interpersonal understandingUnderstanding emotions, implicit needs and different communication preferences in multi-turn conversations.EQ-Bench
- Under observationShort-Story · Short fictionNaturally incorporates specified characters, plot and style into a complete short story.Lech Mazur
- Under observationToneBench · Style and scriptsWrites video scripts in a specified author’s style, assessing openings, structure, expression and spoken pacing.Towards AI
- Under observationTubeLab · Video scriptsWrites scripts for different video channels, comparing story organization, style, pacing and clarity.TubeLab
- Under observationNC Bench · Author workflowHelps authors rewrite, continue, summarize, translate and organize creative ideas.Novelcrafter
- Under observationOpenVibeEval · Web designsCompares websites, accessibility and blind preferences to assess design performance with different tools.OpenVibeEval
- Under observationSVGBench · Vector graphics blind preferenceBlind choices between SVGs from identical prompts assess visual clarity and graphical expression.SVGBench
- Under observationRapidata SVGPeople compare model-generated vector graphics to evaluate visual preferences directly.Rapidata
- Counts toward rankingCreative Writing v3Short-form creative writing and expressionEQ-BenchEvidence budget3.6%
- Counts toward rankingLongform WritingCoherence and completeness of long-form storiesEQ-BenchEvidence budget2.4%
Coding
13From writing code to changing repositories: can the model build working software?
- SWE-bench Verified2024–2025Fixes bugs from GitHub issues and runs unit tests. Leading scores have converged, and many tasks have entered training data.
- DeepSWE2025–2026A fixed controlled system with harder, longer tasks. Retained as a cross-reference from v16.
- FrontierCodeFrom 2026 · PrimaryOpen-source maintainers write tasks and judge whether they would merge the PR. The primary coding benchmark from v16.
- Counts toward rankingFrontierCodeQuality, tests and scope of changes in real repository tasksCognitionEvidence budget4.8%
- Counts toward rankingDeepSWE v1.1Edits repositories and fixes defects in a common coding-assistant environment to compare the coding models themselves.DataCurveEvidence budget1.2%
- Counts toward rankingTapTap MakerBuilds working features in Lua inside a real game engine. Scores come from engine execution, rather than another model’s judgment.TapTap MakerEvidence budget2.4%
- Counts toward rankingLiveBench · Coding averageAverage of Coding + Agentic Coding, weighted 50% each.LiveBenchEvidence budget1.2%
- Under observationHyper-τ-benchBuilds and tests working customer-service agents from business documentation.Sierra
- Under observationSWE-rebenchContinuously collects real repository issues to compare software repair across programming languages.Nebius
- Under observationLHTBSustained terminal use to complete complex engineering and tool tasks over longer periods.Tencent HY / Lehigh et al.
- For reference onlyTerminal-Bench 4Retains raw terminal-task scores for models paired with different Agents for cross-checking.Harbor / Terminal-Bench
- For reference onlyMirrorCodeLong-running software reproductionEpoch AI
- Under observationCode MigrationSoftware migration and interface changesVals AI
- Under observationProgramBenchImplementing and delivering complete programsVals AI
- Under observationFrontierSWE v2Long-running engineering implementation and research tasksProximal Labs
- Under observationWeirdMLImplementing machine-learning solutions on unfamiliar dataWeirdML
Reasoning
11Mathematics, logic and unfamiliar rules: can the model reason through new problems?
- Near saturationCounts toward rankingLiveBench · Reasoning and mathematicsAverage of Reasoning + Mathematics, weighted 50% each.LiveBenchEvidence budget4.2%
- Under observationSimpleBenchEveryday common-sense and logic questions test whether models identify practical constraints.SimpleBench
- Counts toward rankingFrontierMath v2 · Tiers 1–3Research-level mathematical reasoningEpoch AIEvidence budget3.6%
- Counts toward rankingFrontierMath v2 · Tier 4More difficult mathematical research problemsEpoch AIEvidence budget1.2%
- Counts toward rankingChess PuzzlesFinding correct moves from text descriptions of chess positionsEpoch AIEvidence budget1.8%
- Counts toward rankingMystery Game PuzzlesDeriving correct actions from unfamiliar rulesEpoch AIEvidence budget1.2%
- Under observationMathArena · ArXivLeanVerifying mathematical proofs in LeanMathArena / ETH Zurich
- Under observationMathArena · BrokenArXivFinding and fixing errors in mathematical proofsMathArena / ETH Zurich
- Under observationMathArena · ArXivMathNew problems drawn from recent mathematics papersMathArena / ETH Zurich
- Under observationARC-AGI-2Learning and generalizing from unfamiliar rulesARC Prize
- Under observationARC-AGI-3Learning and generalizing from unfamiliar rulesARC Prize
Knowledge
5Factual questions and graduate-level science test knowledge and answer accuracy.
- Under observationFACTS ParametricAnswers factual questions about the world without search tools to test the accuracy of a model’s own knowledge.Google DeepMind / Kaggle
- Under observationHumanity’s Last ExamDifficult academic knowledge and reasoning across disciplines.CAIS / Scale AI
- Under observationBullshitBench · False premisesDetects false premises in questions and checks whether models continue inventing answers from them.Peter Gostev / BullshitBench
- Counts toward rankingSimpleQA VerifiedFactual accuracy on short questionsEpoch AIEvidence budget4%
- Near saturationCounts toward rankingGPQA DiamondGraduate-level physics, chemistry and biology knowledgeEpoch AIEvidence budget2%
Professional work
19Financial analysis, legal advice and banking test whether professional tasks can be completed.
- Counts toward rankingAPEX-Agents 1.1Long tasks in investment banking, consulting and law test whether a model assistant can complete an entire job.MercorEvidence budget4%
- Counts toward rankingVals Finance AgentEveryday financial-analyst tasks assess work such as research reports and modeling.Vals AIEvidence budget3%
- Under observationOfficeQA ProRetrieves information, calculates and answers questions from large collections of real financial documents.Databricks
- Under observationHarvey Legal Agent BenchmarkProcesses legal material and delivers reviewable Word, Excel and slide files.Vals AI / Harvey
- Under observationMCP AtlasUses real MCP services to retrieve material and operate documents and databases, completing tasks across software.Scale AI
- Under observationFinanceBenchmarkCompletes financial pricing, capital calculations and risk-management tasks, verified by numerical and code execution results.FinanceBenchmark
- Under observationPRBench · LegalAnswers complex questions from real legal work, assessing professional judgment and reasoning quality.Scale AI
- Under observationPRBench · FinanceAnswers real financial decision questions, assessing professional judgment, derivations and risk explanations.Scale AI
- Under observationSpreadsheetBench 2Completes spreadsheet workflows including financial modeling, workbook repair and data visualization.SpreadsheetBench / Renmin University / AfterQuery
- Under observationVending-Bench 2Runs a simulated store for a year, handling purchasing, negotiation, inventory and pricing.Andon Labs
- Awaiting resultsτ³-BankingHandles banking in a common tool environment, testing rule comprehension, customer communication and task completion.SierraEvidence budget2%
- For reference onlyEBR-benchLearning new rules and using memory in long tasksEpoch AI
- Under observationExcel · EMBSpreadsheet editing and office work in ExcelVals AI
- Under observationLegal ResearchLegal research and argumentationVals AI
- Under observationTaxAgentBenchProfessional tax questionsVals AI
- Under observationMedScribeOrganizing and writing clinical notesVals AI
- Under observationBioMysteryBenchReasoning for biology researchVals AI
- Under observationAutomationBenchCompleting automated workflows across office softwareZapier
- Under observationTerminal-Bench ScienceCompleting scientific research tasks in a terminalHarbor
Visual understanding
2Image-comprehension evidence remains in the overall board. Understanding images and designing attractive work are different abilities.
Chinese and multilingual
2Complementary language evidence. English writing performance does not establish Chinese capability.
“Under observation” means data, run conditions or usage boundaries are still under review, so the source does not score. Near saturation means the top ten differ by less than three percentage points, giving half weight this round. Duplicate collection and display slices add no votes. Dataset overlap across benchmarks still requires ongoing review.