Skip to content
K = 0.731
Back to model frontier

Ranking method

Explore rankings, evidence and methods across public benchmarks.

0—100

Shared evidence, clear rankings.

Public benchmarks are combined to minimize conflicts with known results. The overall board and four category boards show up to 30 models each.

The consensus index maps differences in ranking support to 0–100 for comparison. It is neither accuracy nor a percentage gap in ability. Close support can produce ties; the full evidence still determines rank.

  1. 01

    Identify the same model

    Names and versions are normalized across leaderboards. One representative reasoning configuration is selected by rules set in advance, rather than choosing the highest score. Anonymous codenames and results mixing other models are excluded. Publicly released Preview versions are eligible.

  2. 02

    Compare models with real results

    Only real results within the same benchmark are compared. Where both models have public uncertainty estimates, small differences within that uncertainty are treated closer to ties. Missing measurements provide no comparison.

  3. 03

    Set vote budgets, then combine evidence

    Each benchmark has a predetermined vote budget. Overall eligibility requires at least three organizations, three evidence families and three specialist areas. Duplicate collection or display slices do not create additional vote weight.

  4. 04

    Find the ranking with the fewest conflicts

    Benchmarks may disagree. We select the order that minimizes violations of their net vote support, then compute the display index separately. Missing results are neither zero-filled nor inferred.

A BALANCED VIEW

Overall tests, human preferences and specialized benchmarks.

Overall benchmarks receive 30%, human blind preferences 10%, and specialist benchmarks 60%. New benchmarks are checked for overlap and funded from their relevant share. Missing shares are not redistributed.

  • Broad benchmarks30%AA Index
  • Human blind preference10%Arena Text · Arena · Creative writing blind preference
  • Coding and design12%Arena WebDev · FrontierCode · DeepSWE v1.1 · TapTap Maker · LiveBench · Coding average
  • Writing and expression9%LiveBench · Language and instruction following · Creative Writing v3 · Longform Writing
  • Mathematics and reasoning12%LiveBench · Reasoning and mathematics · FrontierMath v2 · Tiers 1–3 · FrontierMath v2 · Tier 4 · Chess Puzzles · Mystery Game Puzzles
  • Knowledge and facts6%SimpleQA Verified · GPQA Diamond
  • Visual understanding6%Arena Vision
  • Tools and office work6%APEX-Agents 1.1 · τ³-Banking
  • Chinese and multilingual6%AA · Chinese and multilingual
  • Industry-specific tasks3%Vals Finance Agent

Coding, reasoning, knowledge and professional work use separate real benchmarks and independently computed category indices. Vision and multilingual evidence remain in the overall board; renaming categories adds no votes. Overall scores are not arithmetic means of category scores. Categories generally need two valid benchmarks and five comparable models. Knowledge currently uses two Epoch benchmarks, with the single-organization limitation stated. Creative preference and web development remain overall evidence, without separate aesthetics or writing boards in this version.

BENCHMARKS AGE

Benchmarks age; primary evidence changes.

A new benchmark can distinguish models, but eventually leading scores converge and questions enter training data. Coding has progressed from SWE-bench Verified to DeepSWE and now FrontierCode, authored by open-source maintainers. Once new benchmark data is stable and run conditions are public, we shift vote share to it, while older benchmarks become cross-references instead of retaining full vote weight.

From method v16, the 12% coding budget is allocated as follows: FrontierCode 4.8%, TapTap Maker 2.4%, Arena WebDev 2.4%, DeepSWE 1.2%, LiveBench coding 1.2%. Changing primary evidence increments the method version and is documented here.

Each round also checks saturation automatically. For bounded scores such as pass rates and percentages, a top-ten spread below three percentage points halves the benchmark's weight and flags it near saturation. Unbounded scores such as Elo are excluded. Removed weight is not redistributed.

You may also want to know

Why can a model rank with fewer benchmarks?
Benchmark count is not capability. The overall board requires three organizations and cross-capability evidence. Most categories require two organizations; knowledge uses two benchmarks from one organization, with that limitation stated. Missing results are not zero, but can still introduce bias.
What does “evidence-sensitive” mean?
Evidence-sensitive status is flagged if removing a benchmark or organization, changing a weight by 20%, or altering error treatment yields a rank range of three or more, loses eligibility, or leaves comparisons incomplete. Scenario ranges are not 95% confidence intervals and exclude unknown results.
Does a higher-ranked model always win pairwise comparisons?
No. A may beat B, B beat C, and C beat A. The full ranking balances these conflicts, so nonadjacent ranks can differ from pairwise comparisons. Mathematical optimality means minimum conflict under the current rules, not proof of capability order in every real task.
Why might rankings differ from my experience?
The board combines public benchmarks and reflects their evidence. Real experience also depends on product version, reasoning configuration, tools and long-task reliability. New benchmarks test old rankings; one or two places between close scores do not necessarily indicate a meaningful gap.
When will a new model appear?
e/acc checks upstream results four times a day. A new model can rank only after an organization publishes its results. Page refreshes, price updates and our collection timestamps do not count as new evaluations.
What if a benchmark source is temporarily unavailable?
Valid source snapshots remain usable. Within the same benchmark version, temporarily missing rows can reuse verified records from the last seven days. Valid public results are not deleted because their values have stayed unchanged. Official withdrawals, corrections and version changes invalidate or replace records; historical peak scores are not retained. A failed computation preserves the prior valid board and its original timestamp.
Can overall indices double-count specialized benchmarks?
Overlap is checked against public datasets and methods, and correlated sources have a capped combined share. Unverified correlations remain a limitation. The AA index receives 30%; Arena overall text and creative preference share the original 10% human-preference budget at 5% each. Their overlapping votes count as one evidence family.
Why is a benchmark marked “near saturation”?
Its top ten scores differ by less than three percentage points, limiting its ability to distinguish strong models. It receives half weight this round, automatically restored if scores spread out again. This flag describes score distribution, not benchmark quality.
Do price or speed affect ranking?
Neither affects ranking. Prices indicate API usage cost per million tokens and link to vendor sources. They do not represent subscription fees.
View calculation details and current version

Method version: 2026.10-public-consensus-v16. Weighted incomplete Kemeny ranking combines net support M for pairs with shared evidence and minimizes reversed net support. Integer optimization returns optimality status and objective bounds. Only fully verified results are published.

When both models have explicit standard errors, net support is 2Φ(score difference / combined standard error) − 1, assuming zero covariance. Other comparisons use only the raw lead direction. Unknown error is not zero error; ordinal treatment of small differences remains a limitation. Budgets apply to potential model pairs, so sources covering more models use more comparison positions. Nominal budgets are not exact contributions to final rank.

The display index preserves the original order. Each adjacent pair is reversed while other models may reorder, measuring the smallest added reverse support. Nonnegative support gaps accumulate along the original ranking and are mapped by a sigmoid to 0–100 against a fixed reference group. Equally optimal alternatives retain zero gaps, with one decimal displayed and no artificial minimum. The index does not determine ranking; eligible models, references and evidence can change its scale. Gaps are not real capability distances. Equal-cost solutions use fixed model-ID order. Integer optimization uses HiGHS (highs 1.15.3), with normal-CDF error conversion matching SciPy norm.cdf. Sources, protocols, eligibility and computation inputs are versioned. Disconnected evidence components are not given a falsely precise global order.

Fixed reference models: claude-fable-5, gpt-5-6-sol, kimi-k-3, qwen-3-8-max, gpt-5-4, claude-opus-4-8, claude-sonnet-5, grok-4-5, gemini-3-5-flash, glm-5-2, claude-sonnet-4-6, qwen-3-7-max, kimi-k-2-6, deepseek-v-4-pro, qwen-3-6-plus, minimax-m-3, gpt-5-4-mini, grok-4-3. References define the index scale, not a preferred vendor rank. Indices across categories are not directly comparable.

Explore all benchmark evidence →