Artificial Analysis Releases Decision Index v2 With New Benchmarks and a Calibration-Aware Score

Introducing the Artificial Analysis Decision Index v2: a new proprietary evaluation, updated benchmarks, and a headline metric that rewards model calibration

We’ve released the second version of the Artificial Analysis Decision Index to better reflect real-world utility and guard against benchmark gaming. 45% of the Index is now built on private datasets, up from 25% in v1. Drex Max v1.1 from @NaceAI narrowly leads among decision models at 73, within 1 point of GLiDE v1.2 (@fastinoAI) and Mammoth 27B (@baselabs)

Updates in Decision Index v2:

• Decision Score: We adopt a new headline metric that rewards a model for both correctness and calibrated confidence. Each question scores from 0 to 100: right answers score above 50, wrong answers below 50, and confidence in its top answer moves the score within each half
• AA-Paperclip (30%): We added a new private benchmark of 1,500 documents including resumes, receipts, invoices, and AI and machine learning research papers. Models make decisions and extract facts from each document and are scored jointly as a single decision, correct only if all 15 answers are right
• AA-Symmetry (15%): This evaluates whether models change their answers for equivalent questions worded differently. We test 120 questions with variations such as reordered options, swapped labels, and negated yes/no questions, then measure how consistent the model’s probabilities are across those variations
• AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1 (15% each, down from 25%): AA-Omniscience-Decision now varies the wording of its “I don’t know” option across ten alternatives. HLE-MultiChoice adds an option to decline using the same ten phrasings

We’ve also added EvalSafe-ONET-AA (15%), our implementation of @typesafeai’s ONET benchmark. We have retained When2Call from @NVIDIAAI in the Index at a decreased weighting of 10% (down from 25%), and removed WildGuardTest given its high saturation

Key results:

• Three decision models sit within 1 point at the top of the Index: Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs both score 73, with Mammoth 27B from Base Labs scoring 72
• Cost per 1M Tasks varies by more than 60x across decision models: Celeris-1.1 Decision ($14), Decider v1.2 ($68) and Drex Max v1.1 ($102) sit on the Decision Index vs. Cost Pareto frontier, while Clef costs $859. Token price is not the only cost driver: most decision models, including Drex Max v1.1 and Decider v1.2, bill a document’s input once for all its questions, while GLiDE v1.2 and Decider Flash v1.2 bill it for every question. This means GLiDE v1.2 costs $780, nearly 8x higher than Drex Max v1.1 despite only 33% higher list prices
• Language models can score higher, alongside higher Cost and Time per Task: Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) scores 76 on Decision Index v2, putting them ~3 points ahead of Drex Max v1.1. GPT-6 Luna (Max) costs $1,530 per 1M Tasks vs. $102 for Drex Max v1.1 (15x the cost) and takes 11.34 seconds per task vs. 275 milliseconds (~40x slower)