New Artificial Analysis index widens the gap between open and closed models

What happened

Artificial Analysis revised its Intelligence Index, widening the gap between open models and the leaderboard’s top from 3 to 12 points. In version 5, Kimi K3.1 Pro falls from fourth to ninth behind eight closed models, while Grok 4.7 loses 12 points. Private test sets now account for 60 percent of the index to counter training on known questions, though researcher Nathan Lambert also points to differences in reinforcement learning infrastructure.

Read next

• More changes in today’s briefingMistral releases improved Mistral Large 4 open weights for coding and cybersecurity

The story so far

Moonshot AI released Kimi K3.1 Pro through its apps and API, but its weights remain unreleased, and Artificial Analysis’s revised Intelligence Index now places it ninth, behind eight closed models. The Decoder reported that version 5 assigns 60% of the index to private test sets to counter training on known questions. K3.1 Pro scored 46, trailing the leader by 12 points; under version 4.3, it ranked fourth with 55, three points behind Claude Opus 5.5. Ethan Mollick urged caution in interpreting the revised findings. Simon Willison reported Moonshot’s promise to release the weights by November 4, 2026. On Artificial Analysis’s separate Cyber Index, K3.1 Pro scored 64%, leading openly available models for agentic cyber defense while trailing trusted-access GPT-6 Sol (Daybreak Blue, max) at 69%. Artificial Analysis said K3.1 Pro encountered no safety refusals across the index’s three benchmarks. Its cost per task was $2.81, 15% higher than Kimi K3 despite 33% lower per-token prices. Willison reported standard API prices of $2 per million input tokens and $10 per million output tokens. Moonshot says the Highspeed variant runs two to three times faster; Willison reported that it costs twice as much, except for input token cache hits, which retain the standard price.