Artificial Analysis reports a wider open weights model gap on closed benchmarks

What happened

Ethan Mollick says Artificial Analysis’s move to closed benchmarks reveals a wider open weights model gap, while urging caution about the findings. Artificial Analysis reports that Kimi K3.1 Pro, whose weights are not yet released, scores 46 in v5, 12 points behind the leader versus a 3-point gap in v4.3. Among models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32, below Claude Haiku 5.5 (max). https://twitter.com/ArtificialAnlys/status/2115265171233120627

Read next

• More changes in today’s briefingMistral releases improved Mistral Large 4 open weights for coding and cybersecurity

The story so far

Moonshot AI released Kimi K3.1 Pro through its apps and API, but its weights remain unreleased, and Artificial Analysis’s revised Intelligence Index now places it ninth, behind eight closed models. The Decoder reported that version 5 assigns 60% of the index to private test sets to counter training on known questions. K3.1 Pro scored 46, trailing the leader by 12 points; under version 4.3, it ranked fourth with 55, three points behind Claude Opus 5.5. Ethan Mollick urged caution in interpreting the revised findings. Simon Willison reported Moonshot’s promise to release the weights by November 4, 2026. On Artificial Analysis’s separate Cyber Index, K3.1 Pro scored 64%, leading openly available models for agentic cyber defense while trailing trusted-access GPT-6 Sol (Daybreak Blue, max) at 69%. Artificial Analysis said K3.1 Pro encountered no safety refusals across the index’s three benchmarks. Its cost per task was $2.81, 15% higher than Kimi K3 despite 33% lower per-token prices. Willison reported standard API prices of $2 per million input tokens and $10 per million output tokens. Moonshot says the Highspeed variant runs two to three times faster; Willison reported that it costs twice as much, except for input token cache hits, which retain the standard price.