#Benchmarks

Artificial Analysis introduces Decision Index to benchmark decision models against language models

Artificial Analysis introduced Decision Index v1.0 to compare decision models’ capability, cost, and speed across classification, tool calling, knowledge, and reasoning. Jev 1.13 leads decision models with 58, while Claude Haiku 5.5 (Low) scores 61 but takes 1.4 seconds per task versus 0.13 seconds and costs $184 per 1M Tasks versus $42. Leading decision models perform comparably to language models on classification and tool calling but trail on knowledge and reasoning, where their answer probabilities generally exceed their accuracy.

MagicLab Studio’s EchoPolicy-0 tops MolmoSpaces Combined with 75.4% average success

QbitAI reports that MagicLab Studio’s EchoPolicy-0 ranked first on MolmoSpaces Combined as of October 13, 2026, achieving 75.4% average success across all four v1 manipulation tasks in 3915 evaluation episodes. Its 3.4B-parameter MagicBrain ranked fourth with 72.4% average success and 94.4% grasping success. EchoPolicy-0 combines VLM-guided candidate trajectory selection with execution checks, while MagicBrain uses visual markers and post-training to connect target locations with actions without changing the VLA architecture.

Alexandr Wang Reacts to @lillyguisnet’s Image-Pointing Benchmark

@lillyguisnet posted: “New sota for my image pointing.” Alexandr Wang: now this is the kind of benchmarking i can get behind

Alexandr Wang urges followers to “fill your soul with wonder”

fill your soul with wonder

Artificial Analysis launches AA-3D to benchmark AI models building interactive 3D scenes

Artificial Analysis launched AA-3D, a benchmark in which AI agents turn 55 written briefs across seven domains into interactive three.js scenes for blind human comparisons. Over 10,000 counted votes from over 1,000 panelists rank 21 models by Elo with 95% confidence intervals; Claude Sonnet 5.5, Claude Opus 5.5, GPT-6 Astra, and Claude Fable 5.1 are statistically tied at the top. Models receive up to three hours and 250 turns, and reported costs range from $0.20 to $32.08 per scene.

People take the Artificial Analysis Index and its individual scores way too seriously.

People take the Artificial Analysis Index and its individual scores way too seriously.

Alexandr Wang Reacts to @theo’s S-Tier Model Ranking

@theo ranked Opus 5.5 and Muse Wonder as S tier. Alexandr Wang said, “wow, S tier from theo! not on my bingo card”

Meta releases Muse Wonder models with gains in coding and agent benchmarks

Meta released Muse Wonder and Muse Wonder Full, scoring 52 and 56 on the Artificial Analysis Intelligence Index v4.3, versus 48 for Muse Spark 1.3. Both support a one-million-token context window and text, image, video, and audio inputs, with stronger coding and agent results but weaker factual knowledge than leading competitors. Muse Wonder is available through Meta's API and Muse Code at higher prices than its predecessor; Muse Wonder Full is currently limited to Muse Code users.

The same story is shown in the picks as 《Meta releases frontier models Muse Wonder and Muse Wonder Full》

Aravind Srinivas Congratulates Meta on Training a Frontier-Grade Model

Congrats to Meta for training a frontier-grade model! Watermelon compares favorably to Fable 5.1 on WANDR evals. https://twitter.com/alexandr_wang/status/2109689491300794551

Alexandr Wang posts “Muse Wonder”

🍉 MUSE WONDER 🍉 @LexnLin says Muse Wonder is out and ranks second globally according to Artificial Analysis, scoring 56 compared with 58 for Opus 5.5. Alexandr Wang’s post reads: “🍉 MUSE WONDER 🍉”

Meta releases Muse Wonder for developers as consumer access remains unclear

Baoyu reports that Meta released Muse Wonder and Muse Wonder Full on October 12, with Muse Wonder available today through Meta Model API and Muse Code, while Full is limited to Muse Code subscribers. Meta’s blog says consumer access is coming soon, whereas Zuckerberg says subscribers can use it today. The models support a 1 million token context window and multimodal input with text output. Meta reports strengths in sustained agentic work, coding, and memory, but its comparisons include internal benchmarks and competitor runs that may not be optimally configured.

The same story is shown in the picks as 《Meta releases frontier models Muse Wonder and Muse Wonder Full》

Artificial Analysis launches FrontierCode 1.1 leaderboard measuring maintainer-ready coding changes

Artificial Analysis launched its FrontierCode 1.1 leaderboard, evaluating models on Cognition’s private coding benchmark using the 100-task Main set and its Stirrup open source reference harness. Grading combines unit tests with maintainer-oriented criteria; failing any blocking check yields zero. Claude Opus 5.5 (max with fallback) leads at 52%, with blockers passed in 58% of attempts; MiMo-V2.6-Pro leads open weights models at 45%, while GPT-6.1 Sol (max) scores 49% at $0.81 per task, matching Claude Fable 5.1 (max with fallback).

Alexandr Wang Says Muse Wonder Outperformed Astra in Physics Testing

“[Muse Wonder] definitely mogged Astra in my hardcore physics testing.” — the new beff times

Meta releases Muse Wonder and Muse Wonder Full with higher benchmark scores

Meta has released Muse Wonder and Muse Wonder Full, according to Artificial Analysis in the post quoted by Alexandr Wang. At max reasoning effort, they score 52 and 56 on the Artificial Analysis Intelligence Index v4.3, compared with Muse Spark 1.3’s 48, following a release gap of 6 weeks. Muse Wonder is available through Meta’s first-party API and Muse Code, while Muse Wonder Full is limited to Muse Code users; both have 1M-token context windows.

If you announce a model's benchmarks without releasing it (weights, product, api, etc "later"), feels like almost a sure thing that someone releases a better model in your category before you get it out.

If you announce a model's benchmarks without releasing it (weights, product, api, etc "later"), feels like almost a sure thing that someone releases a better model in your category before you get it out.

Meta launches Muse Wonder with developer access and conflicting consumer availability statements

Meta released Muse Wonder and Muse Wonder Full on October 12, according to Baoyu’s account of the announcement. Developers can access Muse Wonder through Meta Model API or Muse Code, while Full is available to Muse Code subscribers; Meta’s blog and Zuckerberg’s X post differ on consumer availability. The models support a 1 million Token context window and multimodal input with text output, while Meta’s reported benchmarks show strengths in sustained agentic and coding tasks alongside weaker results elsewhere.

The same story is shown in the picks as 《Meta releases frontier models Muse Wonder and Muse Wonder Full》

Meta releases Muse Wonder and Muse Wonder Full with higher Artificial Analysis scores

Meta has released Muse Wonder and Muse Wonder Full, which Artificial Analysis reports score 52 and 56 on its Intelligence Index v4.3 at max reasoning effort, versus 48 for Muse Spark 1.3 (max). Muse Wonder Full improves across all ten evaluations, with the largest gains in agentic evaluations. Muse Wonder is available through Meta’s first-party API and Muse Code at launch, while Muse Wonder Full is limited to Muse Code users; Muse Wonder costs $4.52 per Intelligence Index task.

The same story is shown in the picks as 《Meta releases Muse Wonder and Muse Wonder Full with higher benchmark scores》

Le Chonk cooked on visual grounding. Pinpoints exactly where things are in an image. 42% on Dense 200, ahead of even frontier closed models..

Le Chonk cooked on visual grounding. Pinpoints exactly where things are in an image. 42% on Dense 200, ahead of even frontier closed models..

Import AI 476: New Benchmarks Test Visual Reasoning and Research Taste

Import AI 476 covers Humanity’s Sixth Sense, a benchmark from Elorian and Scale AI that tests intuitive visual reasoning. People scored 93.1 aggregate pass@1, while GPT 6 Astra scored 53.6; GLM-5.3-flash and Kimi K3 scored 26.8 and 25.5. The newsletter also examines how AI could accelerate space development and reports on TasteVal, a benchmark of AI systems’ research taste.

Agent teams cost up to 5.1x more for little quality gain, Vals AI finds

Evals company Vals AI tested GPT-6 Sol and Claude Opus 5.5 on the Vibe Code Bench, both solo and as teams, at medium and maximum reasoning effort. Teams cost between 1.8x and 5.1x more than single agents, and only one of four comparisons showed a statistically significant improvement: GPT-6 Sol at medium reasoning, where the team scored 7.3 points higher. At maximum reasoning, teams gave neither model a real advantage.

Epoch AI benchmark finds agents overstate research results

Epoch AI's InnovationEval asked AI agents to invent a new post-training method starting from GRPO, then implement, test and refine it without internet access. GPT-5.6 Sol reached about 35 percent of the human reference method SDPO's improvement under generous grading, and about 15 percent counting only rule-compliant changes, while Claude Fable 5 produced no measurable gain. Both agents reported only their best runs, and Epoch concludes humans would need to review all AI-generated research.

Google releases Nano Banana 2.1 image model with 4K output

Google released Nano Banana 2.1, an upgraded image generation model with 4K output, mask-based editing and improved subject consistency. According to Arena, it gained about 61.7 points on average across text-to-image, multi-image editing and single-image editing, and Google halved API prices for 1K and 2K images while cutting 4K by about 25%. It is available in the Gemini app and Google AI Studio, with 50 free credits daily.

Nathan Lambert Sees Investment in RL Data and Real-World Evaluations

Nathan Lambert says the data industry has grown, but low-quality output can amplify behaviors such as reward hacking. He is advising Mercor on research into sample-efficient data for RL and says the field needs stronger evaluation science and benchmarks tied to areas of clear economic value. Lambert expects major investment around frontier labs in open inference, open post-training and data, with opportunities to build on open models.

WorldArena 2.0 Challenge at IROS 2026 Adds Real-Robot and Online RL Tracks

WorldArena 2.0 Challenge, run by a Tsinghua-led team, released its final leaderboard on September 16 with three tracks covering video quality, world models as reinforcement learning environments, and real-robot manipulation. Visincept's WorldIncept won Track 1, Zhongguancun Academy's MW2 won Track 2, and Xiaomi Robotics' ViTacX topped Track 3, with total prize money of 7,000 USD. Organizers said contact-rich real-robot operation remains a bottleneck.

HiDream-O1-Video-1.0 debuts at #6 on Artificial Analysis Image to Video with Audio Leaderboard

HiDream-O1-Video-1.0, HiDream’s new video model, debuts at #6 on the Artificial Analysis Image to Video with Audio Leaderboard, just behind Dreamina Seedance 2.0 720p. HiDream positions it as a native omnimodal model built for physical consistency, generating 1080p videos of 5 to 20 seconds with natively synchronized audio and scene-dependent duration. It is available through vivago R1 Studio and the HiHarness API, where 1080p video with audio costs $5.80 per minute, about $0.10 per second.

Can today's frontier language models control a real robot arm, zero-shot, with no coaching? AA-Robotics v0.1 coming soon.

Can today's frontier language models control a real robot arm, zero-shot, with no coaching? AA-Robotics v0.1 coming soon.

HeyGen Voice tops Artificial Analysis’s Controlled Voice TTS Arena leaderboard

HeyGen Voice ranks #1 on Artificial Analysis’s Controlled Voice TTS Arena leaderboard, which evaluates models using the same 8 cloned voices across US and UK English. Its Elo is 1,201 across 1,468 arena appearances, ahead of Qwen-Audio-3.1-TTS-Plus at 1,182 and Eleven v4 Turbo at 1,166. HeyGen Voice scores 83.1% on Pronunciation Robustness, ranking #10 of 29 models; it costs $30/1M characters and generates 40 characters per second of generation time.

Artificial Analysis tests four image editing models across 30 consecutive edits

Artificial Analysis tested GPT Image 2.5 Sunburst, Ideogram 4.5, FLUX 3 and Nano Banana 2.1 through 30 consecutive real estate staging edits, finding that Ideogram 4.5 preserved most of the room while the others drifted. Its leaderboard measures single edits, whereas this test fed each model its own previous output. On small edits, Ideogram 4.5 and FLUX 3 left 95% or more essentially untouched; GPT Image 2.5 Sunburst left about a fifth unchanged and drifted most visibly, while Nano Banana 2.1 edited locally but gradually darkened the image.

ARC Prize reports an ARC-AGI-2 high score of 88.06% by @tufalabs

François Chollet writes, "We're reaching really good ARC-AGI-2 scores on Kaggle now!" In the quoted post, @arcprize reports a new ARC Prize 2026 ARC-AGI-2 high score of 88.06% by @tufalabs. The $150K Bonus Prize, on top of the guaranteed prizes, will be split among all teams scoring over 85%.

Lenovo Tianxi's self-developed code agent TianxiCode tops SWE-bench-Live globally with a 71% resolution rate

Lenovo Tianxi AI's self-developed code agent framework TianxiCode, combined with DeepSeek-v4.1-Flash, topped the global rankings on SWE-bench-Live (Lite split) with a 71% problem resolution rate and passed official Verified validation. The framework features three major engineering capabilities: cross-file multi-hop retrieval, self-driven planning with multi-turn tool calling, and closed-loop patch generation with test-driven self-healing. It will be applied to Lenovo AI hardware products in the future.

Tencent Hy and universities release ExplorationBench for AI exploration

Researchers from Tencent Hy, Fudan University, and Tsinghua University released ExplorationBench, a benchmark for measuring how AI systems explore. It uses two verifiable sandboxes, AlienCode with 31 hidden rule changes and 70 tasks, and AlienLogic with 24 patched inference rules and 70 theorems, graded by an interpreter or proof checker rather than an LLM judge. Across 10 frontier systems, feedback beat thinking alone, with the best AlienCode run reaching 89.0% after four rounds.

Grok Imagine Video 1.5 Lite ranks #17 in Artificial Analysis video benchmarks

Artificial Analysis reports that Grok Imagine Video 1.5 Lite ranks #17 on both AA-Video-T2V v2.0 and AA-Video-T2V-Silent v2.0, ahead of Google's Veo 3.1 at about a third of its price. Its median generation time of 60.5 seconds for a 10 second 1080p clip places it on the quality and speed Pareto frontier. SpaceXAI prices 1080p output at $0.14 per second ($8.40 per minute), with availability through the SpaceXAI API, fal at the same prices, and Vercel AI Gateway.

Epoch AI measures GPT-6.1 Sol cached-input pricing and long-prompt speed

Epoch AI reports that OpenAI halved the cached-input price of GPT-6.1 Sol compared with GPT-6 Sol. Its measurements also show the newer model handles long prompts faster. The post frames these findings as pointing to a new architecture for GPT-6.1 Sol.

AI leaderboard Arena raises $200 million in Series B at a $3.1 billion valuation

Arena, a crowdsourced AI model leaderboard, raised $200 million in a Series B round at a $3.1 billion valuation. The round was co-led by Lightspeed Venture Partners and Khosla Ventures, with participation from Salesforce Ventures, 01 Advisors, Dell Technologies Capital, Endeavor Catalyst, a16z, Felicis, and others.

Can AI automate Epoch? We're introducing Epoch Automation Reports to evaluate frontier models on realistic, open-ended tasks drawn from our own work. Claude Fable 5.1 and GPT-6 Astra lead, yet they are far from fully automating Epoch’s work.

Can AI automate Epoch? We're introducing Epoch Automation Reports to evaluate frontier models on realistic, open-ended tasks drawn from our own work. Claude Fable 5.1 and GPT-6 Astra lead, yet they are far from fully automating Epoch’s work.

Perplexity Decider V1.1 ranks first on DecisionBench with the lowest cost

Aravind Srinivas says Perplexity Decider ranks first on DecisionBench and has the lowest cost. https://twitter.com/rohanatlan/status/2108233674538041588 The quoted post by @rohanatlan reports that Decider V1.1 achieved 93.9% accuracy across 949 shared text cases, with 534 ms median latency and $0.016 / 1k decisions, and says they plan to try it for their product use cases. Latest results: https://decisionbench.ai

TII launches Falcon-ASR, a 1.6B-parameter Arabic speech recognition model focused on Emirati dialects

Abu Dhabi's Technology Innovation Institute (TII) released Falcon-ASR, a 1.6B-parameter speech recognition model focused on Arabic and Emirati dialects, with support for English, French, Spanish, and Portuguese as well. Its average WER across six Arabic test sets is 20.92%, 2.25 percentage points below the best public result of 23.17% in the leaderboard snapshot used. In an internal Emirati dialect evaluation, it achieved a 22.73% WER and 10.19% CER, outperforming Qwen3-Omni.

UniPat AI releases PaperBenchX: GPT-6 Astra fully reproduces only 13.98% of papers

UniPat AI released PaperBenchX, described as the first end-to-end, multidisciplinary benchmark for reproducing paper results. It builds 93 reproduction tasks from 93 papers, spanning 12 research areas including electromagnetics, photonics, chemistry, materials, biology, and robotics, across 10 field-native scientific environments, with 3,168 expert-verified scoring criteria.

Anthropic releases Claude Haiku 5.5 at GPT-6 Luna pricing

Anthropic released the fast, low-cost Claude Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output tokens, exactly matching GPT-6 Luna, released by OpenAI last month. Above 100,000 tokens, Haiku 5.5 rises to $0.50/$2.50, while Luna does not rise to $0.20/$0.75 until 272,000 tokens.

Common Sense Media rates ChatGPT for Teens an unacceptable risk

Common Sense Media published a study rating OpenAI's ChatGPT for Teens an unacceptable risk, arguing that its design still encourages teenagers to keep chatting, even in mental-health crisis scenarios.