Skip to content
All news · Today3 items

#Benchmarks

All collected and published items, ordered by time. Picks are a selected subset.
Models

New models and versions, open weights, and changes in capability and pricing, with their evaluations.

Today10/17Sat
  1. 量子位67

    UBTECH introduces Thinker-WM, unifying video, depth, and robot actions in one world model

    UBTECH introduces Thinker-WM, a multimodal world model that jointly models RGB video, depth, and robot actions through a shared Unified Dynamics Transformer. In material supplied by UBTECH and republished by QbitAI, the company reports leading results on WorldOlympiad, EWMBench, and WorldArena1.0 Track-1, including scores of 0.740 on WorldOlympiad and 4.66 on EWMBench. The model supports six capability modes through shared weights, and adding depth improved RoboTwin single-task success by 15 percentage points over a baseline without depth.

  2. The Verge · AI77

    A week with Nvidia RTX Spark laptops reveals performance strengths and practical compromises

    The Verge’s three reviewers spent a week testing five Nvidia RTX Spark laptops, finding capable graphics and generally workable Windows on Arm apps alongside high prices, weaker CPU performance than Apple rivals, and demanding local AI workloads. Sean Hollister’s Asus ProArt P16 lasted roughly 5 hours to 5 hours and 40 minutes during everyday work. Their local AI experiments encountered substantial battery drain, heat, slow responses, and uneven results, despite configurations offering up to 128GB of unified memory.

    Why it matters: The hands-on comparisons show how large unified memory and capable graphics translate into everyday tradeoffs in local AI workloads, battery life, and software compatibility.

10/16Fri
  1. The Decoder78

    Google's Gemini 4 Argon is closer to the AI frontier than any Google model in months

    Google DeepMind’s Gemini 4 Argon scored 97.5 percent on ARC-AGI-3 using the Provider Adapter Harness at a cost of $4,800, behind GPT-6 Astra’s 99.9 percent. Its score fell to 49.8 percent at low reasoning in the Standard Harness, where API problems prevented completion of all games. Epoch AI placed Argon at 165 ECI points, close to Opus 5.5 and GPT-6 Astra, although early estimates ranked it eighth in math and sixth in software engineering.

    Why it matters: The comparison shows how testing environments can change benchmark standings, providing context for interpreting Argon’s proximity to leading models alongside its weaker math and software engineering results.

  2. Aravind Srinivas · X70

    Perplexity adds Gemini 4 Argon, citing a new WANDR benchmark high

    Aravind Srinivas says Gemini 4 Argon has set a new SoTA on the Wide-And-Deep-Research (WANDR) benchmark and is now available to all Perplexity users in fast search and Computer (agent) modes. https://t.co/0U671BhbMj The quoted @perplexity_ai announcement specifies availability for Pro and Max subscribers. https://t.co/n1TyI9RhBX

    Why it matters: The post identifies availability across fast search and Computer modes, while its broader access claim differs from the quoted announcement limiting availability to Pro and Max subscribers.

  3. Artificial Analysis · X59

    Liquid AI releases d1-ultra, leading the Artificial Analysis Decision Index v1.0

    Liquid AI has released d1-ultra, which Artificial Analysis reports leads its Decision Index v1.0 with a score of 65. On AA-Omniscience-Decision, its non-hallucination rate rose to 85% from d1’s 2%, while accuracy fell to 32% from 41%. This non-hallucination metric measures how often the model says it does not know among questions it does not answer correctly. d1-ultra takes 92 ms per task and costs $0.042 per 1M input tokens, billed only on input tokens.

  4. Artificial Analysis · X79

    Meta releases Muse Spark Open, scoring 39 on Artificial Analysis Intelligence Index

    Meta has released Muse Spark Open, an 816B (42B active) open-weights model that Artificial Analysis reports scores 39 on its Intelligence Index, making it the #1 US open-weights model. Muse Spark Open (xhigh) scores 22 points above Muse Glimmer (high, 17), Meta's 30B dense open-weights model. https://t.co/jf31mQNINe

    Why it matters: The comparison places Muse Spark Open 22 points above Muse Glimmer on the same intelligence index, providing a concrete reference for the reported capability gap between Meta's open-weights models.

  5. Artificial Analysis · X55

    RTX Spark laptop completes Artificial Analysis local agent workload in 18.7 minutes

    NVIDIA's RTX Spark N1X puts up to 128 GB of unified memory and a Blackwell GPU in a Windows laptop. In Artificial Analysis's first AA-AgentPerf-Local results, an RTX Spark laptop completes its local agent workload on Qwen3.8 27B in 18.7 minutes. Artificial Analysis describes RTX Spark's N1X chip as essentially the GB10, linking to https://t.co/520A4lOOdB

10/15Thu
  1. Epoch AI · X51

    Gemini 4 Argon scores 165 on Epoch AI’s Capabilities Index

    Gemini 4 Argon scores 165 on the Epoch Capabilities Index (ECI), on par with Fable 5.1 (165) but behind Opus 5.5 (167) and GPT-6 Astra (166). While behind the ECI frontier, Argon is the closest Google has come since Gemini 3.1 Pro in Feb, and a big jump from Gemini 3.8 Flash (157). https://t.co/PvZvD4KyvI

  2. Artificial Analysis · X72

    Grok Voice 3.0 leads Artificial Analysis Speech to Speech Index at 86.9

    Artificial Analysis evaluated SpaceXAI’s Grok Voice 3.0 using the Rex voice through the SpaceXAI Realtime API, reporting a #1 Speech to Speech Index score of 86.9, up from Grok Voice 2.0 at 81.3. It ranked #3 in arena preference and #1 in task success rate at 94.8%. Average time to first audio on Big Bench Audio was 0.81 seconds versus 0.70s for Grok Voice 2.0, while input audio pricing remained $4.80 per hour.

10/14Wed
  1. Artificial Analysis · X46

    Claude Sonnet 5.5 Scores 68 on the Coding Agent Index at Max Effort

    In Claude Code, Sonnet 5.5 scores 68 on the Coding Agent Index at max effort and 63 at xhigh. The xhigh setting costs $3.33 per task versus $14.19 at max, a 77% reduction for 5.5 fewer Index points. Max generates 4.8× as many output tokens and takes 3.4× as many turns, while improving all three benchmark scores; its largest gain is on Terminal-Bench 4.0, from 58.1% to 66.2%.

  2. Artificial Analysis · X78

    Fish Audio’s Drama 3 Preview leads open-weights models in Artificial Analysis TTS rankings

    Artificial Analysis reports that Fish Audio’s Drama 3 Preview ranks #1 among open-weights models on all three of its Text to Speech leaderboards and leads all models in Pronunciation Robustness. It scores 93.2% on pronunciation robustness, ahead of Eleven v4 at 91.7% and Eleven v4 Turbo at 90.1%, while ranking #10 in Provider Voice and #14 in Controlled Voice. Its weights are available on Hugging Face under the Fish Audio Community License, and Fish Audio API pricing is $15/1M characters.

    Why it matters: The comparison separates pronunciation robustness from voice preference rankings, showing how Drama 3 Preview’s lead on one measure differs from its placement on others.

  3. Artificial Analysis · X87

    Gemini 4 Argon reaches GA, matching GPT-6 Astra on Artificial Analysis Intelligence Index

    Artificial Analysis reports that Gemini 4 Argon is now GA, matching GPT-6 Astra on its Intelligence Index at 60% of the Cost per Task under discounted pricing until November 30. Argon costs $1.99 per task and $2/$10 per 1M input/output tokens, rising to $4/$20 after November 30, with a 95% cache-read discount. Artificial Analysis also reports substantial agentic evaluation improvements and a 15% hallucination rate on AA-Omniscience, the lowest among models scoring 45+ on its Intelligence Index.

    Why it matters: The comparison separates intelligence scores from task costs, showing how Argon's temporary pricing affects its cost advantage over GPT-6 Astra at a comparable intelligence level.

10/13Tue
  1. 量子位63

    MagicLab Studio’s EchoPolicy-0 tops MolmoSpaces Combined with 75.4% average success

    QbitAI reports that MagicLab Studio’s EchoPolicy-0 ranked first on MolmoSpaces Combined as of October 13, 2026, achieving 75.4% average success across all four v1 manipulation tasks in 3915 evaluation episodes. Its 3.4B-parameter MagicBrain ranked fourth with 72.4% average success and 94.4% grasping success. EchoPolicy-0 combines VLM-guided candidate trajectory selection with execution checks, while MagicBrain uses visual markers and post-training to connect target locations with actions without changing the VLA architecture.

  2. The Decoder83

    Meta releases Muse Wonder models with gains in coding and agent benchmarks

    Meta released Muse Wonder and Muse Wonder Full, scoring 52 and 56 on the Artificial Analysis Intelligence Index v4.3, versus 48 for Muse Spark 1.3. Both support a one-million-token context window and text, image, video, and audio inputs, with stronger coding and agent results but weaker factual knowledge than leading competitors. Muse Wonder is available through Meta's API and Muse Code at higher prices than its predecessor; Muse Wonder Full is currently limited to Muse Code users.

    The same story is shown in the picks as 《Meta releases frontier models Muse Wonder and Muse Wonder Full》

  3. 宝玉 · X80

    Meta releases Muse Wonder for developers as consumer access remains unclear

    Baoyu reports that Meta released Muse Wonder and Muse Wonder Full on October 12, with Muse Wonder available today through Meta Model API and Muse Code, while Full is limited to Muse Code subscribers. Meta’s blog says consumer access is coming soon, whereas Zuckerberg says subscribers can use it today. The models support a 1 million token context window and multimodal input with text output. Meta reports strengths in sustained agentic work, coding, and memory, but its comparisons include internal benchmarks and competitor runs that may not be optimally configured.

    The same story is shown in the picks as 《Meta releases frontier models Muse Wonder and Muse Wonder Full》

  4. Alexandr Wang · X82

    Meta releases Muse Wonder and Muse Wonder Full with higher benchmark scores

    Meta has released Muse Wonder and Muse Wonder Full, according to Artificial Analysis in the post quoted by Alexandr Wang. At max reasoning effort, they score 52 and 56 on the Artificial Analysis Intelligence Index v4.3, compared with Muse Spark 1.3’s 48, following a release gap of 6 weeks. Muse Wonder is available through Meta’s first-party API and Muse Code, while Muse Wonder Full is limited to Muse Code users; both have 1M-token context windows.

    Why it matters: Artificial Analysis’s comparison highlights a tradeoff between Muse Wonder’s improved benchmark performance and its higher cost per task relative to Muse Spark 1.3 despite lower output token usage.

  5. 宝玉 · X82

    Meta launches Muse Wonder with developer access and conflicting consumer availability statements

    Meta released Muse Wonder and Muse Wonder Full on October 12, according to Baoyu’s account of the announcement. Developers can access Muse Wonder through Meta Model API or Muse Code, while Full is available to Muse Code subscribers; Meta’s blog and Zuckerberg’s X post differ on consumer availability. The models support a 1 million Token context window and multimodal input with text output, while Meta’s reported benchmarks show strengths in sustained agentic and coding tasks alongside weaker results elsewhere.

    The same story is shown in the picks as 《Meta releases frontier models Muse Wonder and Muse Wonder Full》

  6. Artificial Analysis · X81

    Meta releases Muse Wonder and Muse Wonder Full with higher Artificial Analysis scores

    Meta has released Muse Wonder and Muse Wonder Full, which Artificial Analysis reports score 52 and 56 on its Intelligence Index v4.3 at max reasoning effort, versus 48 for Muse Spark 1.3 (max). Muse Wonder Full improves across all ten evaluations, with the largest gains in agentic evaluations. Muse Wonder is available through Meta’s first-party API and Muse Code at launch, while Muse Wonder Full is limited to Muse Code users; Muse Wonder costs $4.52 per Intelligence Index task.

    The same story is shown in the picks as 《Meta releases Muse Wonder and Muse Wonder Full with higher benchmark scores》

10/11Sun
  1. 量子位78

    Google releases Nano Banana 2.1 image model with 4K output

    Google released Nano Banana 2.1, an upgraded image generation model with 4K output, mask-based editing and improved subject consistency. According to Arena, it gained about 61.7 points on average across text-to-image, multi-image editing and single-image editing, and Google halved API prices for 1K and 2K images while cutting 4K by about 25%. It is available in the Gemini app and Google AI Studio, with 50 free credits daily.

    Why it matters: The piece rounds up Arena ranking gains, API price cuts and early user tests of Nano Banana 2.1, including reported watermark and repeated-edit issues.

10/10Sat
  1. Artificial Analysis · X66

    HiDream-O1-Video-1.0 debuts at #6 on Artificial Analysis Image to Video with Audio Leaderboard

    HiDream-O1-Video-1.0, HiDream’s new video model, debuts at #6 on the Artificial Analysis Image to Video with Audio Leaderboard, just behind Dreamina Seedance 2.0 720p. HiDream positions it as a native omnimodal model built for physical consistency, generating 1080p videos of 5 to 20 seconds with natively synchronized audio and scene-dependent duration. It is available through vivago R1 Studio and the HiHarness API, where 1080p video with audio costs $5.80 per minute, about $0.10 per second.

  2. Artificial Analysis · X55

    HeyGen Voice tops Artificial Analysis’s Controlled Voice TTS Arena leaderboard

    HeyGen Voice ranks #1 on Artificial Analysis’s Controlled Voice TTS Arena leaderboard, which evaluates models using the same 8 cloned voices across US and UK English. Its Elo is 1,201 across 1,468 arena appearances, ahead of Qwen-Audio-3.1-TTS-Plus at 1,182 and Eleven v4 Turbo at 1,166. HeyGen Voice scores 83.1% on Pronunciation Robustness, ranking #10 of 29 models; it costs $30/1M characters and generates 40 characters per second of generation time.

10/9Fri
  1. Artificial Analysis · X74

    Grok Imagine Video 1.5 Lite ranks #17 in Artificial Analysis video benchmarks

    Artificial Analysis reports that Grok Imagine Video 1.5 Lite ranks #17 on both AA-Video-T2V v2.0 and AA-Video-T2V-Silent v2.0, ahead of Google's Veo 3.1 at about a third of its price. Its median generation time of 60.5 seconds for a 10 second 1080p clip places it on the quality and speed Pareto frontier. SpaceXAI prices 1080p output at $0.14 per second ($8.40 per minute), with availability through the SpaceXAI API, fal at the same prices, and Vercel AI Gateway.

  2. Aravind Srinivas · X50

    Perplexity Decider V1.1 ranks first on DecisionBench with the lowest cost

    Aravind Srinivas says Perplexity Decider ranks first on DecisionBench and has the lowest cost. https://twitter.com/rohanatlan/status/2108233674538041588 The quoted post by @rohanatlan reports that Decider V1.1 achieved 93.9% accuracy across 949 shared text cases, with 534 ms median latency and $0.016 / 1k decisions, and says they plan to try it for their product use cases. Latest results: https://decisionbench.ai

10/8Thu
  1. Hugging Face Blog40

    TII launches Falcon-ASR, a 1.6B-parameter Arabic speech recognition model focused on Emirati dialects

    Abu Dhabi's Technology Innovation Institute (TII) released Falcon-ASR, a 1.6B-parameter speech recognition model focused on Arabic and Emirati dialects, with support for English, French, Spanish, and Portuguese as well. Its average WER across six Arabic test sets is 20.92%, 2.25 percentage points below the best public result of 23.17% in the leaderboard snapshot used. In an internal Emirati dialect evaluation, it achieved a 22.73% WER and 10.19% CER, outperforming Qwen3-Omni.

  2. Simon Willison78

    Anthropic releases Claude Haiku 5.5 at GPT-6 Luna pricing

    Anthropic released the fast, low-cost Claude Haiku 5.5 at $0.10 per million input tokens and $0.50 per million output tokens, exactly matching GPT-6 Luna, released by OpenAI last month. Above 100,000 tokens, Haiku 5.5 rises to $0.50/$2.50, while Luna does not rise to $0.20/$0.75 until 272,000 tokens.

    Why it matters: The author tested Haiku 5.5 pricing, tokenizer changes, and the cost and latency of different reasoning tiers, offering a basis for weighing it against GPT-6 Luna.