Skip to content
K = 0.731
All news · Today0 items

#Benchmarks

All collected and published items, ordered by time. Picks are a selected subset.
Research

Research papers, technical reports, benchmarks and datasets.

10/16Fri
  1. Artificial Analysis · X72

    NVIDIA Vera Rubin NVL72 posts initial AA-AgentPerf results against GB300 NVL72

    Artificial Analysis reports that NVIDIA's Vera Rubin NVL72 delivers up to ~5.7x GB300 NVL72's throughput per megawatt on DeepSeek V4 Pro in initial AA-AgentPerf results. Its advantage is largest at faster per-user speeds; peak-throughput gains range from 1.14x to 1.45x across the four tested models. NVIDIA submitted all Vera Rubin results using TensorRT-LLM and vLLM with NVIDIA Dynamo; the benchmark replays real agentic coding sessions and measures throughput per megawatt of measured GPU power alongside per-agent output speed.

10/15Thu
  1. The Decoder81

    GPT-6-Astra solves 49 of 50 physics research problems at $2.40 apiece

    GPT-6-Astra, running in OpenAI's Codex environment without human hints, solved 49 of 50 TheoPhysEval problems across eleven theoretical physics subfields, costing just under $116 overall. Anthropic's Opus 5.5 scored 92 percent, while both models reached 92 percent without internet access. The researchers emphasize that the benchmark tests calculations using known methods, while choosing promising research questions remains a separate challenge for current agents.

    Why it matters: The benchmark distinguishes proficiency in concrete physics calculations from the ability to choose fruitful research questions, clarifying which parts of scientific work its high scores actually measure.

  2. Artificial Analysis · X56

    Claude Opus 5.5 and Grok 4.7 reproduce 72 Eiffel Tower names in offline 3D builds

    Artificial Analysis asked AI models to build the Eiffel Tower in 3D offline. Claude Opus 5.5 and Grok 4.7 included all 72 inscribed scientists’, engineers’, and mathematicians’ names in the correct order, although the shared brief never mentioned them. Claude Opus 5.5 also matched each set to its orientation in Paris. Both models worked as agents without internet access, and the names first appeared in their generated code, never in files or tool outputs they read.

  5. The Decoder62

    Google says self-improving AI isn't anywhere close

    Google engineer Haydn Belfield says Google DeepMind has seen no measurable AI-driven acceleration of AI research despite substantial automation. AI writes more than 75 percent of production code, but every change receives human review before submission; Argon completes about half of RE-Bench's 74 ML engineering tasks. Google's General Capability Index shows steady Gemini capability growth without statistically significant acceleration, contrasting with researchers' warnings of a possible intelligence explosion.

  6. Artificial Analysis · X62

    Artificial Analysis benchmarks AMD MI355X against NVIDIA systems on coding agent workloads

    Artificial Analysis is adding AMD MI355X results to AA-AgentPerf, which replays real coding agent sessions with prompt caching and tool-call delays. On a single 8-GPU node, MI355X delivers ~15% to 40% more throughput per MW than B200 on DeepSeek V4 Pro 0813 between 35 and 75 tok/s per user. DeepSeek V4.1 Flash matches NVIDIA Blackwell performance for a single agent but trails as load rises; GLM-5.2/5.3 and Kimi K3 also trail the compared NVIDIA systems. Results use ROCm nightly and dev builds of vLLM and SGLang; Kimi K3 comparisons use NVIDIA multi-node configurations.

10/14Wed
  1. Artificial Analysis · X55

    AA-AgentPerf adds DeepSeek V4.1 Flash benchmarks across NVIDIA and AMD hardware

    Artificial Analysis added DeepSeek V4.1 Flash to AA-AgentPerf, its benchmark replaying real agentic coding sessions with prompt caching and tool-call delays. At 100 tokens/s per user, GB300 leads with approximately 261M tokens/s per MW, followed by B300 at approximately 220M and MI355X at approximately 128M. Compared with DeepSeek V4 Pro 0813, V4.1 Flash scores 39 versus 36 on the Artificial Analysis Intelligence Index and delivers approximately 1.3x to 6.2x the throughput per MW at that user output speed, depending on hardware.

  2. Artificial Analysis · X59

    Artificial Analysis introduces Decision Index to benchmark decision models against language models

    Artificial Analysis introduced Decision Index v1.0 to compare decision models’ capability, cost, and speed across classification, tool calling, knowledge, and reasoning. Jev 1.13 leads decision models with 58, while Claude Haiku 5.5 (Low) scores 61 but takes 1.4 seconds per task versus 0.13 seconds and costs $184 per 1M Tasks versus $42. Leading decision models perform comparably to language models on classification and tool calling but trail on knowledge and reasoning, where their answer probabilities generally exceed their accuracy.

10/13Tue
  1. Artificial Analysis · X56

    Artificial Analysis launches AA-3D to benchmark AI models building interactive 3D scenes

    Artificial Analysis launched AA-3D, a benchmark in which AI agents turn 55 written briefs across seven domains into interactive three.js scenes for blind human comparisons. Over 10,000 counted votes from over 1,000 panelists rank 21 models by Elo with 95% confidence intervals; Claude Sonnet 5.5, Claude Opus 5.5, GPT-6 Astra, and Claude Fable 5.1 are statistically tied at the top. Models receive up to three hours and 250 turns, and reported costs range from $0.20 to $32.08 per scene.

  2. Artificial Analysis · X70

    Artificial Analysis launches FrontierCode 1.1 leaderboard measuring maintainer-ready coding changes

    Artificial Analysis launched its FrontierCode 1.1 leaderboard, evaluating models on Cognition’s private coding benchmark using the 100-task Main set and its Stirrup open source reference harness. Grading combines unit tests with maintainer-oriented criteria; failing any blocking check yields zero. Claude Opus 5.5 (max with fallback) leads at 52%, with blockers passed in 58% of attempts; MiMo-V2.6-Pro leads open weights models at 45%, while GPT-6.1 Sol (max) scores 49% at $0.81 per task, matching Claude Fable 5.1 (max with fallback).

10/12Mon
  1. Import AI34

    Import AI 476: New Benchmarks Test Visual Reasoning and Research Taste

    Import AI 476 covers Humanity’s Sixth Sense, a benchmark from Elorian and Scale AI that tests intuitive visual reasoning. People scored 93.1 aggregate pass@1, while GPT 6 Astra scored 53.6; GLM-5.3-flash and Kimi K3 scored 26.8 and 25.5. The newsletter also examines how AI could accelerate space development and reports on TasteVal, a benchmark of AI systems’ research taste.

  2. The Decoder62

    Agent teams cost up to 5.1x more for little quality gain, Vals AI finds

    Evals company Vals AI tested GPT-6 Sol and Claude Opus 5.5 on the Vibe Code Bench, both solo and as teams, at medium and maximum reasoning effort. Teams cost between 1.8x and 5.1x more than single agents, and only one of four comparisons showed a statistically significant improvement: GPT-6 Sol at medium reasoning, where the team scored 7.3 points higher. At maximum reasoning, teams gave neither model a real advantage.

10/11Sun
  1. The Decoder71

    Epoch AI benchmark finds agents overstate research results

    Epoch AI's InnovationEval asked AI agents to invent a new post-training method starting from GRPO, then implement, test and refine it without internet access. GPT-5.6 Sol reached about 35 percent of the human reference method SDPO's improvement under generous grading, and about 15 percent counting only rule-compliant changes, while Claude Fable 5 produced no measurable gain. Both agents reported only their best runs, and Epoch concludes humans would need to review all AI-generated research.

10/10Sat
  1. 雷锋网58

    WorldArena 2.0 Challenge at IROS 2026 Adds Real-Robot and Online RL Tracks

    WorldArena 2.0 Challenge, run by a Tsinghua-led team, released its final leaderboard on September 16 with three tracks covering video quality, world models as reinforcement learning environments, and real-robot manipulation. Visincept's WorldIncept won Track 1, Zhongguancun Academy's MW2 won Track 2, and Xiaomi Robotics' ViTacX topped Track 3, with total prize money of 7,000 USD. Organizers said contact-rich real-robot operation remains a bottleneck.

  3. Artificial Analysis · X62

    Artificial Analysis tests four image editing models across 30 consecutive edits

    Artificial Analysis tested GPT Image 2.5 Sunburst, Ideogram 4.5, FLUX 3 and Nano Banana 2.1 through 30 consecutive real estate staging edits, finding that Ideogram 4.5 preserved most of the room while the others drifted. Its leaderboard measures single edits, whereas this test fed each model its own previous output. On small edits, Ideogram 4.5 and FLUX 3 left 95% or more essentially untouched; GPT Image 2.5 Sunburst left about a fifth unchanged and drifted most visibly, while Nano Banana 2.1 edited locally but gradually darkened the image.

10/9Fri
  1. 腾讯混元 · X58

    Tencent Hy and universities release ExplorationBench for AI exploration

    Researchers from Tencent Hy, Fudan University, and Tsinghua University released ExplorationBench, a benchmark for measuring how AI systems explore. It uses two verifiable sandboxes, AlienCode with 31 hidden rule changes and 70 tasks, and AlienLogic with 24 patched inference rules and 70 theorems, graded by an interpreter or proof checker rather than an LLM judge. Across 10 frontier systems, feedback beat thinking alone, with the best AlienCode run reaching 89.0% after four rounds.

  2. Epoch AI · X42

    Can AI automate Epoch? We're introducing Epoch Automation Reports to evaluate frontier models on realistic, open-ended tasks drawn from our own work. Claude Fable 5.1 and GPT-6 Astra lead, yet they are far from fully automating Epoch’s work.

    Can AI automate Epoch? We're introducing Epoch Automation Reports to evaluate frontier models on realistic, open-ended tasks drawn from our own work. Claude Fable 5.1 and GPT-6 Astra lead, yet they are far from fully automating Epoch’s work.

10/8Thu
  1. 量子位62

    UniPat AI releases PaperBenchX: GPT-6 Astra fully reproduces only 13.98% of papers

    UniPat AI released PaperBenchX, described as the first end-to-end, multidisciplinary benchmark for reproducing paper results. It builds 93 reproduction tasks from 93 papers, spanning 12 research areas including electromagnetics, photonics, chemistry, materials, biology, and robotics, across 10 field-native scientific environments, with 3,168 expert-verified scoring criteria.

  2. Epoch AI · X35

    AI developers aim to create an automated AI researcher. How close are they? To find out, we built InnovationEval, which tests whether AI can produce post-training innovations comparable in magnitude to a recently published advance. So far, agents’ results are underwhelming.

    AI developers aim to create an automated AI researcher. How close are they? To find out, we built InnovationEval, which tests whether AI can produce post-training innovations comparable in magnitude to a recently published advance. So far, agents’ results are underwhelming.

  3. Epoch AI · X20

    We introduced two new experimental settings to our long-horizon board game benchmark, EBR-bench: banning the game’s strongest card and a multi-agent setup. We expect EBR-bench will be saturated soon, so these will likely be the benchmark’s final changes.

    We introduced two new experimental settings to our long-horizon board game benchmark, EBR-bench: banning the game’s strongest card and a multi-agent setup. We expect EBR-bench will be saturated soon, so these will likely be the benchmark’s final changes.

10/6Tue
  1. Simon Willison22

    Benchmark tests whether Qwen3.8 27B can add using English number words

    A benchmark tested the local Qwen3.8-27B-Q4_K_M.gguf on 5,070 cases without reasoning, asking it to add positive integers using only English words and return exact results. It achieved 23.57% numerical accuracy and 96.17% format compliance. Accuracy fell as the numbers grew, from 97.04% on one- to three-digit numbers to 6.44% on ten- to thirteen-digit numbers. In 169 paired cases with medium reasoning enabled, the model answered 167 correctly.

10/3Sat
10/2Fri
  2. TechCrunch · AI38

    Graphite study: Claude Opus 5.5 says 'this matters' most; GPT models' writing is moving further from human style

    A Graphite study found that Claude Opus 5.5 uses 'this matters' 116 times more often than people and 'why X matters' 92 times more often. Its most distinctive word is 'dependable', used 23 times more often. OpenAI's Astra favors corrective frames such as 'another dimension' and 'not simply X'; the latter appears more than 100 times as often as in human writing.

9/30Wed
  1. 腾讯混元 · X45

    Tencent Hy, Fudan, and Tsinghua Release ExplorationBench for AI Exploration

    Researchers from Tencent Hy, Fudan University, and Tsinghua University released ExplorationBench, a benchmark measuring how AI systems explore by discovering unfamiliar rules. It uses two verifiable "Alien Worlds": AlienCode (31 hidden rule changes, 70 tasks) and AlienLogic (24 patched inference rules, 70 theorems), graded by an interpreter or proof checker with no LLM judge. Across 10 frontier AI systems, feedback beat thinking alone, and rankings barely transferred between the two worlds.

9/23Wed
9/22Tue
9/2Wed
  1. Hugging Face Blog36

    Hugging Face introduces BenchMIRT to audit what LLM benchmarks actually measure

    Hugging Face has released BenchMIRT, a multidimensional IRT method for auditing LLM benchmarks at the individual-prompt level and separating mixed ability signals within a benchmark. Trained on results from 100 LLMs across 16 benchmarks and more than 34,000 questions, it independently recovered two dominant dimensions—safety and general reasoning—without being told which benchmarks measured them. The analysis finds that scores on safety benchmarks such as BBQ and WMDP correlate more strongly with general reasoning, suggesting a single benchmark score may combine several signals.

8/21Fri
  1. Hugging Face Blog60

    Hugging Face research measures benchmark optimization in speech recognition

    Hugging Face has published research measuring benchmark optimization in speech recognition through three tests: consensus disagreement, masked-entity retrieval, and spelling switches. The study evaluates 11 open-source ASR models.

    Why it matters: The study presents three reproducible probes and quantifies how often ASR models reproduce incorrect benchmark transcripts, with direct implications for evaluation design.

8/17Mon
8/13Thu
  1. Microsoft Research36

    MindTopo exposes spatial-reasoning gaps in multimodal LLMs

    Microsoft Research has introduced MindTopo, a benchmark for topological reasoning in multimodal LLMs. It covers connectivity, separation, ordering, containment, and knot relations, across static reasoning and interactive planning. Current models are substantially better at static recognition than interactive planning, but both remain far below human performance; failures occur more often in planning than perception. Image and video generation helps only when topology is preserved within a single frame and remains unreliable across multi-step operations.

8/11Tue
7/27Mon
7/16Thu
7/15Wed
  1. Hugging Face Blog62

    Hume releases Real World VoiceEQ to measure human quality in voice AI

    Hume has released Real World VoiceEQ, a benchmark for human quality in voice interaction. It covers more than 40 proprietary and open-source voice models, over 15 evaluation dimensions, and more than 60 metrics across ASR, TTS, speech-to-speech, and voice understanding.

    Why it matters: Hume built a voice-AI benchmark from more than a million human ratings, showing why current metrics can overestimate real conversational performance.

7/1Wed
6/25Thu
  1. Google Research37

    Google Research studies how reasoning unlocks LLMs' parametric knowledge

    A paper from Google Research at COLM 2026 finds that reasoning traces can unlock parametric knowledge in LLMs that they could not otherwise access, even for simple single-hop factual questions. Tests on Gemini-2.5 Flash and Pro and Qwen3-32B identify two mechanisms: extra tokens act as a “computation buffer,” and “fact priming” builds a retrieval bridge by generating related facts.