Skip to content
K = 0.731
All news · Today4 items

#Benchmarks

All collected and published items, ordered by time. Picks are a selected subset.
10/8Thu
  1. Epoch AI · X35

    AI developers aim to create an automated AI researcher. How close are they? To find out, we built InnovationEval, which tests whether AI can produce post-training innovations comparable in magnitude to a recently published advance. So far, agents’ results are underwhelming.

    AI developers aim to create an automated AI researcher. How close are they? To find out, we built InnovationEval, which tests whether AI can produce post-training innovations comparable in magnitude to a recently published advance. So far, agents’ results are underwhelming.

  2. Epoch AI · X20

    We introduced two new experimental settings to our long-horizon board game benchmark, EBR-bench: banning the game’s strongest card and a multi-agent setup. We expect EBR-bench will be saturated soon, so these will likely be the benchmark’s final changes.

    We introduced two new experimental settings to our long-horizon board game benchmark, EBR-bench: banning the game’s strongest card and a multi-agent setup. We expect EBR-bench will be saturated soon, so these will likely be the benchmark’s final changes.

10/7Wed
  1. Simon Willison42

    Hands-on with Mistral Large 4: comparing Claude, GPT, and Gemini on an SVG generation test

    Simon Willison used the same prompt to ask Mistral Large 4, claude-opus-5.5, gpt-6.1-sol, and gemini-3.8-flash each to generate an SVG of an armadillo jaywalking on Mars in fishnet stockings, comparing frontier models after benchmark saturation. He ran the test with the LLM command-line tool, using each model's default reasoning level.

  2. Microsoft Research22

    Microsoft Research: why AI evaluations can fail unexpectedly outside traditional benchmarks

    Jennifer Neville of Microsoft says models can fail unexpectedly when tested beyond traditional benchmarks. Evaluation is essential to pushing the performance boundaries of current AI systems and aligning them with user needs. She recommends paying close attention to data when using AI systems, since the data itself is often the first thing to examine when results differ from expectations.

10/6Tue
  2. Simon Willison22

    Benchmark tests whether Qwen3.8 27B can add using English number words

    A benchmark tested the local Qwen3.8-27B-Q4_K_M.gguf on 5,070 cases without reasoning, asking it to add positive integers using only English words and return exact results. It achieved 23.57% numerical accuracy and 96.17% format compliance. Accuracy fell as the numbers grew, from 97.04% on one- to three-digit numbers to 6.44% on ten- to thirteen-digit numbers. In 169 paired cases with medium reasoning enabled, the model answered 167 correctly.

10/3Sat
  3. The Verge · AI58

    Hands-on with OpenAI's Dots agent: more like enterprise software, but it can also help order dinner

    A Verge reporter tested Dots, the agent platform OpenAI launched this week. It is currently limited to top-tier accounts, including the $100-per-month Pro account. Users can have only one Dot and can customize its name and avatar. The interface resembles Meta Muse, with one window for conversation and another showing it operating in a virtual machine.

10/2Fri
  3. TechCrunch · AI38

    Graphite study: Claude Opus 5.5 says 'this matters' most; GPT models' writing is moving further from human style

    A Graphite study found that Claude Opus 5.5 uses 'this matters' 116 times more often than people and 'why X matters' 92 times more often. Its most distinctive word is 'dependable', used 23 times more often. OpenAI's Astra favors corrective frames such as 'another dimension' and 'not simply X'; the latter appears more than 100 times as often as in human writing.

10/1Thu
  1. The Decoder78

    Google releases Gemini 4 Argon, matching OpenAI and Anthropic but without a clear lead

    Google released frontier model Gemini 4 Argon, matching OpenAI and Anthropic on several benchmarks without a clear lead. It is initially available to 'trusted cyber defenders' in the Fairwind program and Google's internal teams before expanding to developers, businesses, and consumers. Promotional API pricing is $2 per million input tokens and $10 per million output tokens.

    Why it matters: This summary compares third-party and human-preference leaderboards, scores, token use, and pricing for Gemini 4 Argon against OpenAI and Anthropic.

  2. Perplexity · X42

    We built a new way to train contextual embedding models, which encode each chunk of a document with the whole document in view. pplx-embed-v2-context-9b-preview sets a new state of the art on ConTEB and @turbopuffer's new, privately held context-bench. https://www.perplexity.ai/hub/blog/contextual-embedding-beyond-the-gold-passage

    We built a new way to train contextual embedding models, which encode each chunk of a document with the whole document in view. pplx-embed-v2-context-9b-preview sets a new state of the art on ConTEB and @turbopuffer's new, privately held context-bench. https://www.perplexity.ai/hub/blog/contextual-embedding-beyond-the-gold-passage

9/30Wed
  1. Hugging Face Blog60

    Hugging Face releases Open TTS Leaderboard for objective evaluation of multilingual TTS and voice cloning

    Hugging Face released Open TTS Leaderboard, which uses objective metrics including WER/CER, RTFx, TTFA, and speaker similarity to evaluate open-source TTS models, reducing evaluation cycles from weeks of voting to hours.

    Why it matters: Hugging Face replaces vote-based evaluation with objective metrics, showing how to compare open-source TTS models across languages and voice cloning.

  2. 腾讯混元 · X45

    Tencent Hy, Fudan, and Tsinghua Release ExplorationBench for AI Exploration

    Researchers from Tencent Hy, Fudan University, and Tsinghua University released ExplorationBench, a benchmark measuring how AI systems explore by discovering unfamiliar rules. It uses two verifiable "Alien Worlds": AlienCode (31 hidden rule changes, 70 tasks) and AlienLogic (24 patched inference rules, 70 theorems), graded by an interpreter or proof checker with no LLM judge. Across 10 frontier AI systems, feedback beat thinking alone, and rankings barely transferred between the two worlds.

  8. 雷锋网38

    Hands-on with Seedance 2.5 versus Kling VIDEO 3.0: the gap in AI video models is about workflow, not just image quality

    Tests found that Seedance 2.5 has stronger visual presentation and character consistency, but blocked a plot rewrite based on reference images from the series Journey to Love, first saying the images might depict public figures and then stopping generation over 'copyright-protected elements' after the references were removed. Kling VIDEO 3.0 generated a complete 15-second short film with a twist ending.

  9. 雷锋网60

    Testing WorkBuddy, Qianwen Office, and Doubao Work: the same dirty data produces three different totals

    The author tested WorkBuddy, Qianwen Office, and Doubao Work using the same messy sales records and prompt. All three delivered an Excel file, a presentation, and a summary, but the highest quarterly total was 57.7% above the lowest. The difference came from how each handled orders with missing department data: exclude them, keep them, or mark them for verification.

9/23Wed
  1. Qwen · X78

    Qwen launches Qwen Intelligence with three state-of-the-art mobile agents

    Qwen has launched Qwen Intelligence with three state-of-the-art agents for mobile. The first, Mobile Planner Agent, handles task planning and decomposition, and is evaluated on MobilePA-Bench.

    Why it matters: The official release gives benchmark results and access points for three mobile agents, helping readers assess the current capabilities of on-device agents.

9/22Tue
9/16Wed
  1. Latent Space38

    How Good Start Labs uses games to train AI models and transfer skills to real work

    Good Start Labs turned the 1830 railway board game into a training environment for experiments with 30B models. Both single-turn question-and-answer and multi-turn terminal-agent training improved performance in the game, but only the terminal-agent setup improved results on the Finance-Agent benchmark. The company, spun out of Every and funded with $3.6 million, argues that how a game is presented determines which skills a model learns and whether they transfer to work outside games.

9/2Wed
  1. Hugging Face Blog36

    Hugging Face introduces BenchMIRT to audit what LLM benchmarks actually measure

    Hugging Face has released BenchMIRT, a multidimensional IRT method for auditing LLM benchmarks at the individual-prompt level and separating mixed ability signals within a benchmark. Trained on results from 100 LLMs across 16 benchmarks and more than 34,000 questions, it independently recovered two dominant dimensions—safety and general reasoning—without being told which benchmarks measured them. The analysis finds that scores on safety benchmarks such as BBQ and WMDP correlate more strongly with general reasoning, suggesting a single benchmark score may combine several signals.

8/21Fri
  1. Hugging Face Blog60

    Hugging Face research measures benchmark optimization in speech recognition

    Hugging Face has published research measuring benchmark optimization in speech recognition through three tests: consensus disagreement, masked-entity retrieval, and spelling switches. The study evaluates 11 open-source ASR models.

    Why it matters: The study presents three reproducible probes and quantifies how often ASR models reproduce incorrect benchmark transcripts, with direct implications for evaluation design.

8/17Mon
8/13Thu
  1. Microsoft Research36

    MindTopo exposes spatial-reasoning gaps in multimodal LLMs

    Microsoft Research has introduced MindTopo, a benchmark for topological reasoning in multimodal LLMs. It covers connectivity, separation, ordering, containment, and knot relations, across static reasoning and interactive planning. Current models are substantially better at static recognition than interactive planning, but both remain far below human performance; failures occur more often in planning than perception. Image and video generation helps only when topology is preserved within a single frame and remains unreliable across multi-step operations.

8/11Tue
8/10Mon
7/27Mon
7/16Thu