Skip to content
K = 0.731
All news · Today0 items

#Benchmarks

All collected and published items, ordered by time. Picks are a selected subset.
First-party

Direct announcements from model vendors, companies and research institutions on their official sites, blogs and accounts.

10/8Thu
  1. Hugging Face Blog40

    TII launches Falcon-ASR, a 1.6B-parameter Arabic speech recognition model focused on Emirati dialects

    Abu Dhabi's Technology Innovation Institute (TII) released Falcon-ASR, a 1.6B-parameter speech recognition model focused on Arabic and Emirati dialects, with support for English, French, Spanish, and Portuguese as well. Its average WER across six Arabic test sets is 20.92%, 2.25 percentage points below the best public result of 23.17% in the leaderboard snapshot used. In an internal Emirati dialect evaluation, it achieved a 22.73% WER and 10.19% CER, outperforming Qwen3-Omni.

10/7Wed
  1. Microsoft Research22

    Microsoft Research: why AI evaluations can fail unexpectedly outside traditional benchmarks

    Jennifer Neville of Microsoft says models can fail unexpectedly when tested beyond traditional benchmarks. Evaluation is essential to pushing the performance boundaries of current AI systems and aligning them with user needs. She recommends paying close attention to data when using AI systems, since the data itself is often the first thing to examine when results differ from expectations.

10/6Tue
9/30Wed
  1. Hugging Face Blog60

    Hugging Face releases Open TTS Leaderboard for objective evaluation of multilingual TTS and voice cloning

    Hugging Face released Open TTS Leaderboard, which uses objective metrics including WER/CER, RTFx, TTFA, and speaker similarity to evaluate open-source TTS models, reducing evaluation cycles from weeks of voting to hours.

    Why it matters: Hugging Face replaces vote-based evaluation with objective metrics, showing how to compare open-source TTS models across languages and voice cloning.

9/23Wed
9/22Tue
9/2Wed
  1. Hugging Face Blog36

    Hugging Face introduces BenchMIRT to audit what LLM benchmarks actually measure

    Hugging Face has released BenchMIRT, a multidimensional IRT method for auditing LLM benchmarks at the individual-prompt level and separating mixed ability signals within a benchmark. Trained on results from 100 LLMs across 16 benchmarks and more than 34,000 questions, it independently recovered two dominant dimensions—safety and general reasoning—without being told which benchmarks measured them. The analysis finds that scores on safety benchmarks such as BBQ and WMDP correlate more strongly with general reasoning, suggesting a single benchmark score may combine several signals.

8/21Fri
  1. Hugging Face Blog60

    Hugging Face research measures benchmark optimization in speech recognition

    Hugging Face has published research measuring benchmark optimization in speech recognition through three tests: consensus disagreement, masked-entity retrieval, and spelling switches. The study evaluates 11 open-source ASR models.

    Why it matters: The study presents three reproducible probes and quantifies how often ASR models reproduce incorrect benchmark transcripts, with direct implications for evaluation design.

8/13Thu
  1. Microsoft Research36

    MindTopo exposes spatial-reasoning gaps in multimodal LLMs

    Microsoft Research has introduced MindTopo, a benchmark for topological reasoning in multimodal LLMs. It covers connectivity, separation, ordering, containment, and knot relations, across static reasoning and interactive planning. Current models are substantially better at static recognition than interactive planning, but both remain far below human performance; failures occur more often in planning than perception. Image and video generation helps only when topology is preserved within a single frame and remains unreliable across multi-step operations.

8/11Tue
7/16Thu
7/15Wed
  1. Hugging Face Blog62

    Hume releases Real World VoiceEQ to measure human quality in voice AI

    Hume has released Real World VoiceEQ, a benchmark for human quality in voice interaction. It covers more than 40 proprietary and open-source voice models, over 15 evaluation dimensions, and more than 60 metrics across ASR, TTS, speech-to-speech, and voice understanding.

    Why it matters: Hume built a voice-AI benchmark from more than a million human ratings, showing why current metrics can overestimate real conversational performance.

7/7Tue
  1. Hugging Face Blog74

    LeRobot v0.6.0 released: imagine, evaluate, improve

    Hugging Face has released LeRobot v0.6.0, adding world-model policies (VLA-JEPA, FastWAM, LingBot-VA), a new set of VLA models (GR00T N1.7, MolmoAct2, EO-1, EVO1, Multitask DiT), and a unified reward-model API (Robometer, TOPReward) to the robotics-learning loop.

    Why it matters: The release brings world-model policies, a reward-model API, and six simulation benchmarks into one CLI, giving readers a view of how production-ready the robotics-learning loop is becoming.

7/1Wed
6/30Tue
6/25Thu
  1. Google Research37

    Google Research studies how reasoning unlocks LLMs' parametric knowledge

    A paper from Google Research at COLM 2026 finds that reasoning traces can unlock parametric knowledge in LLMs that they could not otherwise access, even for simple single-hop factual questions. Tests on Gemini-2.5 Flash and Pro and Qwen3-32B identify two mechanisms: extra tokens act as a “computation buffer,” and “fact priming” builds a retrieval bridge by generating related facts.

4/14Tue
  1. Google Research50

    Google introduces Vantage, using generative AI to assess future skills

    Google has released Vantage as a research experiment. It uses generative AI to create simulated conversations that assess high-school and university students' future skills, including problem-solving and collaboration. It is available for English registration through Google Labs. An Executive LLM dynamically guides the conversation and an AI Evaluator scores it against a rubric. In a study with New York University involving 188 participants aged 18–25, AI scores aligned with human experts nearly as well as experts aligned with one another.

4/3Fri
  1. Google Research38

    Google Research evaluates how closely LLM behavioral tendencies align with humans

    Google Research proposes a framework using situational judgment tests (SJTs), adapted from psychology scales, to assess how closely LLM behavioral tendencies align with human consensus across four traits including empathy and emotional regulation. A large-scale analysis of 25 LLMs finds two kinds of gap: models' tendencies can diverge from annotator consensus, and models may fail to cover the range of human opinions when there is no consensus. Small models under 120B and frontier closed models align nearly perfectly when humans unanimously agree, but alignment levels off just above 80% when consensus is below 90%.

4/1Wed
3/17Tue
4/9Wed