Skip to content
K = 0.731
All news · Today4 items

#Benchmarks

All collected and published items, ordered by time. Picks are a selected subset.
7/15Wed
  1. Hugging Face Blog62

    Hume releases Real World VoiceEQ to measure human quality in voice AI

    Hume has released Real World VoiceEQ, a benchmark for human quality in voice interaction. It covers more than 40 proprietary and open-source voice models, over 15 evaluation dimensions, and more than 60 metrics across ASR, TTS, speech-to-speech, and voice understanding.

    Why it matters: Hume built a voice-AI benchmark from more than a million human ratings, showing why current metrics can overestimate real conversational performance.

7/7Tue
  1. Hugging Face Blog74

    LeRobot v0.6.0 released: imagine, evaluate, improve

    Hugging Face has released LeRobot v0.6.0, adding world-model policies (VLA-JEPA, FastWAM, LingBot-VA), a new set of VLA models (GR00T N1.7, MolmoAct2, EO-1, EVO1, Multitask DiT), and a unified reward-model API (Robometer, TOPReward) to the robotics-learning loop.

    Why it matters: The release brings world-model policies, a reward-model API, and six simulation benchmarks into one CLI, giving readers a view of how production-ready the robotics-learning loop is becoming.

7/6Mon
7/1Wed
6/30Tue
6/25Thu
  1. Google Research37

    Google Research studies how reasoning unlocks LLMs' parametric knowledge

    A paper from Google Research at COLM 2026 finds that reasoning traces can unlock parametric knowledge in LLMs that they could not otherwise access, even for simple single-hop factual questions. Tests on Gemini-2.5 Flash and Pro and Qwen3-32B identify two mechanisms: extra tokens act as a “computation buffer,” and “fact priming” builds a retrieval bridge by generating related facts.

6/15Mon
  1. Import AI42

    Import AI 461: Sequent says 'alignment is not on track,' plus FrontierCode and synthetic research interns

    The alignment team at the UK's AI Security Institute and Timaeus have jointly founded Sequent, a nonprofit research organization. It argues that “alignment is not on track” and plans to grow to 40–80 full-time staff within two years, with an initial fundraising target of $100–150 million. Research areas include scalable oversight, learning theory, heuristic argumentation, game theory, and personas.

4/14Tue
  1. Google Research50

    Google introduces Vantage, using generative AI to assess future skills

    Google has released Vantage as a research experiment. It uses generative AI to create simulated conversations that assess high-school and university students' future skills, including problem-solving and collaboration. It is available for English registration through Google Labs. An Executive LLM dynamically guides the conversation and an AI Evaluator scores it against a rubric. In a study with New York University involving 188 participants aged 18–25, AI scores aligned with human experts nearly as well as experts aligned with one another.

4/3Fri
  1. Google Research38

    Google Research evaluates how closely LLM behavioral tendencies align with humans

    Google Research proposes a framework using situational judgment tests (SJTs), adapted from psychology scales, to assess how closely LLM behavioral tendencies align with human consensus across four traits including empathy and emotional regulation. A large-scale analysis of 25 LLMs finds two kinds of gap: models' tendencies can diverge from annotator consensus, and models may fail to cover the range of human opinions when there is no consensus. Small models under 120B and frontier closed models align nearly perfectly when humans unanimously agree, but alignment levels off just above 80% when consensus is below 90%.

4/1Wed
3/17Tue
4/9Wed