Skip to content
K = 0.731
All news · Today0 items

#Multimodal

All collected and published items, ordered by time. Picks are a selected subset.
Research

Research papers, technical reports, benchmarks and datasets.

10/15Thu
  1. Artificial Analysis · X56

    Claude Opus 5.5 and Grok 4.7 reproduce 72 Eiffel Tower names in offline 3D builds

    Artificial Analysis asked AI models to build the Eiffel Tower in 3D offline. Claude Opus 5.5 and Grok 4.7 included all 72 inscribed scientists’, engineers’, and mathematicians’ names in the correct order, although the shared brief never mentioned them. Claude Opus 5.5 also matched each set to its orientation in Paris. Both models worked as agents without internet access, and the names first appeared in their generated code, never in files or tool outputs they read.

  2. Hugging Face Blog73

    HiRO-ACEv2 simulates extreme weather at 3 km resolution for a fraction of the cost

    Ai2 introduced HiRO-ACEv2, an updated climate emulator that generates global 3 km rain, wind, and pressure data at about 1/300th the cost of X-SHiELD. Given coarsened X-SHiELD data, its HiROv2 downscaling model matches high-resolution statistics to within 3%, even for events occurring once in a million data points. The system currently works only with ACE tuned to match X-SHiELD and cannot yet generate high-resolution data for other climates; downloadable models, tutorial notebooks, and a web demo are available.

    Why it matters: The comparison with X-SHiELD gives researchers a concrete cost and accuracy reference for extreme weather simulations, with applicability currently limited to the climate represented in training.

10/14Wed
  1. Ars Technica · AI74

    Researchers report jailbreak vulnerabilities in OpenAI and Google image models

    White Circle researchers report that their RISE attack method bypassed safeguards in DALL·E 3, Nano Banana 2, and GPT-Image-2 to generate explicit images about 13 percent of the time. The method uses widely available models to iteratively develop and evaluate jailbreak strategies. White Circle told Ars Technica that subsequent testing found ChatGPT Images 2.5 and Nano Banana 2.1 significantly more vulnerable than their predecessors; Google and OpenAI did not respond to requests for comment.

10/13Tue
  1. Artificial Analysis · X56

    Artificial Analysis launches AA-3D to benchmark AI models building interactive 3D scenes

    Artificial Analysis launched AA-3D, a benchmark in which AI agents turn 55 written briefs across seven domains into interactive three.js scenes for blind human comparisons. Over 10,000 counted votes from over 1,000 panelists rank 21 models by Elo with 95% confidence intervals; Claude Sonnet 5.5, Claude Opus 5.5, GPT-6 Astra, and Claude Fable 5.1 are statistically tied at the top. Models receive up to three hours and 250 turns, and reported costs range from $0.20 to $32.08 per scene.

10/12Mon
  1. Import AI34

    Import AI 476: New Benchmarks Test Visual Reasoning and Research Taste

    Import AI 476 covers Humanity’s Sixth Sense, a benchmark from Elorian and Scale AI that tests intuitive visual reasoning. People scored 93.1 aggregate pass@1, while GPT 6 Astra scored 53.6; GLM-5.3-flash and Kimi K3 scored 26.8 and 25.5. The newsletter also examines how AI could accelerate space development and reports on TasteVal, a benchmark of AI systems’ research taste.

10/11Sun
  1. 量子位71

    He Kaiming team releases VISTA visual harness for multimodal reasoning

    He Kaiming's team published VISTA, a visual-native harness that lets existing multimodal models observe environments, store raw frames outside context, and recall, zoom or inspect them during reasoning without retraining. Paired with VISTA, Claude Opus 5.0 completed all 25 public ARC-AGI-3 games with a perfect 100 action-efficiency score, using 57.4% fewer actions than the human first-play baseline, while GPT-5.6 Sol scored 99. The team also tested browser games, mazes and connect-the-dots tasks, and lists embodied tasks as future work.

10/9Fri
  1. 量子位62

    MirroS launches AgentGarten: code environments and real-time neural rendering let agents learn by playing

    The MirroS team launched AgentGarten, connecting an executable code environment to a real-time neural renderer: code defines the physics, while the model generates visual feedback, enabling continuous agent interaction at more than 30 fps. In a recreated hide-and-seek experiment, hiders learned to move barriers into cover by round 4, while seekers learned to use a ramp to climb over a wall by round 10; comparable OpenAI experiments in 2019 took about 25 million and 100 million rounds, respectively. The team also published a blog, technical report, code repository, and project site.

10/8Thu
  3. vLLM · X62

    vLLM-Omni technical report proposes a unified serving runtime for omni-modality generation

    The vLLM project shared the vLLM-Omni technical report, describing a unified serving runtime for omni-modality generation. It argues that speech assistants, visual generation, world models and robot loops diverge from a single text decode loop into multi-stage autoregressive pipelines, iterative diffusion and stateful sessions, which LLM servers and diffusion stacks each cover only partially. vLLM-Omni acts as a shared control plane with an orchestrator, specialized engines, a connector and one session path. Paper and repo links are included.

10/3Sat
10/2Fri
  1. 量子位62

    Kaiming He's team proposes NAT-ARC: ImageNet pretraining for a purely visual ARC solver

    Kaiming He's team proposed NAT-ARC, a purely visual ARC-solving system that does not rely on LLMs. It pretrains an encoder with MAE on ImageNet and transfers it to abstract grid reasoning. The best single model has 0.6B parameters and reaches 63.4% pass@2 on ARC-1; an ensemble reaches 70.2%, with about one-quarter the parameters of the specialized LLM system The ARChitects, which uses an 8B model and reaches 71.6%.

9/30Wed
9/25Fri
  1. Google Research65

    Google Research introduces AI video co-director framework for coherent long-form video generation

    Google Research has introduced an AI video co-director multi-agent framework, an orchestration layer on top of Gemini and Veo. It models long-form video generation as a problem of global optimization and world-state tracking to mitigate semantic drift and cascading failures.

    Why it matters: Google breaks long-video generation into four composable frameworks and reports continuity improvements across several benchmarks, illustrating how multi-agent orchestration can reduce semantic drift.

8/26Wed
  1. Google Research26

    Google introduces AgentHands to generate spatial gestures for XR agent conversations

    Google has released AgentHands, a research prototype described in a paper at CHI 2026, that maps LLM reasoning output to gestures synchronized with speech in XR. This lets an agent point things out and demonstrate objects or steps in the user's physical space. The system comprises four modules: object registration, a gesture-event library, gesture-embedding inference, and synchronized execution in a headset. Gestures are categorized by dimensions including left or right hand, spatial anchoring, and VFX. In a controlled study with N=12, AgentHands significantly improved spatial reference over a voice-only baseline.

8/22Sat
  1. Jim Fan · X69

    NVIDIA and Berkeley open-source T-Rex methodology for tactile robot learning

    Jim Fan announces T-Rex, an open-source tactile robot learning methodology developed through a collaboration between NVIDIA and Berkeley. Its mixture-of-transformer architecture pairs a slow visuomotor expert with a fast tactile expert running at 4 touch ticks per vision tick; a 50-hour (~5,500 episodes) synchronized robot play dataset collected on tactile hand hardware with 22 degrees of freedom is available today on HuggingFace. Extending EgoScale, its training recipe combines human egocentric video pretraining with tactile robot play mid-training, which Fan says helps bridge contact-free pretraining and contact-rich manipulation.

    Why it matters: T-Rex offers a transferable approach to contact-rich robot learning by combining human video pretraining, synchronized tactile robot data, and separate visual and tactile processing rates.

8/17Mon
  1. Google Research58

    Google Research introduces PhotoScan to estimate body composition from phone photos and predict insulin resistance

    Google Research has proposed PhotoScan, a deep-learning framework that estimates body-fat percentage, the android-to-gynoid fat ratio (A/G), and the visceral-to-subcutaneous fat area ratio (V/S) from ordinary 2D phone photos. It was pretrained on more than 35,000 UK Biobank records, fine-tuned on a cohort of 677 adults, and validated on an independent cohort of 132 people.

8/13Thu
  1. Microsoft Research36

    MindTopo exposes spatial-reasoning gaps in multimodal LLMs

    Microsoft Research has introduced MindTopo, a benchmark for topological reasoning in multimodal LLMs. It covers connectivity, separation, ordering, containment, and knot relations, across static reasoning and interactive planning. Current models are substantially better at static recognition than interactive planning, but both remain far below human performance; failures occur more often in planning than perception. Image and video generation helps only when topology is preserved within a single frame and remains unreliable across multi-step operations.

8/12Wed
  1. Google Research72

    Google researchers advance AMIE to expert-level clinical interviews with audio and video

    Google Research has released AMIE (Video), which uses Gemini and Project Astra for real-time video clinical interviews. It can perceive nonverbal cues, guide virtual physical examinations, and reason about diagnoses in real time.

    Why it matters: AMIE extends from text to live video consultations, showing how an asynchronous multi-agent architecture can balance conversational latency with clinical reasoning.

  2. Microsoft Research36

    Microsoft Research introduces CARE-X, a clinically oriented radiology VLM combining auxiliary supervision, reward alignment, and tool-assisted measurement

    Microsoft Research has released CARE-X, a research model and unified chest X-ray vision-language model that can generate reports, make structured diagnostic predictions, and localize lesions. It uses reinforcement learning (DAPO) to reward clinical correctness across multiple tasks.

7/23Thu
  1. Google Research62

    Google introduces SymptomAI, a conversational AI agent for everyday symptom assessment

    Google Research has published SymptomAI, a study in which five Gemini Flash 2.0-based conversational agents interviewed 13,917 participants and evaluated differential diagnoses (DDx).

    Why it matters: Google's randomized study compares five questioning strategies across 13,917 people and, for the first time, compares diagnoses with Fitbit physiological signals, offering evidence on real-world conversational triage.

6/13Sat
  1. Google Research32

    Google studies how AI can help people understand skin concerns

    Google reports on consumer understanding of dermatology AI tools. Among 2,345 participants, more than 62% of those using AI tried to name a skin condition, with 23% accuracy—nearly three times the control group's 8%. The “Wizard of Oz” group reached 36% accuracy. But the AI group showed no statistically significant improvement in deciding what care to seek and was slightly more likely than dermatologists to recommend less urgent care (30% vs. 27%).

6/5Fri
  1. Google Research67

    Google Research studies passive heart-health monitoring with a phone camera

    In a paper published in Nature, Google Research describes PHRM, a system that uses the front-facing camera to record a few seconds of video after face unlock and estimate heart rate and resting heart rate in the background using deep learning.

    Why it matters: Google passively measures heart rate with a phone's front camera after unlock and has released its largest phone-video dataset to date along with a pretrained model.

4/30Thu
  1. Google DeepMind62

    Google DeepMind launches AI co-clinician research program

    Google DeepMind has announced an AI co-clinician research program exploring a “triple-care” model in which AI agents assist with patient care under clinical supervision. In blind evaluation of 98 real primary-care queries, the system had no critical errors in 97 cases, and doctors preferred its responses to those from existing evidence-synthesis tools. It also outperformed other frontier models on open-ended medication questions in OpenFDA's RxQA.

    Why it matters: Google DeepMind reports results from physician blind reviews, RxQA, and simulated remote consultations, offering a view of current medical-AI capabilities.

4/9Thu
  1. Google Research36

    Google introduces PaperVizAgent and ScholarPeer academic agents

    Google has released two AI agents for academic workflows: PaperVizAgent generates figures for papers, while ScholarPeer automates peer review. PaperVizAgent coordinates five agents for retrieval, planning, style, visualization, and review. It scored 60.2 on a 0–100 evaluation, ahead of GPT-Image-1.5, Nano-Banana-Pro, and Paper2Any, and was the only framework to exceed the human baseline of 50.0.