Skip to content
K = 0.731
← All topics

Multimodal

Beyond text: models and products for visual understanding, mixed image and text, and audio and video in and out.

63 picksLast 30 days 28 itemsCollected 216 items

New

Picks archive · page 2

9/29Tue
  1. Latent Space80

    AMD acquires World Labs for $8.2 billion; Atlas addresses sparse reconstruction

    AMD acquired World Labs for $8.2 billion. The official blog says that since its founding in 2024, World Labs has built a model-training team for image, video, and spatial reconstruction, and has advanced robotics simulation through its acquisition of SceniX.

    Why it matters: AMD's $8.2 billion acquisition of World Labs and its disclosure of Atlas's sparse-reconstruction progress offer a view into the integration of spatial intelligence and robotics simulation.

  2. Latent Space78

    AINews: Opus 5.5 excels at generating explainer videos

    Latent Space's AINews roundup of developments from September 24–25 says community feedback was positive after Opus 5.5 launched that week, especially for generating explainer videos with code.

    Why it matters: The weekly roundup offers a quick look at model and agent-infrastructure developments, including community feedback on Opus 5.5's code-generated videos and how vendors position their new models.

  3. 量子位82

    AMD to acquire Fei-Fei Li's World Labs for $8.2 billion in an all-stock deal; Li to become chief scientist

    AMD announced an all-stock acquisition of Fei-Fei Li's World Labs for $8.2 billion. The deal is expected to close by the end of 2026, subject to regulatory approval. After closing, Li will join AMD as executive vice president and chief scientist, reporting directly to Lisa Su; co-founders Justin Johnson and Ben Mildenhall will continue leading the team as it joins AMD.

    Why it matters: The full deal details and the companies' year-and-a-half collaboration are laid out, helping readers understand why a chipmaker would acquire a spatial-intelligence team.

9/25Fri
  1. Google Research65

    Google Research introduces AI video co-director framework for coherent long-form video generation

    Google Research has introduced an AI video co-director multi-agent framework, an orchestration layer on top of Gemini and Veo. It models long-form video generation as a problem of global optimization and world-state tracking to mitigate semantic drift and cascading failures.

    Why it matters: Google breaks long-video generation into four composable frameworks and reports continuity improvements across several benchmarks, illustrating how multi-agent orchestration can reduce semantic drift.

  2. Google DeepMind71

    Google DeepMind releases Gemini 3.8 Live with Live Avatar

    Google DeepMind has released Gemini 3.8 Live with Live Avatar, combining real-time video generation and voice to give Gemini's conversational model a near-real-time visual avatar, with accurate lip-sync, natural expressions, and smooth turn-taking.

    Why it matters: Gemini 3.8 Live adds real-time video avatars, giving readers a view of how multimodal conversation is being applied in enterprise settings and where the feature is available.

9/23Wed
  1. Google DeepMind71

    Google DeepMind releases Gemini 3.8 Flash TTS and Flash-Lite TTS

    Google DeepMind has released two speech-generation models: Gemini 3.8 Flash TTS for character design and fine-grained creative control, and Gemini 3.8 Flash-Lite TTS for high-volume voiceovers and voice agents.

    Why it matters: The article outlines the models' voice cloning, line-by-line direction, and benchmark results, helping readers assess their suitability for dubbing and podcast production.

  2. Qwen · X78

    Qwen launches Qwen Intelligence with three state-of-the-art mobile agents

    Qwen has launched Qwen Intelligence with three state-of-the-art agents for mobile. The first, Mobile Planner Agent, handles task planning and decomposition, and is evaluated on MobilePA-Bench.

    Why it matters: The official release gives benchmark results and access points for three mobile agents, helping readers assess the current capabilities of on-device agents.

  3. Qwen · X71

    Alibaba releases Qwen-Audio-3.1 family, including TTS-Next and ASR-Next

    Alibaba has released Qwen-Audio-3.1 with upgrades to ASR, TTS, and Realtime, plus two new models, TTS-Next and ASR-Next. Five models now cover audio understanding, generation, interaction, and creation.

    Why it matters: The article lists the capabilities of all five audio models and their price reductions, helping readers assess voice-stack choices and cost changes.

9/10Thu
  1. DeepSeek · X68

    DeepSeek releases DeepSeek-V4.1-Flash

    DeepSeek has released DeepSeek-V4.1-Flash, describing it as smarter, faster, and more efficient. It is the smallest model in a new architecture family, with native visual understanding, and is designed for stronger capabilities, faster inference, and higher throughput, with the architecture able to scale to larger models.

    Why it matters: As the smallest model in DeepSeek's new architecture family, its native visual understanding and higher inference throughput offer a view of its lightweight multimodal positioning.

9/2Wed
  1. Google DeepMind74

    Google DeepMind adds agentic video understanding to Gemini 3.7 Flash and other models

    Google DeepMind has introduced agentic video understanding for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite. Video analysis uses up to 88% fewer tokens and costs up to 66% less, while accuracy improves by up to 7%.

    Why it matters: The article gives token and cost reductions and access details, helping readers assess how the economics of long-video analysis are changing.

8/28Fri
  1. Google DeepMind68

    Google DeepMind releases Gemini Omni 1.1 Flash

    Google DeepMind has released Gemini Omni 1.1 Flash, offering developers video-generation controls through the Gemini API in Google AI Studio.

    Why it matters: The article details capabilities such as extending scenes, interpolating between first and last frames, and 4K upscaling, helping readers assess how controllable video-generation workflows are becoming.

8/22Sat
  1. Jim Fan · X69

    NVIDIA and Berkeley open-source T-Rex methodology for tactile robot learning

    Jim Fan announces T-Rex, an open-source tactile robot learning methodology developed through a collaboration between NVIDIA and Berkeley. Its mixture-of-transformer architecture pairs a slow visuomotor expert with a fast tactile expert running at 4 touch ticks per vision tick; a 50-hour (~5,500 episodes) synchronized robot play dataset collected on tactile hand hardware with 22 degrees of freedom is available today on HuggingFace. Extending EgoScale, its training recipe combines human egocentric video pretraining with tactile robot play mid-training, which Fan says helps bridge contact-free pretraining and contact-rich manipulation.

    Why it matters: T-Rex offers a transferable approach to contact-rich robot learning by combining human video pretraining, synchronized tactile robot data, and separate visual and tactile processing rates.

8/21Fri
  1. DeepSeek · X78

    DeepSeek-V4-Flash-Vision-Exp launches on the DeepSeek API platform

    DeepSeek has launched the experimental multimodal model DeepSeek-V4-Flash-Vision-Exp on its API platform. Its text capabilities match DeepSeek-V4-Flash across agents, reasoning, and world knowledge.

    Why it matters: The official release gives the new model's text-capability alignment with V4-Flash and improvements on multimodal-agent benchmarks, helping readers assess its readiness for multimodal agents.

8/12Wed
  1. Google DeepMind71

    Google DeepMind releases SL2T sign-language-to-text model, first rolling out in Gboard and Live Transcribe on Pixel 11

    Google DeepMind has released SL2T, a multilingual sign-language-to-text model, bringing sign-language AI to consumer products for the first time. On Pixel 11, Gboard and Live Transcribe support dictation from American Sign Language (ASL) to English; more devices and languages are planned.

    Why it matters: The source describes the capabilities and limits of a multilingual sign-to-text model and where it is available, helping readers assess sign-language translation in real products.

  2. Google Research72

    Google researchers advance AMIE to expert-level clinical interviews with audio and video

    Google Research has released AMIE (Video), which uses Gemini and Project Astra for real-time video clinical interviews. It can perceive nonverbal cues, guide virtual physical examinations, and reason about diagnoses in real time.

    Why it matters: AMIE extends from text to live video consultations, showing how an asynchronous multi-agent architecture can balance conversational latency with clinical reasoning.

8/10Mon
  1. Hugging Face Blog82

    Meta releases open-source 30B multimodal Muse Glimmer for local agents

    Meta has released Muse Glimmer, a 30B-parameter multimodal model distilled from Muse and licensed under Apache 2.0. It targets local-agent use cases such as coding, document analysis, and personal assistance.

    Why it matters: Meta has open-sourced a 30B multimodal model with reproducible details on its architecture, benchmarks, local deployment, and self-improvement demonstrations.

8/4Tue
  1. Mistral AI66

    Mistral AI releases Shieldstral, a 3B open-weight multimodal safety classifier

    Mistral AI has released Shieldstral, a 3B open-weight multimodal safety classifier available under Apache 2.0 and runnable on a single 16 GB NVIDIA GPU. It frames content moderation as a policy-adaptive question-answering task: policies are written in natural language at inference time, without retraining. It handles text and images together and outputs calibrated continuous safety scores. Mistral says it matches models up to seven times larger on text safety and sets a new state of the art in multimodal moderation.

    Why it matters: The 3B open safety classifier makes moderation policy a natural-language prompt at inference time, offering a view of what smaller models can do for content moderation.

7/30Thu
  1. Google DeepMind71

    Google DeepMind releases Gemini Robotics ER 2 embodied-reasoning model

    Google DeepMind has released Gemini Robotics ER 2, positioned as the robot's high-level brain: it converses with people, understands the physical world, and plans multi-step tasks before handing execution to a lower-level VLA model. It can also call tools such as Google Search natively.

    Why it matters: The official release gives concrete figures for ER 2's progress classification and temporal localization, along with access details, helping readers assess its limits on long tasks.

7/28Tue
  1. Google DeepMind80

    Google DeepMind releases Gemini Robotics 2 for full-body robot intelligence

    Google DeepMind has released Gemini Robotics 2 with three models: Gemini Robotics 2, a vision-language-action model for translating visual and language input into motion; Gemini Robotics ER 2 for embodied reasoning; and Gemini Robotics On-Device 2, which runs locally on devices.

    Why it matters: The article explains the three models' roles and access points, helping readers assess pathways to full-body robot control and multi-robot collaboration.

7/23Thu
  1. Google Research62

    Google introduces SymptomAI, a conversational AI agent for everyday symptom assessment

    Google Research has published SymptomAI, a study in which five Gemini Flash 2.0-based conversational agents interviewed 13,917 participants and evaluated differential diagnoses (DDx).

    Why it matters: Google's randomized study compares five questioning strategies across 13,917 people and, for the first time, compares diagnoses with Fitbit physiological signals, offering evidence on real-world conversational triage.