Skip to content

Multimodal

Beyond text: models and products for visual understanding, mixed image and text, and audio and video in and out.

63 picksLast 30 days 28 itemsCollected 216 items

New

Picks archive · page 3

7/21Tue
7/16Thu
  1. Lilian Weng · X65

    Inkling is our open weights model

    Inkling is Lilian Weng’s organization’s open weights model. It aims to serve as a foundation with solid performance across broad categories of capabilities, for practical use and customization. Weng invites users to try it on Tinker.

    Why it matters: The release positions Inkling as a foundation for practical customization, with full weights and fine-tuning availability giving concrete context for its intended use.

7/15Wed
  1. Hugging Face Blog88

    Thinking Machines releases Inkling family of open multimodal models

    Thinking Machines Lab has released the open-source Inkling family on Hugging Face. Inkling is an MoE model with about 1T total and 41B active parameters, natively supporting image, audio, and text inputs with a 1M-token context. The release also includes Inkling-Small, with 276B total and 12B active parameters.

    Why it matters: The article gives Inkling's architecture, deployment setup, and benchmark comparisons, helping readers assess the practical requirements of an approximately 1T-parameter open multimodal model.

7/1Wed
6/9Tue
  1. Google DeepMind78

    Google DeepMind releases Gemma 4 12B, a unified encoder-free multimodal model

    Google DeepMind has released Gemma 4 12B, an agentic multimodal model aimed at laptops, positioned between the edge-focused E4B and the 26B MoE. It is the company's first medium-sized model with native audio input.

    Why it matters: Gemma 4 12B removes multimodal encoders and feeds visual and audio input directly into the LLM, giving readers a view of hardware requirements for local multimodal agents.

6/5Fri
  1. Google Research67

    Google Research studies passive heart-health monitoring with a phone camera

    In a paper published in Nature, Google Research describes PHRM, a system that uses the front-facing camera to record a few seconds of video after face unlock and estimate heart rate and resting heart rate in the background using deep learning.

    Why it matters: Google passively measures heart rate with a phone's front camera after unlock and has released its largest phone-video dataset to date along with a pretrained model.

5/18Mon
  1. Google DeepMind62

    Google DeepMind brings real-world Street View scenes to Project Genie

    Google DeepMind has added Street View scene anchoring to the experimental Project Genie prototype. It can generate interactive worlds based on real locations in the United States, with optional styles such as Ocean World, Desert Sands, Stone Age, and B&W film.

    Why it matters: The source explains Project Genie's capabilities and availability with Street View, offering a view of how world models may be deployed.

  2. Google DeepMind85

    Google DeepMind releases Gemini Omni Flash for video generation from multimodal inputs

    Google DeepMind has released Gemini Omni Flash, the first model in the Omni family. It can combine images, audio, video, and text as inputs to generate high-quality video and supports iterative video editing through natural-language conversation.

    Why it matters: The source describes the first Omni model's input and output capabilities and where it is available, helping readers assess the practical reach of multimodal video generation.

5/17Sun
  1. Google DeepMind67

    Google expands SynthID and C2PA content verification to Search, Gemini, and Chrome

    Google has announced that it is extending content transparency and verification tools across Search, Gemini, Chrome, Pixel, and Cloud. The Gemini app already supports SynthID verification for images, video, and audio, with 50 million uses to date. The feature is coming to Search today and Chrome in the coming weeks.

    Why it matters: The source details where SynthID and C2PA verification are available across Search, Gemini, and Chrome, helping readers assess the practical reach of provenance tools.

5/16Sat
  1. Google DeepMind88

    Google DeepMind releases Gemini 3.5 Flash for agents and coding

    Google DeepMind has released Gemini 3.5 Flash, the first model in the Gemini 3.5 family, positioning it as a frontier model for agents and coding. It is available starting today in the Gemini app, AI Mode in Google Search, Google Antigravity, the Gemini API, and Gemini Enterprise.

    Why it matters: Gemini 3.5 Flash beats 3.1 Pro on coding and agent benchmarks and includes speed and cost comparisons, helping readers assess its potential for long-horizon tasks.

4/30Thu
  1. Google DeepMind62

    Google DeepMind launches AI co-clinician research program

    Google DeepMind has announced an AI co-clinician research program exploring a “triple-care” model in which AI agents assist with patient care under clinical supervision. In blind evaluation of 98 real primary-care queries, the system had no critical errors in 97 cases, and doctors preferred its responses to those from existing evidence-synthesis tools. It also outperformed other frontier models on open-ended medication questions in OpenFDA's RxQA.

    Why it matters: Google DeepMind reports results from physician blind reviews, RxQA, and simulated remote consultations, offering a view of current medical-AI capabilities.

4/13Mon
  1. Google DeepMind71

    Google DeepMind releases Gemini Robotics-ER 1.6

    Google DeepMind has released Gemini Robotics-ER 1.6, a major upgrade to its reasoning-first robotics model. It improves spatial reasoning and multi-view understanding and adds the ability to read complex instruments such as pressure gauges and level meters.

    Why it matters: The source details changes in spatial reasoning, multi-view understanding, and instrument reading, along with access information, giving a view of progress in high-level robot reasoning.

4/3Fri
3/25Wed
  1. Google Research62

    Google releases Vibe Coding XR, turning natural language into Android XR apps with Gemini and XR Blocks

    Google has released the Vibe Coding XR workflow, combining Gemini's long-context reasoning with the XR Blocks framework, based on WebXR, three.js, and LiteRT.js, to turn natural language directly into physics-aware Android XR apps. Google says the process takes under 60 seconds.

    Why it matters: Google combines Gemini and XR Blocks into a workflow for generating XR apps from a single sentence and reports preliminary VCXR60 benchmark results, offering a view of the changing barrier to spatial-computing prototypes.

3/17Tue
  1. Mistral AI78

    Mistral releases Mistral Small 4, unifying reasoning, multimodal, and coding capabilities

    Mistral AI has released Mistral Small 4, the next major version of the Mistral Small family and its first model to combine Magistral's reasoning, Pixtral's multimodal, and Devstral's agentic-coding capabilities in one model. It is licensed under Apache 2.0.

    Why it matters: Mistral has combined reasoning, multimodal, and coding capabilities in one open small model, with adjustable reasoning effort and deployment requirements to help assess implementation costs.

12/17Wed
  1. Mistral AI65

    Mistral releases OCR 3, with a 74% win rate over its predecessor on forms and handwriting

    Mistral AI has released Mistral OCR 3, reporting a 74% overall win rate against Mistral OCR 2 on forms, scanned documents, complex tables, and handwriting. The company says it outperforms both enterprise document-processing and AI-native OCR solutions in accuracy.

    Why it matters: The official post compares win rates with the previous version on forms, handwriting, and tables and gives pricing per thousand pages, helping readers assess document-parsing costs and use cases.

12/3Wed
  1. Mistral AI82

    Mistral releases Mistral 3 family, from 675B Mistral Large 3 to 3B, 8B, and 14B Ministral 3

    Mistral has released the Mistral 3 family: dense small models at 3B, 8B, and 14B, plus Mistral Large 3, a sparse MoE model with 41B active and 675B total parameters. All are open-source under Apache 2.0.

    Why it matters: Mistral has released Apache-2.0 weights ranging from 3B to 675B, giving readers a view of open-model options from edge devices to frontier scale.

7/17Thu
  1. Mistral AI62

    Mistral adds Deep Research, voice mode, and more to Le Chat

    Mistral has introduced several new features in Le Chat, including a preview of Deep Research mode, voice mode powered by the new Voxtral model, multilingual reasoning supported by the Magistral reasoning model, Projects for organizing conversations, and image editing in partnership with Black Forest Labs.

    Why it matters: Mistral has brought deep research, voice, reasoning, and image editing into Le Chat at once, giving readers a view of the assistant's expanding capabilities.

5/7Wed
  1. Mistral AI72

    Mistral AI releases Mistral Medium 3 for low-cost enterprise deployment

    Mistral AI has released Mistral Medium 3, which it says reaches at least 90% of Claude Sonnet 3.7's performance on several benchmarks. Pricing is $0.40 per million input tokens and $2 per million output tokens.

    Why it matters: The official release gives Mistral Medium 3's pricing, comparison model, and deployment requirements, helping readers assess the cost of enterprise self-hosting.

3/17Mon
  1. Mistral AI71

    Mistral releases Mistral Small 3.1 with multimodal support and 128K context

    Mistral AI has released Mistral Small 3.1, improving text performance and multimodal understanding over Mistral Small 3. It extends the context window to 128K tokens, runs at 150 tokens per second, and is open-source under Apache 2.0.

    Why it matters: Mistral reports comparisons for a small model across text, multimodal, and long-context tasks, helping readers assess on-device deployment options.