Skip to content

Multimodal

Beyond text: models and products for visual understanding, mixed image and text, and audio and video in and out.

63 picksLast 30 days 28 itemsCollected 216 items

New

Picks about Multimodal

10/16Fri
  1. Ollama · X72

    NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is now on Ollama

    NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is now on Ollama. This vision-language model (VLM) is designed to run everyday agentic tasks locally, using native 2-bit quantization to fit in 24GB of memory and run alongside other applications. ollama run nemotron-3.5-bolt

    Why it matters: The stated memory footprint provides a concrete deployment consideration for running local agentic workloads alongside other applications on the same machine.

  2. Hugging Face Blog87

    Meta releases Muse Spark Open, an open-weight multimodal mixture-of-experts model

    Meta’s Muse Spark Open is an open-weight mixture-of-experts model with approximately 816 billion parameters, around 42 billion active per token, and a context window of up to one million tokens. Trained for agentic coding, it accepts text, images, audio, and video and generates text. Hugging Face provides inference and deployment recipes, reports its own quantization tests, and demonstrates attention-layer LoRA fine-tuning with TRL while keeping the experts frozen.

    Why it matters: The deployment recipes clarify how sparse expert activation reduces per-token computation while retaining substantial memory requirements, helping developers assess hardware needs across checkpoint formats and serving frameworks.

  3. The Decoder81

    Nvidia releases Nemotron 3.5 Super VL for continuously running enterprise AI agents

    Nvidia released Nemotron 3.5 Super VL, an open multimodal model with 120 billion parameters and 12 billion active per token, for continuously running enterprise agents. Nvidia says it approaches Nemotron 3 Ultra’s accuracy on specified agent tasks with about one-fifth the parameters, and is releasing weights, datasets, and training recipes. The accompanying Nemotron 3.5 Bolt 74B-A9B 24GB targets local agents, while Bolt E4B 4GB is planned for later this month.

    Why it matters: The deployment details clarify how quantization, caching, and hardware requirements shape the practical tradeoffs of running continuous enterprise agents on company infrastructure.

10/15Thu
  1. NVIDIA Blog81

    NVIDIA Expands Nemotron Family With Super VL and Bolt Open Models

    NVIDIA released Nemotron 3.5 Super VL, an open 120-billion-parameter model for multimodal agents, and Nemotron 3.5 Bolt 74B-A9B 24GB for local agentic workloads; Bolt E4B 4GB is coming Oct. 23. Super VL targets tool use, business workflows and visual document understanding, with open weights, datasets and training recipes. Bolt 74B-A9B uses native 2-bit quantization to fit in 24GB of memory, while Bolt E4B targets physical and conversational agents.

    Why it matters: Open weights, datasets and training recipes give organizations a path to adapting agents to their own workflows while meeting local deployment and data residency requirements.

  2. Hugging Face Blog74

    Apertus 1.5 Release Notes Detail Multimodal Understanding, Thinking Mode, and Longer Context

    Project Apertus released Apertus 1.5 in 8B and 70B sizes, adding image and audio understanding, optional thinking, a 262,144-token context window, and improved instruction-following and tool use. The weights were released on Hugging Face on July 24, 2026 under Apache 2.0; the models generate only text, and spoken-language processing remains experimental. The team also publishes training data, recipes, intermediate checkpoints, and reproducible evaluation records.

    Why it matters: Published intermediate checkpoints, training mixtures, and evaluation settings provide a concrete basis for investigating how individual training stages change model behavior and for reproducing reported results.

  3. Hugging Face Blog73

    HiRO-ACEv2 simulates extreme weather at 3 km resolution for a fraction of the cost

    Ai2 introduced HiRO-ACEv2, an updated climate emulator that generates global 3 km rain, wind, and pressure data at about 1/300th the cost of X-SHiELD. Given coarsened X-SHiELD data, its HiROv2 downscaling model matches high-resolution statistics to within 3%, even for events occurring once in a million data points. The system currently works only with ACE tuned to match X-SHiELD and cannot yet generate high-resolution data for other climates; downloadable models, tutorial notebooks, and a web demo are available.

    Why it matters: The comparison with X-SHiELD gives researchers a concrete cost and accuracy reference for extreme weather simulations, with applicability currently limited to the climate represented in training.

10/14Wed
  1. 宝玉 · X76

    How Gemini 4 Argon Built a Japanese Temple Complex in Blender

    Baoyu describes a demonstration in which Gemini 4 Argon used Antigravity to build a Japanese temple complex in Blender from one prompt containing a detailed workflow. The prompt referenced a scene-building skill document and requested architectural plans, four Nano Banana reference images, and coordinated agents that checked renders against those references and iteratively revised the scene. The demonstrator requested animation and programmatically generated bell sounds and ambient audio the next day, making those later additions to the initial modeling workflow.

    Why it matters: The workflow offers a transferable approach to complex 3D generation by combining construction rules, visual references, agent coordination, and repeated comparison of rendered results.

  2. Google DeepMind · X73

    Gemini 4 Argon taught a robot to play cornhole just by watching a video

    Gemini 4 Argon taught a robot to play cornhole just by watching a video. It mapped the physics, coded its own simulation, and taught @Apptronik's Apollo humanoid to sink 7 throws in a row.

    Why it matters: The demonstration connects video observation, physics modeling, and a model-written simulation in a concrete example of teaching a humanoid robot a physical task.

  3. Google DeepMind94

    Google DeepMind launches Gemini 4 Argon for complex reasoning and agentic tasks

    Google DeepMind has launched Gemini 4 Argon globally through its API and selected Google products, with access depending on subscription or service tier. Introductory pricing is $2 per million input tokens and $10 per million output tokens, with cached input tokens at 95% off and a 1M unified input-output token limit through the Gemini API. Google reports improvements across coding, professional workflows, scientific research, and robotics, alongside stronger agent safeguards and prompt injection resistance.

    Why it matters: Google’s reported gains in code submissions and reductions in rollbacks connect Argon’s engineering capabilities to delivery speed and code stability in internal workflows.

10/9Fri
10/8Thu
  1. Mark Chen · X81

    OpenAI rolls out GPT-6 and Intelligent UI in ChatGPT for everyone

    OpenAI says GPT-6 and Intelligent UI are now rolling out in ChatGPT for everyone, offering fast, interactive answers and tools for users’ tasks. Mark Chen encourages users to try Intelligent UI, saying every ChatGPT completion now has the potential to create a custom UI and calling it phenomenal for learning new things. https://twitter.com/OpenAI/status/2107894997538525580

    Why it matters: The described custom interfaces could make learning more visual and interactive by placing explanations and task tools directly within the flow of a ChatGPT answer.

10/7Wed
  1. Josh Woodward · X74

    Nano Banana announces Nano Banana 2.1 with improved image generation and editing

    Nano Banana announced Nano Banana 2.1, claiming improvements over its previous models across the board, including visual design, mask-based editing, subject consistency, and more natural-looking imagery. It is available in @GeminiApp, @GoogleAIStudio, and more. Josh Woodward commented, “Mask-based editing is my favorite from today, much more coming soon!” https://twitter.com/NanoBanana/status/2107521202466046453

    Why it matters: The announcement frames the upgrade around visual design, editing, subject consistency, and natural appearance, providing a concrete comparison with previous models.

  2. Google DeepMind71

    Google DeepMind releases EmbeddingGemma 2, a lightweight open multimodal embedding model

    Google DeepMind released EmbeddingGemma 2, a 740M-parameter open model based on the Gemma 4 architecture and licensed under Apache 2.0. It maps text, images, audio, and video into a shared embedding space.

    Why it matters: EmbeddingGemma 2 maps text, images, audio, and video into one embedding space and reports concrete on-device memory and vector truncation figures, helping assess local multimodal retrieval.

10/6Tue
  1. Mistral AI80

    Mistral launches public preview of Mistral Large 4, a 1-trillion-parameter multimodal model

    Mistral released a public preview of Mistral Large 4, a natively multimodal model with 1 trillion parameters and 49 billion active parameters. Its weights will be opened at the end of the month.

    Why it matters: Mistral outlined the open-weight release schedule for its 1-trillion-parameter multimodal model and its investment in European compute, allowing comparison with mainstream open models.

  2. Mistral AI · X80

    Mistral releases Mistral Large 4, a natively multimodal 1T-parameter model

    Mistral AI released Mistral Large 4, codenamed Le Chonk, with 1T total parameters, native multimodality, and 49B active parameters. The company says it is the best open-weight model on aggregated benchmarks in Europe and the U.S. It reaches state of the art on workloads in cyber defense, manufacturing, and finance, and exceeds closed frontier models in visual grounding. Built end-to-end in Europe, it can be deployed from Europe through Mistral Cloud. The API is available today, and open weights are due at the end of October.

    Why it matters: Mistral specified a 1T-parameter, 49B-active open-weight model and a local European deployment path, offering a view of the open-weight field's scale and availability.

10/1Thu
  1. Google DeepMind88

    Google DeepMind releases frontier model Gemini 4 Argon

    Google DeepMind released frontier model Gemini 4 Argon, initially making it available through the Fairwind Program to trusted cybersecurity defenders before gradually expanding access. Pricing starts at $2 per million input tokens and $10 per million output tokens.

    Why it matters: Gemini 4 Argon raises the maximum output to 1M tokens and reports multiple benchmark results and internal deployment data, offering a view of its limits on long-running workflows.

9/30Wed
  1. The Decoder78

    AMD to acquire world-model startup World Labs for $8.2 billion

    AMD announced an all-stock acquisition of spatial intelligence startup World Labs for about $8.2 billion. The deal is expected to close by the end of 2026, subject to regulatory approval. World Labs founder Fei-Fei Li will join AMD's leadership team as executive vice president and chief scientist, reporting directly to CEO Lisa Su and leading frontier research.

    Why it matters: Beyond the price and leadership changes, the source outlines the technical case for world models and compares the ecosystem with Nvidia's.

9/29Tue
  1. Microsoft Research62

    Microsoft Research releases Quine, an AI research system for biological complexity

    Microsoft Research has released Quine, an AI research system designed for biological complexity. It combines a biological world model with an interactive harness connecting models, scientific tools, literature, and experimentalists. In work with the Broad Institute on pancreatic ductal adenocarcinoma, it predicted and prioritized thousands of compounds; the top-ranked compound produced the largest classical-to-ground-state transition in wet-lab experiments. The process from narrowing the search space to selecting a candidate for validation took one weekend.

    Why it matters: Microsoft Research describes Quine's multimodal biological world model and experimental loop, including wet-lab validation in pancreatic cancer compound screening.