Skip to content
2026.10.17 · Sat

#多模态

#多模态

First-party

Direct announcements from model vendors, companies and research institutions on their official sites, blogs and accounts.

10/16Fri
  1. Hugging Face Blog87

    Meta releases Muse Spark Open, an open-weight multimodal mixture-of-experts model

    Meta’s Muse Spark Open is an open-weight mixture-of-experts model with approximately 816 billion parameters, around 42 billion active per token, and a context window of up to one million tokens. Trained for agentic coding, it accepts text, images, audio, and video and generates text. Hugging Face provides inference and deployment recipes, reports its own quantization tests, and demonstrates attention-layer LoRA fine-tuning with TRL while keeping the experts frozen.

    Why it matters: The deployment recipes clarify how sparse expert activation reduces per-token computation while retaining substantial memory requirements, helping developers assess hardware needs across checkpoint formats and serving frameworks.

10/15Thu
  1. NVIDIA Blog81

    NVIDIA Expands Nemotron Family With Super VL and Bolt Open Models

    NVIDIA released Nemotron 3.5 Super VL, an open 120-billion-parameter model for multimodal agents, and Nemotron 3.5 Bolt 74B-A9B 24GB for local agentic workloads; Bolt E4B 4GB is coming Oct. 23. Super VL targets tool use, business workflows and visual document understanding, with open weights, datasets and training recipes. Bolt 74B-A9B uses native 2-bit quantization to fit in 24GB of memory, while Bolt E4B targets physical and conversational agents.

    Why it matters: Open weights, datasets and training recipes give organizations a path to adapting agents to their own workflows while meeting local deployment and data residency requirements.

  2. Hugging Face Blog74

    Apertus 1.5 Release Notes Detail Multimodal Understanding, Thinking Mode, and Longer Context

    Project Apertus released Apertus 1.5 in 8B and 70B sizes, adding image and audio understanding, optional thinking, a 262,144-token context window, and improved instruction-following and tool use. The weights were released on Hugging Face on July 24, 2026 under Apache 2.0; the models generate only text, and spoken-language processing remains experimental. The team also publishes training data, recipes, intermediate checkpoints, and reproducible evaluation records.

    Why it matters: Published intermediate checkpoints, training mixtures, and evaluation settings provide a concrete basis for investigating how individual training stages change model behavior and for reproducing reported results.

  3. Hugging Face Blog73

    HiRO-ACEv2 simulates extreme weather at 3 km resolution for a fraction of the cost

    Ai2 introduced HiRO-ACEv2, an updated climate emulator that generates global 3 km rain, wind, and pressure data at about 1/300th the cost of X-SHiELD. Given coarsened X-SHiELD data, its HiROv2 downscaling model matches high-resolution statistics to within 3%, even for events occurring once in a million data points. The system currently works only with ACE tuned to match X-SHiELD and cannot yet generate high-resolution data for other climates; downloadable models, tutorial notebooks, and a web demo are available.

    Why it matters: The comparison with X-SHiELD gives researchers a concrete cost and accuracy reference for extreme weather simulations, with applicability currently limited to the climate represented in training.

10/14Wed
  1. Google DeepMind94

    Google DeepMind launches Gemini 4 Argon for complex reasoning and agentic tasks

    Google DeepMind has launched Gemini 4 Argon globally through its API and selected Google products, with access depending on subscription or service tier. Introductory pricing is $2 per million input tokens and $10 per million output tokens, with cached input tokens at 95% off and a 1M unified input-output token limit through the Gemini API. Google reports improvements across coding, professional workflows, scientific research, and robotics, alongside stronger agent safeguards and prompt injection resistance.

    Why it matters: Google’s reported gains in code submissions and reductions in rollbacks connect Argon’s engineering capabilities to delivery speed and code stability in internal workflows.

10/8Thu
  1. Hugging Face Blog62

    Liquid AI open-sources d1-3B and d1-omni-600M on-device decision models

    Liquid AI released two open-weight models in its d1 decision-model family: d1-3B and the experimental d1-omni-600M.

    Why it matters: Liquid AI open-sourced two decision models and reported on-device latency and results across seven benchmarks, providing evidence on the use of small models for structured decisions.

10/7Wed
  1. Google DeepMind71

    Google DeepMind releases EmbeddingGemma 2, a lightweight open multimodal embedding model

    Google DeepMind released EmbeddingGemma 2, a 740M-parameter open model based on the Gemma 4 architecture and licensed under Apache 2.0. It maps text, images, audio, and video into a shared embedding space.

    Why it matters: EmbeddingGemma 2 maps text, images, audio, and video into one embedding space and reports concrete on-device memory and vector truncation figures, helping assess local multimodal retrieval.

10/6Tue
  1. Mistral AI80

    Mistral launches public preview of Mistral Large 4, a 1-trillion-parameter multimodal model

    Mistral released a public preview of Mistral Large 4, a natively multimodal model with 1 trillion parameters and 49 billion active parameters. Its weights will be opened at the end of the month.

    Why it matters: Mistral outlined the open-weight release schedule for its 1-trillion-parameter multimodal model and its investment in European compute, allowing comparison with mainstream open models.

10/1Thu
  1. Google DeepMind88

    Google DeepMind releases frontier model Gemini 4 Argon

    Google DeepMind released frontier model Gemini 4 Argon, initially making it available through the Fairwind Program to trusted cybersecurity defenders before gradually expanding access. Pricing starts at $2 per million input tokens and $10 per million output tokens.

    Why it matters: Gemini 4 Argon raises the maximum output to 1M tokens and reports multiple benchmark results and internal deployment data, offering a view of its limits on long-running workflows.

9/29Tue
  1. Microsoft Research62

    Microsoft Research releases Quine, an AI research system for biological complexity

    Microsoft Research has released Quine, an AI research system designed for biological complexity. It combines a biological world model with an interactive harness connecting models, scientific tools, literature, and experimentalists. In work with the Broad Institute on pancreatic ductal adenocarcinoma, it predicted and prioritized thousands of compounds; the top-ranked compound produced the largest classical-to-ground-state transition in wet-lab experiments. The process from narrowing the search space to selecting a candidate for validation took one weekend.

    Why it matters: Microsoft Research describes Quine's multimodal biological world model and experimental loop, including wet-lab validation in pancreatic cancer compound screening.

9/25Fri
  1. Google Research65

    Google Research introduces AI video co-director framework for coherent long-form video generation

    Google Research has introduced an AI video co-director multi-agent framework, an orchestration layer on top of Gemini and Veo. It models long-form video generation as a problem of global optimization and world-state tracking to mitigate semantic drift and cascading failures.

    Why it matters: Google breaks long-video generation into four composable frameworks and reports continuity improvements across several benchmarks, illustrating how multi-agent orchestration can reduce semantic drift.

  2. Google DeepMind71

    Google DeepMind releases Gemini 3.8 Live with Live Avatar

    Google DeepMind has released Gemini 3.8 Live with Live Avatar, combining real-time video generation and voice to give Gemini's conversational model a near-real-time visual avatar, with accurate lip-sync, natural expressions, and smooth turn-taking.

    Why it matters: Gemini 3.8 Live adds real-time video avatars, giving readers a view of how multimodal conversation is being applied in enterprise settings and where the feature is available.

9/23Wed
  1. Google DeepMind71

    Google DeepMind releases Gemini 3.8 Flash TTS and Flash-Lite TTS

    Google DeepMind has released two speech-generation models: Gemini 3.8 Flash TTS for character design and fine-grained creative control, and Gemini 3.8 Flash-Lite TTS for high-volume voiceovers and voice agents.

    Why it matters: The article outlines the models' voice cloning, line-by-line direction, and benchmark results, helping readers assess their suitability for dubbing and podcast production.

9/2Wed
  1. Google DeepMind74

    Google DeepMind adds agentic video understanding to Gemini 3.7 Flash and other models

    Google DeepMind has introduced agentic video understanding for Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite. Video analysis uses up to 88% fewer tokens and costs up to 66% less, while accuracy improves by up to 7%.

    Why it matters: The article gives token and cost reductions and access details, helping readers assess how the economics of long-video analysis are changing.

8/28Fri
  1. Google DeepMind68

    Google DeepMind releases Gemini Omni 1.1 Flash

    Google DeepMind has released Gemini Omni 1.1 Flash, offering developers video-generation controls through the Gemini API in Google AI Studio.

    Why it matters: The article details capabilities such as extending scenes, interpolating between first and last frames, and 4K upscaling, helping readers assess how controllable video-generation workflows are becoming.

8/12Wed
  1. Google DeepMind71

    Google DeepMind releases SL2T sign-language-to-text model, first rolling out in Gboard and Live Transcribe on Pixel 11

    Google DeepMind has released SL2T, a multilingual sign-language-to-text model, bringing sign-language AI to consumer products for the first time. On Pixel 11, Gboard and Live Transcribe support dictation from American Sign Language (ASL) to English; more devices and languages are planned.

    Why it matters: The source describes the capabilities and limits of a multilingual sign-to-text model and where it is available, helping readers assess sign-language translation in real products.

  2. Google Research72

    Google researchers advance AMIE to expert-level clinical interviews with audio and video

    Google Research has released AMIE (Video), which uses Gemini and Project Astra for real-time video clinical interviews. It can perceive nonverbal cues, guide virtual physical examinations, and reason about diagnoses in real time.

    Why it matters: AMIE extends from text to live video consultations, showing how an asynchronous multi-agent architecture can balance conversational latency with clinical reasoning.

8/10Mon
  1. Hugging Face Blog82

    Meta releases open-source 30B multimodal Muse Glimmer for local agents

    Meta has released Muse Glimmer, a 30B-parameter multimodal model distilled from Muse and licensed under Apache 2.0. It targets local-agent use cases such as coding, document analysis, and personal assistance.

    Why it matters: Meta has open-sourced a 30B multimodal model with reproducible details on its architecture, benchmarks, local deployment, and self-improvement demonstrations.

8/4Tue
  1. Mistral AI66

    Mistral AI releases Shieldstral, a 3B open-weight multimodal safety classifier

    Mistral AI has released Shieldstral, a 3B open-weight multimodal safety classifier available under Apache 2.0 and runnable on a single 16 GB NVIDIA GPU. It frames content moderation as a policy-adaptive question-answering task: policies are written in natural language at inference time, without retraining. It handles text and images together and outputs calibrated continuous safety scores. Mistral says it matches models up to seven times larger on text safety and sets a new state of the art in multimodal moderation.

    Why it matters: The 3B open safety classifier makes moderation policy a natural-language prompt at inference time, offering a view of what smaller models can do for content moderation.

7/30Thu
  1. Google DeepMind71

    Google DeepMind releases Gemini Robotics ER 2 embodied-reasoning model

    Google DeepMind has released Gemini Robotics ER 2, positioned as the robot's high-level brain: it converses with people, understands the physical world, and plans multi-step tasks before handing execution to a lower-level VLA model. It can also call tools such as Google Search natively.

    Why it matters: The official release gives concrete figures for ER 2's progress classification and temporal localization, along with access details, helping readers assess its limits on long tasks.