What happened
vLLM-Omni has day-one support for AURA 2.0, a 9B streaming video model that can initiate conversations, call tools based on live scenes and dialogue, and point to discussed objects with frame-aligned boxes. In a single-session setup with two ~140 GB accelerators, the pipeline achieved a 344 ms median time to the first reply audio packet. AURA 2.0 leads the four aggregate benchmark comparisons shown: IPIBench, StreamArena, ProactiveVideoQA, and PhoStream.
Why it matters
AURA 2.0’s 344 ms median time to first reply audio packet shows the latency vLLM-Omni achieved in a single-session setup with two ~140 GB accelerators; the result is limited to that setup.
🎉 Day-0 support for AURA 2.0 in vLLM-Omni!
A 9B streaming video model that watches the scene, starts relevant conversations, and acts on what it sees.
💬 Model-Initiated Conversation: decide when to speak and what to say, without waiting for a question.
🔧 Stream-Aware Tool Calling: use the live scene and conversation to call tools and bring results back into the dialogue.
🎯 Interactive Visual Grounding: point to the objects being discussed with frame-aligned boxes.
vLLM-Omni powers the pipeline from speech recognition through video-language inference to speech generation. Session and prefix reuse, a fast path for silent turns, and sentence-level speech handoff deliver 344 ms median time to the first reply audio packet in the single-session setup with two ~140 GB accelerators.
AURA 2.0 leads the four aggregate benchmark comparisons shown below: IPIBench, StreamArena, ProactiveVideoQA and PhoStream.
Thanks to the AURA team at @Huawei Research, CUHK MMLab and HKUST!
Project & demos: aurateam2026.github.io/Aura-2…
Weights on @huggingface: huggingface.co/aurateam/AURA_2
The story so far
vLLM-Omni has introduced day-one support for AURA 2.0, a 9B streaming video model capable of initiating conversations, executing tool calls based on live video and audio input, and identifying objects with frame-aligned bounding boxes. In a single-session configuration utilizing two 140 GB accelerators, the system achieved a median time to first reply audio packet of 344 ms. According to the release, AURA 2.0 outperforms other models across four primary benchmarks: IPIBench, StreamArena, ProactiveVideoQA, and PhoStream.
