Skip to content
K = 0.731

ReplaySSM Support Reaches vLLM, With Reported Decoding Speedups and Production Use

1 reports 1 reporting sources 4 days agoNew

Understand the event

AI overview

vLLM said ReplaySSM support is already available in vLLM for Mamba2 standard decoding and FlashInfer-backed multi-token prediction (MTP), reducing recurrent-state memory traffic by caching state-space model (SSM) inputs and updating the checkpoint only when needed. In the post quoted by vLLM, @zwljohnny reported that ReplaySSM is used in production in GLM-5.3-Flash and Kimi-K3 and is available in TensorRT-LLM, vLLM, FlashInfer, and SGLang. @zwljohnny reported speedups of up to 1.47x for standard decoding and 2.1x for speculative decoding, and said ReplaySSM supports low-precision SSM state whose rounding error never propagates through the recurrence.

Generated by AI from reports · Updated 4 days ago

Latest development10/13 12:35 (UTC+8)
ReplaySSM reports decoding speedups and production use, with support already in vLLMvLLM · X

Report timeline

Follow the reports to explore different sides of the event.

10/13
  1. vLLM · X
    ReplaySSM reports decoding speedups and production use, with support already in vLLM

    vLLM praises ReplaySSM work by @zwljohnny, @tri_dao, and @NVIDIA, saying Mamba2 standard decode and FlashInfer-backed MTP support are already in vLLM. Caching SSM inputs and updating the checkpoint only when needed reduces recurrent-state memory traffic. In the quoted post, @zwljohnny reports production use in GLM-5.3-Flash and Kimi-K3, availability in TensorRT-LLM, vLLM, FlashInfer, and SGLang, speedups of up to 1.47x for standard decoding and 2.1x for speculative decoding, and low-precision SSM state whose rounding error never propagates through the recurrence.

Keep reading

Latest daily · Issue 24NVIDIA and AI4Bharat Introduce Nemotron-pretraining-Indic-v1 and an Open Data Generation Pipeline

Trending now

  1. 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
  2. 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
  3. 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
  4. 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights