ReplaySSM reports decoding speedups and production use, with support already in vLLM

What happened

vLLM praises ReplaySSM work by @zwljohnny, @tri_dao, and @NVIDIA, saying Mamba2 standard decode and FlashInfer-backed MTP support are already in vLLM. Caching SSM inputs and updating the checkpoint only when needed reduces recurrent-state memory traffic. In the quoted post, @zwljohnny reports production use in GLM-5.3-Flash and Kimi-K3, availability in TensorRT-LLM, vLLM, FlashInfer, and SGLang, speedups of up to 1.47x for standard decoding and 2.1x for speculative decoding, and low-precision SSM state whose rounding error never propagates through the recurrence.

The story so far

vLLM said ReplaySSM support is already available in vLLM for Mamba2 standard decoding and FlashInfer-backed multi-token prediction (MTP), reducing recurrent-state memory traffic by caching state-space model (SSM) inputs and updating the checkpoint only when needed. In the post quoted by vLLM, @zwljohnny reported that ReplaySSM is used in production in GLM-5.3-Flash and Kimi-K3 and is available in TensorRT-LLM, vLLM, FlashInfer, and SGLang. @zwljohnny reported speedups of up to 1.47x for standard decoding and 2.1x for speculative decoding, and said ReplaySSM supports low-precision SSM state whose rounding error never propagates through the recurrence.

Keep reading

Trending now

1. 1Meta Releases Muse Spark Open With Mac and Cloud Support, Topping US Open-Weight Index
2. 2Google Launches Gemini 4 Argon Globally, Expands Paid Access as Benchmark Results Vary
3. 3OpenAI Updates GPT-6 Astra Pro in ChatGPT With Parallel Agents and Faster Responses
4. 4Meta’s New Business Terms Raise Dispute Over Website Crawling and AI Training Rights