How DeepInfer Coordinates Asynchronous Scheduling for DeepSeek-V4.1-Flash Inference

What happened

DeepInfer, the inference engine behind DeepSeek-V4.1-Flash, overlaps CPU scheduling with GPU computation to reduce decoding idle time. Leiphone explains how its asynchronous scheduler handles in-flight tokens, sequence state corrections, deferred resource release, and coordination across expert-parallel GPUs. The open release currently supports offline inference, with some online infrastructure unavailable; DeepSeek has not separately published complete end-to-end speedup measurements isolating the scheduler’s contribution.

Read next

• More changes in today’s briefingGoogle DeepMind announces Gemini 4 Carbon with cybersecurity and long-horizon capabilities

The story so far

DeepSeek has open-sourced DeepInfer under Apache 2.0, releasing the inference framework it says powers DeepSeek-V4.1-Flash in production; vLLM reports matching its performance on DeepSeek’s decode configuration. That comparison covers batch sizes 8 to 256, with vLLM running at DEP32 against DeepInfer’s EP64DP64. vLLM said DeepInfer was originally built primarily on vLLM. According to Baoyu, DeepInfer currently supports only V4.1-Flash on Nvidia Blackwell SM100 or Huawei Ascend 950 hardware, and the release provides offline inference components that require additional infrastructure for online serving. Leiphone explained that its asynchronous scheduler overlaps CPU scheduling with GPU computation to reduce decoding idle time, coordinating in-flight tokens, sequence-state corrections and resource release across expert-parallel GPUs. DeepSeek also released deepseek-recipe. Baoyu reported that the MIT-licensed companion package provides Python and Rust implementations supporting V4 and V4.1, translating common OpenAI and Anthropic request formats and model outputs to align with DeepSeek’s official API.