DeepSeek open-sources DeepInfer, its production inference framework

What happened

DeepSeek announced that DeepInfer, its in-house inference framework powering DeepSeek-V4.1-Flash at scale, is now fully open source. Nathan Lambert called this a wild way to drive model adoption and great for the community, remarking on the timing given what he described as a government ban on DeepSeek in federal agencies. http://github.com/deepseek-ai/DeepInfer https://twitter.com/deepseek_ai/status/2112476496283775135

Read next

• More changes in today’s briefingConfigure the ChatGPT desktop app with Amazon Bedrock in AWS GovCloud (US)

Reported by 5 sources

The story so far

DeepSeek has open-sourced DeepInfer under Apache 2.0, releasing the inference framework it says powers DeepSeek-V4.1-Flash in production; vLLM reports matching its performance on DeepSeek’s decode configuration. That comparison covers batch sizes 8 to 256, with vLLM running at DEP32 against DeepInfer’s EP64DP64. vLLM said DeepInfer was originally built primarily on vLLM. According to Baoyu, DeepInfer currently supports only V4.1-Flash on Nvidia Blackwell SM100 or Huawei Ascend 950 hardware, and the release provides offline inference components that require additional infrastructure for online serving. Leiphone explained that its asynchronous scheduler overlaps CPU scheduling with GPU computation to reduce decoding idle time, coordinating in-flight tokens, sequence-state corrections and resource release across expert-parallel GPUs. DeepSeek also released deepseek-recipe. Baoyu reported that the MIT-licensed companion package provides Python and Rust implementations supporting V4 and V4.1, translating common OpenAI and Anthropic request formats and model outputs to align with DeepSeek’s official API.