DeepSeek has open-sourced DeepInfer, the inference framework that powers its large-scale production services. According to the GitHub release, DeepInfer provides offline inference capabilities designed for serving models at scale.
The framework currently supports DeepSeek-V4.1-Flash models and includes several optimization features. These include optimized kernels (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels), parallelism options for attention and mixture-of-experts layers, and asynchronous scheduling with a zero-overhead batch scheduler.
DeepInfer also includes advanced decoding techniques. DSpark and confidence-scheduled verification accelerate decoding using draft tokens and confidence estimates, while decoder suffix-windowed attention (SWA) bounded replay reduces decoder prefill work by replaying only a bounded suffix. The framework supports batch-invariant inference to keep model results identical across different batch compositions, deterministic sampling for reproducible outputs, and structured outputs that can be constrained to regex patterns, JSON schemas, or EBNF grammars.
The framework supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. According to the documentation, a complete inference service would require additional components not included in this release, including a KV cache system, prefill and decode disaggregation, and load balancing across data-parallel replicas.
DeepInfer requires Linux, GCC 13.0 or later, Python 3.12, and either CUDA 13 for NVIDIA platforms or CANN 9.2 for Ascend. The framework was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM, according to the release notes.
The code is released under the Apache License 2.0, making it freely available for both commercial and non-commercial use. The default model checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017) is automatically downloaded from Hugging Face on first run.
Key facts
- DeepInfer is DeepSeek’s inference framework that powered its production services, now open-sourced
- Currently supports DeepSeek-V4.1-Flash models and NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
- Features include optimized kernels, parallelism options, asynchronous scheduling, deterministic sampling, and structured output constraints
- Released under Apache License 2.0