DeepSeek has released DeepInfer, an open-source inference framework that powers the company’s large-scale production services for its models. The framework is designed for offline inference at scale and is released under the Apache License 2.0.
DeepInfer currently supports DeepSeek-V4.1-Flash models and includes several performance-focused features. According to the GitHub repository, the framework offers optimized kernels including FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. It supports multiple parallelism strategies: Attention DP (data parallelism) and balanced CP (context parallelism), along with MoE EP (expert parallelism) and EPLB.
Key capabilities include DSpark and confidence-scheduled verification, which accelerate decoding using draft tokens and confidence estimates to schedule verification. The framework also features SWA bounded replay to reduce decoder prefill work by replaying only a bounded suffix. Asynchronous scheduling with zero-overhead batch scheduling is enabled by default, and the framework supports batch-invariant inference to keep model results identical across different batch compositions on NVIDIA hardware.
DeepInfer supports deterministic sampling to produce identical outputs for identical input tokens and sampling parameters, and includes structured outputs that constrain generation to regex, JSON schema, or EBNF grammar specifications.
The framework is compatible with NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Building DeepInfer requires Linux, GCC version 13.0 or higher, Python 3.12, CUDA 13 for NVIDIA platforms, and CANN 9.2 for Ascend platforms.
According to the repository, DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. The release notes indicate that a complete inference service requires additional components not included in this release, including a KV cache system and VE/Prefill/Decode disaggregation functionality.
Key facts
- DeepInfer is an open-source inference framework released by DeepSeek under Apache License 2.0
- Currently supports DeepSeek-V4.1-Flash models
- Compatible with NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs
- Includes optimized kernels, parallelism strategies, and structured output support
- Built primarily on vLLM with inspiration from SGLang and TensorRT-LLM