A GitHub repository documents a complete production setup for running DeepSeek-V4-Flash on a single AMD MI300X GPU, addressing compatibility issues not covered by the official vLLM recipe.
The MI300X, an AMD CDNA3 processor with 192 GB of HBM3 memory and 5.3 TB/s bandwidth, provides 2.4× the HBM capacity of an H100 SXM5 and costs roughly half as much at list price, according to the repository. The 304-billion-parameter DeepSeek V4 Flash checkpoint weighs 156.67 GB in full precision and fits entirely in GPU memory without quantization or weight offloading.
Performance metrics from the pinned vLLM ROCm nightly build (0.26.1rc1.dev229+g124154a88.rocm723, AITER 0.1.19) show: single-stream decode at 168.6 tokens per second, prefill throughput of approximately 7.9–8.5K tokens per second with tuned kernels, aggregate 542 tokens per second across 8 concurrent streams, and burst capacity of 830 tokens per second across 64 streams without out-of-memory errors or engine failures. The model supports a 256K validated context window, with the architecture designed to support 1 million tokens.
The official vLLM recipe targets NVIDIA and newer AMD hardware. Achieving reliable operation on MI300X required fixes for FP8 format incompatibility (MI300X uses AMD/Graphcore fnuz variant E4M3, while newer MI325X and MI355X use OCP-standard FP8), MoE routing at high concurrency, causal speculative verification, CPU-KV synchronization, and several untuned kernel shapes.
The repository includes a Docker Compose stack with digest-pinned artifacts, Python overlay patches for vLLM components, and AITER GEMM tuning tables. The configuration uses a hybrid KV strategy with 20 GB of fp8_ds_mla GPU cache and 96 GB native CPU offload. Speculative decoding employs DSpark-7 with probabilistic drafting and block rejection, and the scheduler uses a 2,048-token budget with a 1,024-token long-prefill cap to prevent cold prompts from stalling concurrent streams.
System requirements include an MI300X (gfx942, 304 compute units), working AMD kernel driver, recent Docker Compose, approximately 235 GB of RAM for the CPU KV tier, and about 500 GB of disk space. The vLLM ROCm image is specified by digest, allowing repeatable deployments. After startup (typically 5 minutes), GPU memory usage reaches a high-water mark of approximately 204.5 GB of 205.8 GB available.
Key facts
- DeepSeek V4 Flash (304B parameters) fits entirely on one AMD MI300X without quantization or offloading at 156.67 GB
- Single MI300X achieves 542 tok/s aggregate throughput across 8 concurrent streams and 830 tok/s in 64-stream bursts
- MI300X has 2.4× the HBM capacity of H100 SXM5 and costs roughly half as much at list price
- MI300X FP8 format (fnuz E4M3) differs from newer AMD GPUs (OCP standard), requiring kernel fixes for correctness
- Repository provides Docker Compose stack, Python overlays, and tuning tables to address vLLM recipe limitations on MI300X