Llama-modchip, a new open-source project, makes it possible to run large language models on consumer CPU hardware without GPU acceleration. According to the GitHub repository, the tool enables DeepSeek-V4-Flash—a 284-billion-parameter model—to run on a 64GB RAM system at 44.36 tokens per second during prefill and 9.06–10.74 tokens per second during decoding.
The project achieves this through several optimization techniques. The DeepSeek-V4-Flash model is compressed from its standard 76.87 gigabytes down to 56.56 gigabytes on disk using what the repository describes as “closed-form Analytical SwiGLU-H2 Second-Order Hessian Compression” combined with custom quantization methods. This allows the model plus a 256K context window to fit entirely in RAM at 57.61–58.9 gigabytes peak memory usage.
The tool supports models of varying sizes for different hardware configurations. On a developer laptop or mini-PC with 16–32GB of RAM, the Nemotron-3.5-30B-A3B model runs at 25.0–60.4 tokens per second. For systems with both CPU and modest GPU resources (32–64GB RAM plus 16GB GPU), Qwen3-30B or Nemotron-30B achieve 80.8 tokens per second. An 8-channel workstation with Threadripper processors can reach 111.62 tokens per second on 30B models.
The project includes a command-line interface with several modes: ./modchip chat for interactive terminal use, ./modchip web for a browser interface, and ./modchip claude for integration with Claude Code tools. According to the repository, the Claude Code integration eliminates a 45-second CPU agent prompt re-evaluation bottleneck through KV cache warm restore mechanisms that take 13.1–32.6 milliseconds.
Optimizations are hardware-specific. The repository documents support for AMD Zen 5 processors, including desktop, mobile, and server variants, with active optimizations including single-cycle 512-bit AVX-512 instructions and L1/L3 cache tiling strategies.
Models are distributed as custom GGUF format files using a proprietary quantization type (GGML_TYPE_Q2_0) that is incompatible with standard llama.cpp, Ollama, or LM Studio without patching. The project provides pre-compiled binaries and detailed installation instructions via Hugging Face repositories.
Key facts
- DeepSeek-V4-Flash 284B model compressed to 56.56GB disk size, fitting in 64GB RAM systems without GPU
- Achieves 44.36 tokens/second prefill and 9.06–10.74 tokens/second decoding on 16-core Zen 5 CPU
- Smaller Nemotron-3.5-30B-A3B model runs at 25.0–60.4 tokens/second on 16–32GB laptops, 111.6 tokens/second on Threadripper
- Uses custom compression techniques including Analytical SwiGLU-H2 Hessian Compression and Fractal-64 quantization
- Supports 256K context window entirely in RAM
- Includes Claude Code integration with 13.1–32.6ms cache warm restore time