Cerebras has outlined its approach to deploying large frontier AI models on its wafer-scale chips, addressing a key challenge: trillion-parameter models require terabytes of storage while a single Cerebras WSE-3T wafer has only 44 GB of on-chip memory.
The company uses the open-source Kimi K3 model, with approximately 3 trillion parameters, as an illustrative example. Kimi K3 scores 88.3% on Terminal-Bench 2.1 and 91.2% on BrowseComp, demonstrating frontier-level coding and agentic capabilities that outperform some proprietary models.
Cerebras splits large models across multiple wafers using a pipeline architecture. The approach begins by calculating layer size. With 4-bit weight quantization, each of Kimi K3’s 93 layers requires approximately 16 GB of storage. The model’s 69 linear-attention layers (which don’t require a key-value cache) fit two per wafer at 32 GB total, while the 24 dense layers each require one wafer to accommodate their KV cache. This configuration requires 59 total wafers: 35 for linear layers and 24 for dense layers.
The wafers connect via a pipeline where each acts as a processing stage. Model weights remain stationary on their assigned wafers, while only activations—the intermediate computational results—transfer between wafers using RDMA over Ethernet, taking roughly 2 microseconds per hop. This design prevents model weights from constantly moving between chips.
To serve multiple requests efficiently, Cerebras keeps the entire pipeline full with different requests simultaneously. While one wafer processes a token for request A, another processes one for request B, and a third for request C. Each pipeline stage can also process a micro-batch of multiple requests rather than single requests. The tradeoff: smaller micro-batches reduce latency per request, while larger ones increase total throughput.
The maximum number of concurrent requests depends on on-chip KV cache memory. For the Kimi K3 example with a 256k token request size, each dense-layer wafer—after allocating space for its layer—has sufficient remaining memory to support multiple simultaneous requests. According to Cerebras, this pipeline approach enables throughput of 1000-2000+ tokens per second.
The same methodology scales to larger models. Each additional wafer contributes compute and memory, allowing deployment of models exceeding any single chip’s capacity.
Key facts
- Cerebras WSE-3T wafers have 44 GB of on-chip memory
- Kimi K3 has 3 trillion parameters across 93 layers (69 linear-attention, 24 dense)
- The example deployment uses 59 wafers total: 35 for linear layers, 24 for dense layers
- Model weights stay on-chip; only activations transfer between wafers via RDMA
- Inter-wafer communication takes roughly 2 microseconds per hop
- Kimi K3 scores 88.3% on Terminal-Bench 2.1 and 91.2% on BrowseComp
- Cerebras achieves 1000-2000+ tokens per second with this deployment