Baidu has released ERNIE 5.5 Flash Base, a sparse Mixture-of-Experts language model designed as a foundation for further development by researchers and developers. The model contains 280 billion total parameters, with 14 billion activated per token during inference.
According to Baidu’s release documentation, the model was pre-trained from scratch on 33.5 trillion tokens with a training context of up to 128 kilobytes. It supports a 1-million-token context window for fine-tuning and inference applications. The release provides the weights at the end of pre-training, before mid- or post-training stages, making it available as a starting point for domain adaptation, instruction tuning, and reinforcement-learning research.
The architecture prioritizes efficiency across multiple components. The model uses a hybrid attention mechanism combining Multi-head Latent Attention (MLA) and Heavily Compressed Attention (HCA). Most layers use HCA, which compresses every 128 tokens into a single cached entry while attending to recent, uncompressed tokens. Only one in seven layers maintains token-level access to the full context using MLA. This design reduces KV cache requirements while preserving access to full sequence information.
The attention system also incorporates sparse attention in full-context layers, where a lightweight indexer scores past positions for each query and core attention is computed only over the top-k highest-scoring positions. This keeps per-query attention costs constant as context length grows.
For the feed-forward component, ERNIE 5.5 Flash uses a Multi-Router Latent Mixture-of-Experts approach. Rather than using a single router to select experts, the system employs multiple independently initialized routers that jointly score experts. According to the documentation, this design reduces sensitivity to noisy features and distribution shifts compared to single-router approaches.
The model contains 43 Transformer layers with a hidden size of 4,096 and 64 attention heads. Each MoE layer includes 512 routed experts plus one shared expert, using auxiliary-loss-free, bias-corrected load balancing. Experts operate in a compressed 2,048-dimensional latent space rather than at full width, reducing computational overhead.
Baidu states the release is intended to reduce the cost of large-scale pre-training for developers, allowing them to focus resources on capabilities, tasks, and training methods specific to their applications.
Key facts
- ERNIE 5.5 Flash Base contains 280 billion total parameters with 14 billion activated per token
- The model was pre-trained on 33.5 trillion tokens with up to 128K training context and supports 1M-token context for inference
- The release provides pre-training weights only, without mid- or post-training, for community-driven development
- The architecture uses hybrid attention combining Multi-head Latent Attention and Heavily Compressed Attention to reduce KV cache costs
- Multiple independent routers score experts jointly in the MoE system rather than using a single router