According to the blog post by Noukhovitch, reinforcement learning (RL) post‑training of large language models (LLMs) shows a pattern known as the Matthew Effect: improvements accrue mainly to easy problems while hard problems see little gain. The author evaluates this using the AIME 2025 benchmark, splitting 30 questions into three difficulty levels based on the pre‑RL model’s pass@32 score. The initial pass@1 averages are 0 % for hard, 3.8 % for medium, and 22.7 % for easy problems. Over RL training, the easy set improves substantially, whereas the hard set remains almost unchanged, illustrating that the average eval curve masks divergent progress across difficulty buckets. The same trend appears in code and agentic RL tasks using Deepcoder, DeepSWE, and SWEBench, where gains are proportional to a model’s initial competence on each task. The post links this bias to the Matthew Effect from network science and economics, describing it as “the rich get richer.” One hypothesized cause is signal loss: when none of the k sampled completions for a prompt are correct, the model receives no gradient and cannot learn from that prompt. Increasing k can mitigate signal loss but also raises the chance of sampling rare incorrect solutions for easy problems, wasting compute on already‑solved tasks. To address this, the author proposes Never Give Up (NGU), an adaptive sampling scheme. Training starts with a small k; if a prompt is solved within the first k completions, it is used for training. If all k completions are wrong, with probability p the prompt is returned to the generator for another round of k samples, preserving previously collected completions. When the prompt is eventually solved, the model trains on the accumulated k × rounds completions. This yields a geometric distribution of samples, allowing more attempts on hard problems while limiting effort on easy ones. Experiments on GSM8k with Qwen 2.5 0.5B Instruct show that NGU with k=4 and p=0.9 outperforms standard GRPO across all k values, especially on the hardest subset. The post also discusses practical concerns such as stale completions in asynchronous RL and offers tricks—filtering by age, adjusting the GRPO baseline, or anchoring positives—to preserve training stability. Finally, NGU is scaled up to a larger math RL setup using DeepScaler with Qwen 3 4B base, where it further improves performance atop a strong GRPO k=16 baseline.
Key facts
- Initial AIME 2025 pass@1 averages: hard 0 %, medium 3.8 %, easy 22.7 %
- NGU uses small k for easy problems, large k for hard problems via probability p
- On GSM8k, NGU (k=4, p=0.9) beats standard GRPO across all k values
- Stale completions can be mitigated by age filtering and baseline adjustments