Recent research into how neural networks process information suggests the field may have fundamentally misunderstood their internal structure, according to analysis from Goodfire.
After studying neurons and circuits in image and language models, researchers concluded that networks’ mechanisms serve as legible building blocks and that mechanistic interpretability—the effort to understand how neural networks work internally—has better prospects than previously thought.
A key finding challenges the theory of superposition, which proposed that neural networks pack unrelated concepts together when running out of neurons, creating what researchers called “knots” that obscure understanding. According to the analysis, what looked like arbitrary packing in models like InceptionV1 was actually “recursive generalization,” a less concerning phenomenon.
Architectural problems in InceptionV1’s later layers created the appearance of incoherence, the research found. When studied with modern architectural improvements, 99-100% of neurons in the last layer became easily interpretable, despite no changes to model size or training data. The network proved broadly legible across a 300× size range, with smaller models developing coarser-grained concepts and larger ones differentiating into more specific features.
The canonical example of polysemanticity—neurons appearing to perform multiple unrelated jobs—was also reexamined. A frequently cited 2019 circuit appeared to show a car feature spreading into unrelated dog neurons, interpreted as the network hiding information. Further analysis revealed this was ordinary generalization and composition, with nothing downstream actually reading that combination.
Modern language models show stronger interpretability than recent studies suggested. Earlier research found median neurons scored near zero on interpretability rubrics, with only 1.7% interpretable by generous measures. Preliminary audits of Qwen3-32B, however, estimate roughly 90% of neuron poles have one main job, with small related functions; approximately 7% have two related jobs; and only 2-4% remain unexplained. Similar results appeared in models up to trillions of parameters.
Researchers note these percentages compress complex multidimensional patterns into single numbers, and precise scores depend on where thresholds are set for weak activations and other factors. The broad conclusion is that neural networks are cleaner—more directly understandable—than the field’s recent consensus suggested, potentially enabling direct study rather than requiring untangling first.
Key facts
- Analysis of InceptionV1 showed 99-100% of neurons in later layers are easily interpretable with modern architectural improvements
- Preliminary audits of Qwen3-32B estimate roughly 90% of neuron poles have one main job
- Earlier research found only 1.7% of neurons interpretable; new findings suggest substantially higher interpretability across large language models
- Superposition theory, which proposed networks pack unrelated concepts together, was identified as the wrong explanation for InceptionV1’s apparent incoherence