A blog post by researcher Honnibal Matthew challenges the framing of a widely-shared paper claiming that GPT-4o’s vision model Astra can read and modify neural network weights to improve performance. According to the critique, the original post presents experiments using fundamentally different methodologies under a unified narrative, creating a misleading impression of a single coherent capability.
The original post’s abstract states that Astra can identify issues in model weights and fix them by editing small percentages of parameters. The most prominent example involves improving Qwen3.5’s math performance from 7.9% to 48.1% on MATH-500, presented as evidence of direct weight manipulation.
However, Honnibal’s analysis reveals that the experiments employ three distinct approaches. Only experiments 1, 2, and 4—conducted on small networks of 17,000 to 172,000 weights—actually work “just by looking at raw weights” without running the model or using code. These include naming games that small MLPs were trained on and fixing a stuck policy in a 172K-parameter network.
In contrast, the major results involving large language models, including the featured Qwen improvement, rely on Python code and model execution. The Qwen experiment allows Astra to run the model on up to 50 batches of practice problems before suggesting weight edits. Similarly, the experiment writing a 9×9 Go network uses Python and executes the network during development.
Honnibal compares the presentation to magic tricks using multiple techniques: “When you watch the performance you assume there’s only one technique, so each time they do it they seem to debunk your hypothesis about how it must be working.”
The critique doesn’t dismiss the results entirely. Honnibal acknowledges that prompting Astra to devise and execute code-based weight analysis or modification tasks represents legitimate capability—something earlier AI models couldn’t do. However, the argument is that grouping these Python-dependent experiments with direct weight-reading experiments misrepresents what capability is being demonstrated. A single technique capable of both directly reading small network weights and improving large models would suggest a more profound understanding than what the varied methodology actually demonstrates.
Key facts
- The original post groups experiments using raw weight inspection with experiments requiring Python code and model execution
- Only three of eight main experiments use direct weight reading without code or model execution, all on networks under 172K parameters
- The featured Qwen math improvement experiment uses Python and allows 50 passes of model execution, not raw weight inspection alone
- The Go network and other large-model experiments similarly depend on Python and running the model, differing fundamentally from the small-network experiments