The author begins by stating three theses for readers to consider. First, frontier labs are valued on the idea that they will soon provide a drop‑in replacement for most knowledge workers, yet today’s leading models still require extensive supervision and guardrails even for simple tasks. The author points out that impressive demonstrations such as Navier‑Stokes proofs, FreeBSD remote code execution exploits, or the HuggingFace incident do not indicate genuine autonomy, noting that companies continue to employ and hire low‑performing software engineers who would score below the models they oversee on benchmark tests. Second, the author claims that LLMs generalize well only within a narrow range of the tasks they were trained on, and even then they can fail or engage in reward hacking when faced with small perturbations. Frontier labs have a recipe for teaching models specific, well‑defined tasks, but any deviation from the training distribution often leads to breakdown. Third, the author argues that solving reward hacking demands rigorous specification by domain experts, a costly and specialized skill. Because the pool of people who are both domain experts and specification experts is tiny, the labor cost of creating such specifications can exceed the cost of directly implementing an informal specification. The hardware engineering field is cited as an example, where specification and validation engineers often outnumber design engineers by ratios of 3:1 or higher, and specifications frequently evolve during implementation, making a “spec‑and‑forget” approach unrealistic for most domains. The author then highlights Navier‑Stokes and similar pure‑mathematics theorems as the best‑case scenario for agentic work: the theorem statement itself is a rigorous specification, vetted by the mathematical community, and its Lean translation relies on battle‑tested components from Mathlib. Even in this favorable setting, soundness bugs in Lean have previously allowed LLMs to launder false proofs, suggesting vulnerabilities remain. The post continues by asserting that human review does not scale with model output and remains susceptible to reward hacking, citing the XZ backdoor and the UMN hypocrite commits in Linux as examples. Consequently, any agentic pipeline that depends on human review is bottlenecked by human time and attention, undermining the vision of a fully autonomous “data center of geniuses.” Taking these points together, the author concludes that for most domains LLMs will behave like a competent intern: useful when guided by an expert but not suited for unrestricted autonomy. Only three types of firms can realistically adopt fully autonomous LLMs: those that can tolerate cheap failure (e.g., firms that would otherwise hire interns or engage in rapid prototyping), those needing a narrow set of well‑guarded tasks (such as repetitive physical labor in controlled environments or call‑center chat work), and those that already bear the costs of rigorous specification and validation (e.g., chip design, drug discovery, and other high‑stakes fields). The first two groups are price‑sensitive and likely best served by inexpensive open models running on modest hardware, possibly locally. The third group might still use frontier models, but the author speculates that cheaper models combined with wider agentic swarms could achieve similar results, and notes these firms tend to guard their IP closely, making them reluctant to send data to external API providers. Finally, the author suggests that even if frontier labs decline, the compute demand driven by large numbers of mediocre models overseen by humans could still be substantial, though limited by human orchestrators, and believes the impact will extend beyond the frontier labs themselves.
