While large language models have become proficient at generating syntactically correct code, a new problem has emerged: the code may be formally valid but poorly structured, bloated with unnecessary abstractions, or filled with duplications. According to a post from Earendil, this raises questions about how to measure what developers call “code sloppiness.”
The challenge lies in the nature of code evaluation itself. Testing correctness is straightforward—code either passes hidden tests or it doesn’t. But evaluating quality requires human judgment and taste, making it difficult to scale automated assessment.
Common industry approaches fall short, according to the post. Using AI models to judge code quality directly—whether by asking for a 1-10 rating or comparing two solutions—produces inconsistent results. The models can change their preferences when code is renamed, and their evaluations often correlate poorly with human assessments. Human review remains the gold standard but is not scalable for training or benchmarking across multiple model providers.
Researchers have identified more promising quantitative metrics. The simplest is lines of code (LOC) change—unexpectedly effective at detecting sloppiness, though the metric loses meaning if optimized directly. Two additional measures come from SlopCodeBench, a research effort that distinguished between legacy code and AI-generated code: verbosity and erosion.
Verbosity measures duplicated and unnecessary verbose lines. Erosion measures how much of a codebase’s mass concentrates in few large, complex functions with high cyclomatic complexity. When compared to established open-source repositories, AI-generated code scored roughly twice as poorly on both metrics. Average verbosity in human repos was 0.15 ± 0.06 versus 0.33 ± 0.10 for agent-generated code. For erosion, repos achieved 0.31 ± 0.17 compared to agents’ 0.68 ± 0.20.
A more revealing finding emerged from SlopCodeBench’s evaluation methodology. Unlike typical coding benchmarks that provide complete instructions upfront, SlopCodeBench uses iterative rounds with context erasure between checkpoints, mimicking how humans actually use coding agents. In this setup, even state-of-the-art models achieved 0% pass rate when all tests at all checkpoints had to pass—suggesting that poor coding decisions accumulate during iterative development.
Key facts
- AI models generate syntactically correct code but often introduce unnecessary complexity, duplication, and poor design decisions
- AI-judged code quality evaluations are unreliable and change based on superficial factors like variable names
- AI-generated code measured 0.33 ± 0.10 verbosity versus 0.15 ± 0.06 for human repos; 0.68 ± 0.20 erosion versus 0.31 ± 0.17 for human repos
- In iterative evaluation mimicking real-world use, state-of-the-art models achieved 0% pass rate when all tests at all checkpoints had to pass