Specific, a software engineering benchmarking firm, has released Real-SWE, a benchmark designed to evaluate frontier AI models on private, real-world enterprise codebases. Unlike existing benchmarks that use public code or synthetic tasks, Real-SWE tasks come directly from licensed production codebases at actual companies, representing work that software engineers genuinely perform.
The benchmark includes tasks from several real companies: a Luma/Partiful competitor with over 200,000 users and top 100 App Store ranking; a consumer fintech platform processing over 100,000 bank statements; and enterprise AI sales platforms. Tasks span cross-functional engineering work including billing fixes, customer migrations, and multi-region deployments—changes with direct business consequences.
According to the benchmark results, Fable 5.1 achieved the highest resolution rate at 38.8%, followed by GPT-6 Astra at 33.8% and Gemini 3.8 Flash at 31.2%. Eight models were tested total, with the lowest performer achieving 16.2%. These resolution rates indicate substantial room for improvement in AI coding agent capabilities.
The benchmark emphasizes three key complexities absent from public benchmarks. First, tasks involve private codebases whose code and solutions don’t exist on the public internet—out-of-distribution problems unlikely to have been in any model’s training data. Second, tasks require understanding company-specific engineering patterns, conventions, and business logic unique to each organization. Third, tasks involve real business consequences; agents must navigate existing architecture, preserve user-facing behavior, and operate within genuine operational constraints.
Individual task performance varies widely. Among ten sampled tasks, six achieved resolution rates below 15%. “Multi-region sweep” and “API keys & environments” tasks achieved higher success rates of 67.2% and 65.6% respectively, while “Billing schedule migration” achieved only 14.1% across models. According to the source, 71.4% of failed rollouts lasted under 10 minutes, compared with 73.4% of longer rollouts failing, suggesting time alone doesn’t guarantee success.
The benchmark uses native harnesses—tools and environments that reflect how enterprise engineers actually work—rather than evaluating models in isolation. Agents must navigate AWS emulators, Docker, Kubernetes, GitHub, PostgreSQL, and other production tools. According to Specific, the results demonstrate that today’s frontier models struggle significantly with understanding company coding patterns and frequently miss requirements or fail to verify assumptions before proceeding.
Key facts
- Fable 5.1 achieved the highest resolution rate at 38.8% on Real-SWE tasks
- Tasks come from licensed private codebases at real companies, not public code or synthetic problems
- Six of ten sampled tasks achieved resolution rates below 15%
- Models must work with company-specific engineering patterns and business logic not found in training data
- Tasks involve cross-functional changes across multiple files and services with real business consequences