Real-SWE Benchmark Launches: Evaluating AI Coding Agents on Private Enterprise Repositories

For more than two years, the artificial intelligence industry treated SWE-bench as the definitive gold standard for software engineering autonomy. Claimed resolution rates climbed steadily from single digits in late 2023 to over 80% on SWE-bench Verified by mid-2026, leading venture capitalists and foundation model labs to declare that autonomous coding agents were on the verge of replacing junior engineers. However, that apparent mastery was increasingly clouded by an open secret: severe benchmark saturation and data contamination from public GitHub pull requests.

To puncture that synthetic bubble, research group Specific Labs has officially released Real-SWE, a comprehensive evaluation suite designed to test autonomous coding agents not on public open-source repositories, but on licensed, private enterprise codebases. Built directly from proprietary production applications—including a social ticketing platform with over 200,000 active users, a commercial fintech engine reconciling 100,000 monthly bank statements, and enterprise B2B sales architectures—Real-SWE introduces tasks that are fundamentally out-of-distribution and impossible for LLMs to have memorized during web pretraining.

The disparity between public benchmark marketing and private production performance is stark. Where frontier models routinely report 75% to 85% success on public SWE-bench subsets, Real-SWE brought the entire industry down to Earth.

Across eight independent runs per task evaluated with 95% confidence intervals, Fable 5.1 deployed via Claude Code captured first place with an aggregate resolution rate of just 38.8%. OpenAI's GPT-6 Astra paired with Codex CLI followed closely at 33.8%, while Google's Gemini 3.8 Flash operating through Gemini CLI secured third at 31.2%. Notably, open-weight flagship GLM-5.3 achieved fourth place at 28.8% running on the Claude Code harness, outpacing proprietary models including Grok 4.6 (23.8%), Meta's Muse Spark 1.3 (23.8%), and Moonshot's Kimi K3 (18.8%).

More revealing than the top-line scores is task distribution: 6 out of the 10 core production tasks saw collective resolution rates plunge below 15%. On a complex distributed analytics stream reducer, zero out of eight tested models achieved a single successful pass across 64 cumulative attempts. Similarly, a multi-service tax jurisdiction billing reconciliation task yielded a dismal 3.1% industry-wide resolution rate.

Real-SWE's diagnostic telemetry highlights three structural realities that distinguish actual enterprise engineering from synthetic academic puzzles:

1. 1.Multi-File Architectural Sprawl: In synthetic benchmarks like FrontierCode or DeepSWE, the median reference solution edits 6 files. On Real-SWE, reference solutions modify a median of 11 files. Real enterprise bugs rarely live in a single isolated function; fixing an invoice tax exemption requires refactoring NestJS controller routes, updating InfluxDB ledger schemas, and handling external TaxJar sandbox API rate limits.

1. 1.The Unverified Assumption Trap: The single most prevalent failure mode was not faulty syntax, but unverified assumptions. Rather than inspecting the local workspace, running unit tests, or querying live Docker containers, models frequently guessed at database schemas or assumed third-party dependencies functioned like standard open-source libraries. GPT-5.6 Sol failed 43.3% of its runs on unverified assumptions, while GPT-6 Astra failed 34.0%.

1. 1.Integration Errors and Regression Breaks: Frontier models exhibited a strong tendency to implement the correct local algorithmic fix while completely breaking surrounding services. In Gemini 3.8 Flash, 49.1% of all failed rollouts were classified as integration errors—wiring working logic into the broader microservice mesh incorrectly.

A critical finding for developer platforms is that task failure was largely invariant to time. Rollouts lasting under 10 minutes failed 71.4% of the time, while rollouts extending beyond 10 minutes failed 73.4% of the time. When an autonomous agent misunderstands company-specific coding patterns or misses unstated business constraints early in its planning phase, extended reasoning steps and multi-turn loops merely compound hallucinated fixes.

By forcing agents to operate across Docker containers, Redis caches, PostgreSQL instances, and external mock APIs with slightly underspecified, realistic instructions (averaging 1,742 characters), Real-SWE demonstrates that true software engineering requires context discovery, architectural conservatism, and rigorous assumption verification. Autonomous coding has not stalled, but Real-SWE establishes that true production reliability will be measured in private enterprise trenches rather than saturated public leaderboards.

---\n### Fact-Checked Sources & Verified References

• Real-SWE Benchmark — Benchmarking AI models on private enterprise codebases — Specific Labs
• Real-SWE Benchmark: Coding Agents on Real Company Code — ExplainX AI
• Specific Labs Launches Private-Code Benchmark Where Top Coding Setup Solves 38.8% — Superpower Daily