Specific Labs Introduces Real-SWE Benchmark Using Private Codebases
Specific Labs has launched Real-SWE, evaluating AI coding agents on licensed, private enterprise codebases across 640 scored rollouts.
Real-SWE benchmarks frontier coding agents across licensed enterprise software repositories.
- Specific Labs released Real-SWE, an evaluation benchmark testing AI coding agents against private production codebases licensed from operational companies.
- The benchmark evaluates eight model and harness configurations across ten complex engineering tasks, totaling 640 scored rollouts.
- Fable 5.1 paired with Claude Code achieved the highest resolution rate at 38.8%, followed by GPT-6 Astra on Codex CLI at 33.8%.
- Unlike synthetic tasks, Real-SWE tests changes across infrastructure, databases, third-party business services, and an average of 11 edited files per reference solution.
Evaluating Coding Agents on Proprietary Enterprise Code
AI benchmarking firm Specific Labs has released Real-SWE, a specialized benchmark designed to evaluate frontier AI coding agents against private, real-world software engineering environments. Unlike conventional coding benchmarks that draw from public open-source repositories or rely on synthetic problem sets, Real-SWE sources its evaluation tasks directly from private production codebases licensed from active enterprises.
According to release details shared by Janak Sunil on X and the benchmark documentation, the evaluation was designed to address a critical limitation of public code benchmarks: data leakage and unrepresentative code architectures. Because public repositories are routinely indexed in large language model training corpora, models can exhibit inflated performance. By using private, out-of-distribution codebases, Real-SWE measures whether coding agents can navigate proprietary architectures that do not appear anywhere on the public internet.
Benchmark Architecture and Environment Complexity
The codebases included in Real-SWE were selected through a screening process focused on organizations with established engineering standards and active production usage. Examples include a consumer events platform with over 200,000 users ranking in the top 100 on the App Store, a consumer fintech platform processing more than 100,000 bank statements, and enterprise AI sales platforms with multi-service operational requirements.
Real-SWE tasks focus on engineering issues with actual business implications, such as calculating jurisdictional tax liabilities, fixing billing pipelines, handling customer migrations, and reconciling external ledger records. Instead of isolating language models in raw completion environments, the benchmark tests model-and-harness combinations using native command-line agent frameworks such as Claude Code, Codex CLI, Gemini CLI, Grok Build, Muse Code, and Kimi Code.
Each task environment exposes the specific external services and infrastructure dependencies required to complete the workflow. Sandboxes integrate tools across languages and operational systems, including Docker, Kubernetes, AWS emulators, PostgreSQL, MySQL, MongoDB, Redis, Gel, InfluxDB, Node.js, Go, Python, and Vitest, as well as business integrations such as Linear MCP, Slack, Intercom, ClickUp, and Google Drive.
Evaluation Results and Frontier Model Rankings
The benchmark tested eight model and harness pairings across ten tasks, totaling 640 scored rollouts. Performance is measured using a resolution rate equivalent to pass@1, calculated as the average over eight independent runs per task, with 95% confidence intervals reported:
- Fable 5.1 (Claude Code): 38.8% resolution rate
- GPT-6 Astra (Codex CLI): 33.8% resolution rate
- Gemini 3.8 Flash (Gemini CLI): 31.2% resolution rate
- GLM 5.3 (Claude Code): 28.8% resolution rate
- Grok 4.6 (Grok Build): 23.8% resolution rate
- Muse Spark 1.3 (Muse Code): 23.8% resolution rate
- Kimi K3 (Kimi Code): 18.8% resolution rate
- GPT-5.6 Sol (Codex CLI): 16.2% resolution rate
Fable 5.1 running inside Claude Code topped the initial leaderboard, followed by GPT-6 Astra using Codex CLI and Gemini 3.8 Flash on Gemini CLI.
Structural Complexity Compared to Existing Benchmarks
Specific Labs compared Real-SWE's task profile against existing coding evaluations, such as FrontierCode, DeepSWE, Terminal-Bench 3, and FrontierSWE v2. Instructions in Real-SWE carry a median length of 1,742 characters, positioned between Terminal-Bench 3 (1,584 characters) and DeepSWE (1,975 characters). However, the tasks exhibit significantly greater implementation dispersion.
The median number of files edited by reference solutions in Real-SWE is 11, compared to a median of 6 files in both FrontierCode and DeepSWE. Tasks require agents to discover implementation requirements across complex directory trees and adapt changes to existing conventions rather than modifying isolated modules. Analysis of run durations showed that agent failure was consistent across time horizons: 71.4% of rollouts under 10 minutes failed (70 of 98), while 73.4% of rollouts running 10 minutes or longer failed, illustrating that difficulty stems from architectural comprehension rather than time-limit constraints.
Enjoyed this?
Get more posts like this delivered to your inbox.
