prod-bench-1
October 2026
Contents
Summary
Coding benchmarks have a short life expectancy. HumanEval lasted about three years; its successors have lived far shorter lives.
We’d like ours to survive at least until the next model release, so we built it out of something models are still bad at: having the judgment of a senior engineer.
We're launching prod-bench-1: a private, agentic coding benchmark mined from the commits in a 33,000-commit production monorepo with 14 years of history.
Today’s frontier models struggle to see the full scope of a change. They're very good at making the exact change prescribed, or finding and fixing a bug.
Every task in prod-bench-1 is a short feature request where the obvious implementation misses something one step away.
The best model, Claude Opus 5.5, passes 51% of our benchmark set.
prod-bench-1 Leaderboard – pass@1
Results
| Model | pass@1 | pass@3 | pass^3 | Cost / trial | Minutes / trial |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 51.4% (54/105)42–61% | 69.5% | 33.3% | $2.54 | 18.9 |
| Claude Fable 5.1 | 45.7% (48/105)37–55% | 73.3% | 21.9% | $5.25 | 24.0 |
| Claude Sonnet 5.5 | 31.4% (33/105)23–41% | 44.0% | 21.1% | $0.43 | 12.7 |
| Grok 4.7 | 28.6% (30/105)21–38% | 45.0% | 15.4% | $5.39 | 20.5 |
| Gemini 4 Argon | 18.1% (19/105)12–27% | 25.0% | 13.5% | $5.69 | 23.9 |
| GPT-6.1 Sol | 9.5% (10/105)5–17% | 24.0% | 0.2% | $0.64 | 15.3 |
| GPT-6 Astra | 8.6% (9/105)5–15% | 19.8% | 0.8% | $3.92 | 11.4 |
| Kimi K3 | 7.6% (8/105)4–14% | 12.6% | 2.1% | $2.00 | 30.1 |
| Inkling | 6.7% (7/105)3–13% | 12.4% | 1.9% | $3.31 | 32.6 |
| Mistral Large 4 | 5.7% (6/105)3–12% | 13.5% | 0.2% | $3.21 | 41.0 |
| Nemotron 3.5 Super VL* | 5.7% (6/105)3–12% | 9.5% | 1.9% | $0.43 | 20.8 |
Given three tries, Claude Fable 5.1 solves about as many tasks as Claude Opus 5.5 (73% and 70% pass@3), but Claude Opus 5.5 is more consistent: the chance that all three of its trials pass (pass^3) is 33%, against 22% for Claude Fable 5.1. Claude Sonnet 5.5 solves fewer tasks but solves them reliably, so its pass^3 is close to Claude Fable 5.1’s. Grok 4.7 solves as many tasks as Claude Sonnet 5.5 in three tries (45% and 44% pass@3), but less reliably, at 15% pass^3. GPT-6.1 Sol and GPT-6 Astra solve a fifth to a fourth of the tasks in three tries, and Kimi K3, Inkling, Mistral Large 4, and Nemotron 3.5 Super VL a tenth to an eighth. These six almost never pass all three.
prod-bench-1 Cost vs. Performance
pass@1 against mean model cost per trial · Up and left is better
Vertical bars show 95% Wilson intervals on pass@1, over trials on these 15 tasks. Hover over or tap a dot to name it.Dashed line: Pareto frontier.
Task construction
Each task comes from merged work in the repository's history: usually one pull request, but sometimes a feature bundled together with the follow-up PRs that finished it. The agent gets a container with the repository at the commit before the change, its full git history up to that point, and everything needed to run the app's test suite. It also gets a short request written the way a PM or ops lead would write it.
The request states the feature and every product decision the tests depend on. It does not mention the parts of the change that follow from how the existing code already works: the second screen that shows the same value, the export that renders the same headers, the event other state changes already fire. Finding those is the task.
Environment
Every model runs in the same agent harness, opencode 1.19.0. Grok 4.7, Kimi K3, Inkling, Mistral Large 4, and Nemotron 3.5 Super VL were served through OpenRouter with zero data retention enforced on every request. Each attempt gets its own sandbox: a container with 12 GB of memory, a 90-minute time limit, and no internet access beyond the model's API. The task is designed to be completed in a single agent turn.
Grading
When the agent ends its turn, its processes are stopped and the hidden tests are restored. An attempt passes if and only if every graded test passes:
- Hidden tests that come from the original PR, rewritten to check behavior through existing interfaces (endpoints, jobs, models, pages) instead of the original implementation's new class names, method signatures, or exact messages. Any reasonable design passes.
- Existing tests around the changed code must still pass, so an attempt that breaks something already there fails (the agent has access to this test suite).
Before a task was accepted, we checked that a literal, request-only implementation fails it, that a different reasonable implementation passes it, and that its tests are stable across repeated runs. We also scanned the git history for earlier copies of the answer, such as a reverted and re-landed PR, to ensure that the answer cannot be found simply by looking through the git history.
Methodology
Most of the work is making sure a failure means the model missed something, not that the test was unfair. Our human experts used frontier LLMs to mine and design tasks from the repo, which has about 850k lines of production code and about 900k lines of test code.
| Stage | Tasks |
|---|---|
| Tasks built from merged pull requests | 350 |
| Deemed fair after the test audit | 156 |
| In the benchmark | 15 |
We performed multiple rounds of QC, with both expert-driven human-in-the-loop checks and fully agentic checks, to make sure that each task is fair and well-designed. Starting from the original proposed task, this was our methodology:
- A history-leak scan. Agents search git history, so tasks whose answer could be recovered from it, such as a re-landed change whose reverted original is still in the log, were dropped.
- A test audit before any model run. A separate auditor checked every task’s tests.
- Two patches per task. The request implemented literally must fail, and a different reasonable design must pass. Attempted by an expert with LLM assistance.
- Failure audits. An auditor read every failing trial on a candidate task, with its transcript and final diff, and classed it as legitimate, unfair, or void. An unfair failure meant fixing the test and re-grading every trial, or dropping the task.
Expert workforce
The tasks were built by a small team of staff-level engineers from FAANG+ companies. They mined and designed the tasks with frontier LLMs, wrote the second, different implementation each task must accept, and ran the human-in-the-loop checks mentioned above.
Failure modes
Each model fails in its own way.
Claude Opus 5.5 and Claude Fable 5.1 most often find the consequence the request leaves out, and Gemini 4 Argon and GPT-6 Astra most often build exactly what the request describes. It isn’t one-way, though. No model solves every task: Claude Opus 5.5 fails 3 of the 15 in all seven trials, and every other model passes at least one of those three.
Here are two tasks that show the scope of the consequences a model has to find.*
Example 1
Headers for repeated fields
- Prompt
- Fields inside a repeatable section all get the same column header in record views, so you can’t tell the first entry’s values from the second’s. Anywhere a field’s header is shown, prefix fields in a repeatable section with the section’s label (its name if it has no label) and the 1-based entry number, in the form ‘<section label> - <n>. ’ followed by the usual header, e.g. ‘Emergency contacts - 2. Phone’. The field-definitions endpoint should expose this as a
prefixstring on each repeated field (null for everything else) so the records table can render prefix + header. - The obvious change
- Compute the prefix in the endpoint and render it in the on-screen records table. Both match the prompt exactly.
- What it misses
- The downloadable CSV export builds its column headers through its own code path, from a separate serialization of the fields, so it still prints duplicate headers. “Anywhere a field’s header is shown” includes it. The original change touched seven files across the data model, the API, the table, and the export. Failing models either never found the export path or decided it was out of scope.
Example 2
Automation rules on tag counts
- Prompt
- Support ops wants an automation-rule condition on customer tags. Add condition type
customer_tag_countwith params category (billing, technical, or general), min, and max. It matches customers whose number of tags in that category falls in the range: bounds are inclusive, a missing bound is open, and with no bounds it matches everyone. Customers with no tags count as zero. In the customer list, each row should report the count undercustomer_tag_count, as {params, count} entries like the other count conditions. - The obvious change
- Add the condition, its range check, and the count in the customer list. It’s easy, and the failing trials built it correctly.
- What it misses
- Automation rules re-check a customer only when an event fires for them. Each existing condition has one: a plan change fires its own event, and so does a profile edit. Adding or removing a tag fires nothing, so a rule on tag counts never triggers when tags change. The fix is to fire an event on tag add and remove that the new condition reacts to.
* The tasks’ wording and details are changed, as our agreement with the data vendor that provided this repository requires.
Gemini 4 Argon tends to miss code right next to its change. When it fails, the code it needed to update usually sits a few dozen lines below the method it edited, in a file it had already read in full.
GPT-6 Astra keeps its sessions short. It packs many tool calls into each step and mentions a token budget the harness doesn’t impose, and its scope cuts seem to come from that.
Claude Fable 5.1 sometimes ignores its own research. Its subagents report the consequence, or file it under “not in scope but related”, and the main agent never acts on it.
Some Claude failures are a judgment call that went the wrong way: the transcript names the consequence, and the model sets it aside as out of scope or as a follow-up.
Access
The tasks are private, due to IP-sharing restrictions with the vendor that provided this data. To evaluate a model on them, or to talk about tasks like these for training, email benchmarks@sherpalabs.ai or get in touch below.
get in touch