150 tasks from one ultra-realistic megafund private equity deal world, graded against expert rubrics.
3.6%pass rate across 3,750 runs
| Rank | Model | pass@1 |
|---|---|---|
| 1 | Claude Fable 5.1 | 6.1% |
| 2 | GPT-6 Astra | 5.6% |
| 3 | Claude Opus 5.5 | 3.2% |
| 4 | GPT-6.1 Sol | 1.7% |
| 5 | Gemini 4 Argon | 1.3% |
15 tasks from a large production monorepo, each a one-paragraph request graded by the change’s hidden tests.
19.9%pass rate across 1,155 trials
| Rank | Model | pass@1 |
|---|---|---|
| 1 | Claude Opus 5.5 | 51.4% |
| 2 | Claude Fable 5.1 | 45.7% |
| 3 | Claude Sonnet 5.5 | 31.4% |
| 4 | Grok 4.7 | 28.6% |
| 5 | Gemini 4 Argon | 18.1% |
An auto-research benchmark designed to evaluate on statistical modeling and superforecasting. Results coming soon.
