Suite results
Suite Score Avg / 20 Tests n tok/s Run
▶ General capability (13-task real-workload suite) 1a 109.2 / 260 8.4 13/13 n=1 74.4 run 2026-09-06
| Test | Category | Score | | tok/s |
| A1 | Dev/Ops Scripting | 7/20 | | 97 |
| A2 | Dev/Ops Scripting | 14/20 | | 101.7 |
| A3 | Dev/Ops Scripting | 7/20 | | 60.7 |
| A4 | Dev/Ops Scripting | 16/20 | | 68.3 |
| A5 | Dev/Ops Scripting | 9/20 | | 64.5 |
| A6 | Dev/Ops Scripting | 4/20 | | 68.5 |
| B1 | Document Processing | 17/20 | | 74.3 |
| C1 | Content Production | 13/20 | | 71.4 |
| C2 | Content Production | 1/20 | | 58.2 |
| C3 | Content Production | 13/20 | | 69.8 |
| C4 | Content Production | 8/20 | | 76 |
Category mean Dev/Ops Scripting 9.5Document Processing 17Content Production 8.8
▶ Agentic tool-calling & protocol adherence 1b 148.8 / 160 18.6 8/8 n=1 98.6 run 2026-09-06
| Test | Category | Score | | tok/s |
| WA1 | Tool Calling & Protocol Adherence | 20/20 | | 95.4 |
| WA2 | Tool Calling & Protocol Adherence | 20/20 | | 99.4 |
| WA3 | Tool Calling & Protocol Adherence | 20/20 | | 96.9 |
| WA4 | Tool Calling & Protocol Adherence | 20/20 | | 97.3 |
| WA5 | Tool Calling & Protocol Adherence | 20/20 | | 98.3 |
| WB1 | Agent Robustness & State Management | 18/20 | | 94.2 |
| WB2 | Agent Robustness & State Management | 17/20 | | 98.9 |
| WB3 | Agent Robustness & State Management | 14/20 | | 81 |
Category mean Tool Calling & Protocol Adherence 20Agent Robustness & State Management 16.3
▶ Coding depth 1c 79.2 / 180 8.8 9/9 n=1 63.7 run 2026-10-02
| Test | Category | Score | | tok/s |
| K1 | Coding — Concurrency | 16/20 | | 70.8 |
| K2 | Coding — Algorithms | 0/20 | | 54.3 |
| K3 | Coding — Security Review | 9/20 | | 70.1 |
| K4 | Coding — Performance | 0/20 | | 57.5 |
| K5 | Coding — Refactoring Under Constraints | 12/20 | | 59.5 |
| K6 | Coding — Test Writing | 13/20 | | 70.8 |
| K7 | Coding — React Component (typed) | 6/20 | | 64.4 |
| K8 | Coding — TypeScript Type System | 7/20 | | 62.9 |
| K9 | Coding — React Debugging | 16/20 | | 59.2 |
Category mean Coding — Concurrency 16Coding — Algorithms 0Coding — Security Review 9Coding — Performance 0Coding — Refactoring Under Constraints 12Coding — Test Writing 13Coding — React Component (typed) 6Coding — TypeScript Type System 7Coding — React Debugging 16
▶ Content-production depth 1e 76.2 / 120 12.7 6/6 n=1 80.6 run 2026-09-29
| Test | Category | Score | | tok/s |
| E1 | SOW & Requirements Decomposition | 17/20 | | 84.6 |
| E2 | SOW & Requirements Decomposition | 20/20 | | 86.5 |
| E3 | Executive & Proposal Content | 13/20 | | 81 |
| E4 | Executive & Proposal Content | 9/20 | | 85 |
| E5 | Executive & Proposal Content | 15/20 | | 81 |
| E6 | Executive & Proposal Content | 2/20 | | 56.6 |
Category mean SOW & Requirements Decomposition 18.5Executive & Proposal Content 9.8
▶ Long-context retrieval & synthesis 1g 90 / 120 15 6/6 n=1 56.3 run 2026-09-25
| Test | Category | Score | | tok/s |
| L1 | Long-Context — Needle Retrieval | 19/20 | | 57.8 |
| L2 | Long-Context — Grounded QA with Distractors | 20/20 | | 51.5 |
| L3 | Long-Context — Ops Log Reasoning | 17/20 | | 24.6 |
| L4 | Long-Context — Faithful Summarization | 2/20 | | 16.4 |
| L5 | Long-Context — Instruction Retention | 12/20 | | 22.4 |
| L6 | Long-Context — Full-Haystack QA | 20/20 | | 19.1 |
Category mean Long-Context — Needle Retrieval 19Long-Context — Grounded QA with Distractors 20Long-Context — Ops Log Reasoning 17Long-Context — Faithful Summarization 2Long-Context — Instruction Retention 12Long-Context — Full-Haystack QA 20
▶ Long-context multi-needle (MRCR) 1g2 11.1 / 60 3.7 3/3 n=1 32.8 run 2026-09-21
| Test | Category | Score | | tok/s |
| L7 | Long-Context — Multi-Needle Retrieval (MRCR) | 4/20 | | 22.9 |
| L8 | Long-Context — Multi-Needle Retrieval (MRCR) | 3/20 | | 9.1 |
| L9 | Long-Context — Multi-Needle Retrieval (MRCR) | 4/20 | | 5.5 |
▶ Live one-shot builds (runtime-verified) 1h 12 / 80 4 3/4 ● n=1 71.4 run 2026-09-21
| Test | Category | Score | | tok/s |
| H1 | Web-Estate One-Shot Builds | 6/20 | | 83.8 |
| H3 | Web-Estate One-Shot Builds | 5/20 | | 60.4 |
| H4 | Web-Estate One-Shot Builds | 1/20 | | 70.1 |
▶ Production replay (real agent workload) 1i 70.8 / 240 5.9 12/12 n=1 40.4 run 2026-10-09
| Test | Category | Score | | tok/s |
| I1 | Nightly Fix Replay | 1/20 | | 43.8 |
| I2 | Nightly Fix Replay | 3/20 | | 37.8 |
| I3 | Nightly Fix Replay | 0/20 | | 13.7 |
| I4 | Nightly Fix Replay | 6/20 | | 45.6 |
| I5 | Nightly Fix Replay | 0/20 | | 28.7 |
| I6 | Nightly Fix Replay | 7/20 | | 26.4 |
| I7 | Audit Judgment | 0/20 | | 33.6 |
| I8 | Audit Judgment | 17/20 | | 56.5 |
| I9 | Cron Dependency Reasoning | 13/20 | | 45 |
| I10 | Cron Dependency Reasoning | 4/20 | | 45.4 |
| I11 | Curation Judgment | 8/20 | | 47.1 |
| I12 | Curation Judgment | 12/20 | | 50.8 |
Category mean Nightly Fix Replay 2.8Audit Judgment 8.5Cron Dependency Reasoning 8.5Curation Judgment 10
▶ Brand-constrained static build (negative-constraint adherence) 1j 49 / 100 9.8 5/5 n=1 79.3 run 2026-09-24
| Test | Category | Score | | tok/s |
| J1 | Brand-Constrained Static Build | 11/20 | | 81.9 |
| J3 | Brand-Constrained Static Build | 10/20 | | 87.9 |
| J4 | Brand-Constrained Static Build | 6/20 | | 84.9 |
| J5 | Brand-Constrained Static Build | 11/20 | | 74 |
| J6 | Brand-Constrained Static Build | 11/20 | | 64.2 |
Each test is LLM-judged 0–20 against a fixed rubric; suite max = tests × 20.
An ADJ chip marks a cell carrying a subjective
adjustment from the maintainer; the judged score is the one shown, is unchanged,
and remains what the axes, ranks and tie bands are computed from. Hover for the reason.
Rows with a ▶ expand to the per-test breakdown.
Where fewer tests ran than the suite total, unrun tests count as zero toward the axis score.
See methodology.