Hollow markers & dashed spokes: axis not yet scored
Specification
- Parameters
- 1B
- Architecture
- gemma3_text
- Size on disk
- 0.74 GB
- Quantization
- QAT 4bit
- Format
- MLX
- Runtime
- LM Studio
- Class
- Small (under 8B total parameters — phone/edge class)
- Reasoning (CoT)
- No
- Internal ID
- M26
- Mean speed
- 343.8 tok/s across suites
- Model card
- huggingface.co
Suite results
General capability (13-task real-workload suite) 1a 74.1 / 260 5.7 13/13 n=1.1 350.9 run 2026-08-18
| Test | Category | Score | tok/s | |
|---|---|---|---|---|
| A1 | Dev/Ops Scripting | 7/20 | 320.4 | |
| A2 | Dev/Ops Scripting | 2/20 | 307.8 | |
| A3 | Dev/Ops Scripting | 2/20 | 314.6 | |
| A4 | Dev/Ops Scripting | 0/20 | 314.7 | |
| A5 | Dev/Ops Scripting | 6/20 | 300.4 | |
| A6 | Dev/Ops Scripting | 2/20 | 306.4 | |
| B1 | Document Processing | 17/20 | 282.3 | |
| B2 | Document Processing | 7/20 | 311.3 | |
| B3 | Document Processing | 4/20 | 329.5 | |
| C1 | Content Production | 10/20 | 320.7 | |
| C2 | Content Production | 7/20 | 310.3 | |
| C3 | Content Production | 7/20 | 330.9 | |
| C4 | Content Production | 3/20 | 317.7 |
Category mean Dev/Ops Scripting 3.2Document Processing 9.3Content Production 6.8
Agentic tool-calling & protocol adherence 1b 52.8 / 160 6.6 8/8 n=1 351.4 run 2026-08-18
| Test | Category | Score | tok/s | |
|---|---|---|---|---|
| WA1 | Tool Calling & Protocol Adherence | 7/20 | 93.9 | |
| WA2 | Tool Calling & Protocol Adherence | 16/20 | 122.5 | |
| WA3 | Tool Calling & Protocol Adherence | 1/20 | 129 | |
| WA4 | Tool Calling & Protocol Adherence | 12/20 | 93.5 | |
| WA5 | Tool Calling & Protocol Adherence | 0/20 | 127.7 | |
| WB1 | Agent Robustness & State Management | 9/20 | 327.4 | |
| WB2 | Agent Robustness & State Management | 5/20 | 308 | |
| WB3 | Agent Robustness & State Management | 3/20 | 122 |
Category mean Tool Calling & Protocol Adherence 7.2Agent Robustness & State Management 5.7
Coding depth 1c 33.3 / 180 3.7 9/9 n=1 351.3 run 2026-10-02
| Test | Category | Score | tok/s | |
|---|---|---|---|---|
| K1 | Coding — Concurrency | 5/20 | 339.9 | |
| K2 | Coding — Algorithms | 3/20 | 328.5 | |
| K3 | Coding — Security Review | 4/20 | 333.3 | |
| K4 | Coding — Performance | 11/20 | 328.5 | |
| K5 | Coding — Refactoring Under Constraints | 6/20 | 333.6 | |
| K6 | Coding — Test Writing | 1/20 | 326.1 | |
| K7 | Coding — React Component (typed) | 1/20 | 317.8 | |
| K8 | Coding — TypeScript Type System | 0/20 | 324.9 | |
| K9 | Coding — React Debugging | 2/20 | 289.8 |
Category mean Coding — Concurrency 5Coding — Algorithms 3Coding — Security Review 4Coding — Performance 11Coding — Refactoring Under Constraints 6Coding — Test Writing 1Coding — React Component (typed) 1Coding — TypeScript Type System 0Coding — React Debugging 2
Content-production depth 1e 34.2 / 120 5.7 6/6 n=1 353.9 run 2026-09-28
| Test | Category | Score | tok/s | |
|---|---|---|---|---|
| E1 | SOW & Requirements Decomposition | 9/20 | 315.3 | |
| E2 | SOW & Requirements Decomposition | 4/20 | 301.8 | |
| E3 | Executive & Proposal Content | 7/20 | 335.7 | |
| E4 | Executive & Proposal Content | 4/20 | 331.5 | |
| E5 | Executive & Proposal Content | 6/20 | 319.7 | |
| E6 | Executive & Proposal Content | 4/20 | 317.7 |
Category mean SOW & Requirements Decomposition 6.5Executive & Proposal Content 5.3
Live one-shot builds (runtime-verified) 1h 4 / 80 1 4/4 n=1 356.2 run 2026-09-21
| Test | Category | Score | tok/s | |
|---|---|---|---|---|
| H1 | Web-Estate One-Shot Builds | 2/20 | 278.8 | |
| H2 | Web-Estate One-Shot Builds | 1/20 | 328.6 | |
| H3 | Web-Estate One-Shot Builds | 1/20 | 325.3 | |
| H4 | Web-Estate One-Shot Builds | 0/20 | 293 |
Production replay (real agent workload) 1i 17.6 / 240 1.6 11/12 ● n=1 299.1 run 2026-10-09
| Test | Category | Score | tok/s | |
|---|---|---|---|---|
| I1 | Nightly Fix Replay | 1/20 | 210.8 | |
| I2 | Nightly Fix Replay | 0/20 | 270.4 | |
| I4 | Nightly Fix Replay | 0/20 | 263.9 | |
| I5 | Nightly Fix Replay | 0/20 | 259.5 | |
| I6 | Nightly Fix Replay | 1/20 | 175.3 | |
| I7 | Audit Judgment | 1/20 | 290.3 | |
| I8 | Audit Judgment | 8/20 | 291.8 | |
| I9 | Cron Dependency Reasoning | 1/20 | 271.6 | |
| I10 | Cron Dependency Reasoning | 0/20 | 284.8 | |
| I11 | Curation Judgment | 6/20 | 276.3 | |
| I12 | Curation Judgment | 0/20 | 302.1 |
Category mean Nightly Fix Replay 0.4Audit Judgment 4.5Cron Dependency Reasoning 0.5Curation Judgment 3
Brand-constrained static build (negative-constraint adherence) 1j 10 / 100 2 5/5 n=1 344 run 2026-09-24
| Test | Category | Score | tok/s | |
|---|---|---|---|---|
| J1 | Brand-Constrained Static Build | 3/20 | 309.8 | |
| J3 | Brand-Constrained Static Build | 0/20 | 317.1 | |
| J4 | Brand-Constrained Static Build | 4/20 | 303.6 | |
| J5 | Brand-Constrained Static Build | 1/20 | 308.7 | |
| J6 | Brand-Constrained Static Build | 2/20 | 292.3 |
Each test is LLM-judged 0–20 against a fixed rubric; suite max = tests × 20. An ADJ chip marks a cell carrying a subjective adjustment from the maintainer; the judged score is the one shown, is unchanged, and remains what the axes, ranks and tie bands are computed from. Hover for the reason. Rows with a ▶ expand to the per-test breakdown. Where fewer tests ran than the suite total, unrun tests count as zero toward the axis score. See methodology.