GAUNTLET Bench
← Leaderboard

Ornith-1.5-35B-A3B Q4_K_M GGUF

Candidate Rank #18 of 53 · 7/8 GAUNTLET progress
Generalist — 90.5 / 100 G Agentic — 99 / 100 A Understanding — 88 / 100 U Needle — 100 / 100 N Thinking — pending T Live — 72.7 / 100 L Engineering — 88.5 / 100 E Throughput — 27.8 / 100 T

Hollow markers & dashed spokes: axis not yet scored

G
90.5
A
99
U
88
N
100
T
n/d
L
72.7
E
88.5
T
27.8

Specification

Parameters
35.95B total / ~3B active per token (MoE, 256 experts, 8 used)
Architecture
qwen35moe
Size on disk
20 GB
Quantization
Q4_K_M
Format
GGUF
Runtime
LM Studio
Class
Standard
Reasoning (CoT)
Yes — emits reasoning tokens
Internal ID
M34
Mean speed
95.6 tok/s across suites
Model card
huggingface.co

Suite results

Suite Score Avg / 20 Tests n tok/s Run
General capability (13-task real-workload suite) 1a 235.3 / 260 18.1 13/13 n=1 85.7 run 2026-09-06
Test Category Score tok/s
A1 Dev/Ops Scripting 20/20 92.5
A2 Dev/Ops Scripting 18/20 99.5
A3 Dev/Ops Scripting 20/20 101
A4 Dev/Ops Scripting 19/20 82.2
A5 Dev/Ops Scripting 20/20 76
A6 Dev/Ops Scripting 20/20 81
B1 Document Processing 20/20 100.2
B2 Document Processing 20/20 82.7
C1 Content Production 20/20 75.1
C2 Content Production 18/20 73.5
C3 Content Production 20/20 72.4
C4 Content Production 20/20 75.4

Category mean Dev/Ops Scripting 19.5Document Processing 20Content Production 19.5

Agentic tool-calling & protocol adherence 1b 158.4 / 160 19.8 8/8 n=1 105.5 run 2026-09-06
Test Category Score tok/s
WA1 Tool Calling & Protocol Adherence 20/20 96.2
WA2 Tool Calling & Protocol Adherence 20/20 88
WA3 Tool Calling & Protocol Adherence 20/20 93.7
WA4 Tool Calling & Protocol Adherence 20/20 73.8
WA5 Tool Calling & Protocol Adherence 20/20 103.6
WB1 Agent Robustness & State Management 20/20 95.5
WB2 Agent Robustness & State Management 20/20 109.3
WB3 Agent Robustness & State Management 18/20 96.2

Category mean Tool Calling & Protocol Adherence 20Agent Robustness & State Management 19.3

Coding depth 1c 159.3 / 180 17.7 9/9 n=1 95.1 run 2026-09-15
Test Category Score tok/s
K1 Coding — Concurrency 20/20 101.7
K2 Coding — Algorithms 20/20 123.3
K3 Coding — Security Review 20/20 101.2
K4 Coding — Performance 20/20 87.6
K5 Coding — Refactoring Under Constraints 20/20 79.6
K6 Coding — Test Writing 20/20 89.6
K7 Coding — React Component (typed) 20/20 90
K8 Coding — TypeScript Type System 19/20 88.7
K9 Coding — React Debugging 0/20 86.4

Category mean Coding — Concurrency 20Coding — Algorithms 20Coding — Security Review 20Coding — Performance 20Coding — Refactoring Under Constraints 20Coding — Test Writing 20Coding — React Component (typed) 20Coding — TypeScript Type System 19Coding — React Debugging 0

Doc/OCR vision 1d 160 / 160 20 8/8 n=1 94.9 run 2026-09-06
Test Category Score tok/s
D1 Doc/OCR — Transcription 20/20 36.7
D2 Doc/OCR — Document QA 20/20 32.3
D3 Doc/OCR — Field Extraction 20/20 54.8
D4 Doc/OCR — Document QA 20/20 44.9
D5 Doc/OCR — Transcription 20/20 61.8
D6 Doc/OCR — Field Extraction 20/20 35
D7 Doc/OCR — Degraded Input 20/20 72
D8 Doc/OCR — Degraded Input 20/20 71

Category mean Doc/OCR — Transcription 20Doc/OCR — Document QA 20Doc/OCR — Field Extraction 20Doc/OCR — Degraded Input 20

Doc/OCR — real-degraded tier 1d2 51.2 / 80 12.8 4/4 n=1 97.6 run 2026-09-06
Test Category Score tok/s
D9 Doc/OCR — Real Degraded Transcription 7/20 79.1
D10 Doc/OCR — Handwriting Extraction 17/20 52.8
D11 Doc/OCR — Degraded Table Transcription 14/20 92.1
D12 Doc/OCR — Degraded Table QA 13/20 76.2

Category mean Doc/OCR — Real Degraded Transcription 7Doc/OCR — Handwriting Extraction 17Doc/OCR — Degraded Table Transcription 14Doc/OCR — Degraded Table QA 13

Content-production depth 1e 118.8 / 120 19.8 6/6 n=1 102.6 run 2026-09-19
Test Category Score tok/s
E1 SOW & Requirements Decomposition 20/20 110.5
E2 SOW & Requirements Decomposition 20/20 123.5
E3 Executive & Proposal Content 20/20 93
E4 Executive & Proposal Content 20/20 97.6
E5 Executive & Proposal Content 19/20 101.4
E6 Executive & Proposal Content 20/20 83.5

Category mean SOW & Requirements Decomposition 20Executive & Proposal Content 19.8

Long-context retrieval & synthesis 1g 120 / 120 20 6/6 n=1 114.8 run 2026-09-18
Test Category Score tok/s
L1 Long-Context — Needle Retrieval 20/20 39.4
L2 Long-Context — Grounded QA with Distractors 20/20 49.5
L3 Long-Context — Ops Log Reasoning 20/20 48.1
L4 Long-Context — Faithful Summarization 20/20 64.5
L5 Long-Context — Instruction Retention 20/20 29.7
L6 Long-Context — Full-Haystack QA 20/20 14.1

Category mean Long-Context — Needle Retrieval 20Long-Context — Grounded QA with Distractors 20Long-Context — Ops Log Reasoning 20Long-Context — Faithful Summarization 20Long-Context — Instruction Retention 20Long-Context — Full-Haystack QA 20

Long-context multi-needle (MRCR) 1g2 60 / 60 20 3/3 n=1 99.3 run 2026-09-17
Test Category Score tok/s
L7 Long-Context — Multi-Needle Retrieval (MRCR) 20/20 45.4
L8 Long-Context — Multi-Needle Retrieval (MRCR) 20/20 82.1
L9 Long-Context — Multi-Needle Retrieval (MRCR) 20/20 16.7
Live one-shot builds (runtime-verified) 1h 59.1 / 80 19.7 3/4 ● n=1 90.1 run 2026-09-06
Test Category Score tok/s
H1 Web-Estate One-Shot Builds 19/20 103.6
H2 Web-Estate One-Shot Builds 20/20 84.9
H4 Web-Estate One-Shot Builds 20/20 81.7
Production replay (real agent workload) 1i 163.2 / 240 13.6 12/12 n=1 74.5 run 2026-09-17
Test Category Score tok/s
I1 Nightly Fix Replay 9/20 93.2
I2 Nightly Fix Replay 8/20 69.5
I3 Nightly Fix Replay 9/20 57.7
I4 Nightly Fix Replay 10/20 68
I5 Nightly Fix Replay 14/20 60
I6 Nightly Fix Replay 10/20 64.7
I7 Audit Judgment 13/20 79.7
I8 Audit Judgment 20/20 77.8
I9 Cron Dependency Reasoning 20/20 74.7
I10 Cron Dependency Reasoning 20/20 71.7
I11 Curation Judgment 20/20 77.4
I12 Curation Judgment 10/20 77.8

Category mean Nightly Fix Replay 10Audit Judgment 16.5Cron Dependency Reasoning 20Curation Judgment 15

Brand-constrained static build (negative-constraint adherence) 1j 83 / 100 16.6 5/5 n=1 91.7 run 2026-09-10
Test Category Score tok/s
J1 Brand-Constrained Static Build 20/20 95.9
J3 Brand-Constrained Static Build 17/20 88.1
J4 Brand-Constrained Static Build 15/20 100
J5 Brand-Constrained Static Build 17/20 80.7
J6 Brand-Constrained Static Build 14/20 88.6

Each test is LLM-judged 0–20 against a fixed rubric; suite max = tests × 20. An ADJ chip marks a cell carrying a subjective adjustment from the maintainer; the judged score is the one shown, is unchanged, and remains what the axes, ranks and tie bands are computed from. Hover for the reason. Rows with a ▶ expand to the per-test breakdown. Where fewer tests ran than the suite total, unrun tests count as zero toward the axis score. See methodology.