Benchmark: MMLU-Pro
Routing performance reports how the router behaves on our own measurement corpus. This page reports something different and, for most readers, more useful: how it performs on a public benchmark. :::
Headline
+5.9 percentage points more accurate and 37.5% cheaper, on the same questions, at the same time. Both differences are statistically resolved rather than within noise.
Against Claude Sonnet 4.6 — a substantially larger and more expensive model — quality-first routing gives up 3.4 points of accuracy for a 77% reduction in cost.
The full table
All six configurations answered the same 1,424 questions, through the same serving path. (The router configurations were each additionally measured on the full 3,826-question set; the table is restricted to what all six have in common so every column is a like-for-like comparison.)
| Configuration | Accuracy | Cost / 1k requests | Saving vs Sonnet | Saving vs Haiku |
|---|---|---|---|---|
| Router — cost-first | 0.769 | $0.66 | 93.0% | 81.0% |
| Router — balanced | 0.795 | $1.35 | 85.8% | 61.4% |
| Router — quality-first | 0.851 | $2.18 | 77.0% | 37.5% |
| Single mid-tier model (gpt-oss-120B) | 0.801 | $0.69 | 92.7% | 80.1% |
| Claude Haiku 4.5 | 0.792 | $3.49 | 63.2% | — |
| Claude Sonnet 4.6 | 0.885 | $9.47 | — | — |
Accuracy is the share of questions answered correctly. Cost is actual measured token spend at current provider prices — not an estimate, not a per-tier constant assumed in advance.
The three router rows are the same router under three weight settings. Moving from cost-first to quality-first buys +8.2 points of accuracy for 3.3× the cost, and every step on that path is a real, resolved difference rather than sampling noise.
Why you can trust these numbers
Benchmark results are easy to produce and easy to get wrong. Four things were done specifically to make this result hold up.
1. The router was not tested on what it was trained on
This is the failure that quietly invalidates most published routing benchmarks.
Our classifier is trained on a corpus that contains 11,786 of MMLU-Pro's 12,032 questions. Left alone, evaluating on MMLU-Pro would grade the router on questions it was fitted to — and would produce a perfectly ordinary-looking number that meant nothing.
Every question in the training span was therefore removed, leaving 3,826 questions the router has provably never seen. Because a question's split is a function of its own text, this filter is reproducible and cannot be tuned to flatter the result.
The router arms were evaluated on all 3,826 — the complete clean set, not a sample.
2. Comparisons are paired, on identical questions
Each configuration answered the same questions, so differences are computed per question and bootstrapped over 10,000 resamples. This is far stronger than comparing two independent averages, and it is why a 5.9-point difference can be stated as resolved rather than suggestive.
Where an interval spans zero we say so, and call it a tie.
3. Grading is mechanical
MMLU-Pro answers are option letters, so correctness is decided by exact match. No model grades another model. An answer that never states a choice counts as wrong, not as a missing sample — so a model cannot improve its score by failing to follow instructions.
4. The harness is calibrated against an independent source
The strongest check available: run models that a third party has already published figures for, and see whether we reproduce them.
| Model | Our harness | Artificial Analysis | Difference |
|---|---|---|---|
| gpt-oss-120b | 0.784 | 0.808 | −2.4 pt |
| Claude Haiku 4.5 | 0.779 | 0.760 | +1.9 pt |
Our measurements land within ±2.5 points of an independent evaluator (ArtificalAnalysis), in both directions. So, there is no systematic bias inflating our numbers. Two reference points is a small sample, but it is two more than most published routing comparisons offer.
What this does not show
The same discipline applies to the limits.
On this benchmark, balanced routing does not beat a single strong model yet. Measured across the full 3,641 paired questions, balanced and one fixed mid-tier model differ by 0.2 points (95% interval −1.3 to +0.9) . Quality-first routing is where routing earns its keep on MMLU-Pro; balanced does not, on this workload.
MMLU-Pro is narrow. It is academic multiple-choice across 14 subjects. It is a good public yardstick precisely because it is fixed and checkable, but it is not your traffic. Routing has least to offer where every tier performs about equally — and a benchmark where one capable mid-tier model does well across all subjects is exactly that kind of workload. Real traffic with genuinely mixed difficulty is where prompt-level routing has more room, and it is also where you should measure before relying on a specific number.
Latency is not reported here. These runs were executed concurrently to make the full benchmark practical, which inflates per-request timings through queueing. Publishing those figures as if they described a single user's experience would be misleading. See Routing performance for latency measured properly.
See also
- Routing performance: how model quality, latency and cost are established
- Intelligent routing: how a request reaches a model