Routing performance
Intelligent routing describes how a request reaches a model. This page describes how the inputs to that decision are established: where each model's quality figure comes from, how precise it is, and how often it is refreshed.
Figures on this page were measured in August 2026. They are re-measured as the catalogue changes; see Re-measurement for what is refreshed and when.
Model quality is measured, not asserted
Routing is only as good as what it believes about the models it routes to. Quality figures in a model catalogue are usually copied from published benchmarks, which measure something other than the traffic you send and go stale silently. AICP measures its own.
Every model in the catalogue is run through a fixed instrument. This is a measurement instrument. It scores models so the tiers mean something; it is not what the router was trained on. For that, see how the router is trained.
| Component | What it fixes |
|---|---|
| Versioned prompt corpus | A frozen set spanning reasoning, mathematics, knowledge, open-ended and retrieval-style tasks. Objectively checkable items are graded by exact match; the rest by an independent LLM grader. |
| Frozen grader specification | Model, temperature, rubric and output contract, fingerprinted, so a change to grading is detectable rather than silent. |
| Per-model generation parameters | A shared configuration measures our settings rather than the model. Reasoning models get output budgets sized for reasoning overhead; models that reject a shared temperature get their own. |
| Two anchor models | Every score sits on one comparable axis instead of being calibrated separately against a moving benchmark. |
| Snapshot-keyed ledger | A record is keyed by corpus, grader, parameters and the provider's resolved model snapshot. When a provider moves a -latest alias underneath a stable name, the affected record is invalidated instead of looking fresh. |
Grader validity was established before use: it agreed with the reference grading on 96.5% of a validation sample (Cohen's κ = 0.91, n = 173), and showed no self-preference. Its lowest agreement was on answers from its own model family.
Each model is measured on ~250 graded prompts, executed through the same serving path your requests take, so end-to-end latency is captured at the same time. The catalogue is measured continuously as models are added, so the number of models on the axis grows; every score on it was produced by the same instrument.
Tiers, and why models inside one are not ranked
A tier is a quality floor. Membership means a model has been measured to clear that floor, which is what makes it safe for your strategy to optimise for cost or latency inside a tier.
At this sample size the instrument resolves quality differences of about 0.046. Models closer
together than that are treated as tied rather than ranked, because an order between them
would be measurement noise. This is why balanced and cheapest are usually better choices than
highest_quality within a tier: the quality difference they would be trading against is often
not real, whereas the cost and latency differences are.
Two consequences worth knowing:
- Model outputs are not deterministic. Re-asking the same model the same questions, roughly 8% of prompts change between correct and incorrect. Published intervals are wider than that re-measurement spread, so they are conservative rather than blind to it.
- A model whose measurement is inconclusive is marked provisional. In production, provisional models are excluded from automatic routing by default; you can still select them explicitly.
What automatic routing delivers
Measured against routing every request to a frontier model, the baseline most teams actually start from:
~83% lower cost, retaining ~99% of that model's quality, on traffic where a meaningful share of requests do not require a frontier model.
Measured quality at the calibrated dial setting was 0.882, 95% CI [0.873, 0.890], against the frontier model's own 0.890, over ~4,200 held-out prompts that were not used to fit the router, with intervals from ~1,000 bootstrap resamples. Quality is each request's actual graded score from the model that served it, and cost is each request's actual token spend at current provider prices. Neither is a per-tier constant assumed in advance. Figures are from the dial sweep of 2026-08-10 and are re-derived whenever the router or the catalogue changes.
The dial moves that trade-off, and the sweep is the menu:
Retention above 100% is not a typo: the router escalates the prompts that need it, and on some prompts the tier below the frontier model happens to score better.
The baseline matters, and this is the honest version.
Automatic routing is not cheaper, it costs more. What it buys is measurably better quality: 0.882 [0.873, 0.890] against that model's 0.863, an interval that clears it outright rather than overlapping it.
So auto is built for teams over-provisioned to a frontier model, paying frontier prices for
the majority of requests that do not need one. If you already send everything to a capable mid-tier
model and are satisfied with the results, auto has little to offer you at this point in time.
Routing overhead itself is ~12 ms at p50, with no additional model call and no additional tokens billed.
How these numbers are produced
Two measurement paths stand behind this page:
- Cost and quality trade-offs (the figures above) come from offline simulation over a large body of prompts whose answers had already been produced and graded across many models independently of us. This makes it possible to evaluate thousands of routing decisions, and every counterfactual baseline, without the result depending on the day it was run.
- Model quality, latency and cost are measured live, through the same serving path your requests take.
A prompt's assignment to the held-out set is a function of the prompt itself, not of its position in the corpus. The held-out set therefore does not move when the dial, the sample size or the corpus changes, so two runs quoted here are comparable, and a better number cannot be produced by reshuffling. Every reported figure is quoted alongside the baselines it should be judged against: the cheapest tier used for everything, the mid tier used for everything, the frontier tier used for everything, and a perfect-knowledge oracle.
Re-measurement
| What | Cadence |
|---|---|
Observed latency per (provider, model) | Recomputed hourly over a rolling 5-day window |
| Model availability | Checked daily; models that stop responding are excluded from routing |
| Catalogue prices and context windows | Audited against live provider data; corrections applied to the catalogue |
| Model quality | Re-measured when a model is added, or when its provider snapshot changes |
Because quality records are keyed by the provider's resolved model snapshot, a provider silently updating a model behind a stable alias invalidates that model's record rather than leaving a stale figure in place.
Limits of these figures
- They describe traffic where some requests genuinely need a stronger model than others. Prompt-level routing cannot create a saving on traffic where every tier performs about equally well. On such traffic the correct answer is to pin a cheap model rather than route.
- Cost figures depend on the models you have enabled and their current provider prices.
- Quality figures are measured on a general corpus. They are not a substitute for evaluating routing on your own traffic before relying on a specific saving.
See also
- Benchmark: MMLU-Pro: the same router measured on a public benchmark, against Claude Haiku 4.5 and Sonnet 4.6, with the training-span questions excluded
- Intelligent routing: how the routing decision is made
- Routing: configuring strategies, fallback and aliases