Intelligent routing

Most teams building on LLMs are over-provisioned: they picked a frontier model once, pointed every request at it, and never went back. That is a rational choice. You cannot tell in advance which requests need the expensive model, so you pay frontier prices for all of them, including the majority a far cheaper model would answer just as well.

AICP's auto routing removes that trade-off. For every prompt it predicts the cheapest class of model that can still answer it well, and sends it there. Hard prompts still reach a frontier model; easy ones stop costing like they do.

This page explains how that decision is made. For what it delivers and how we measured it, see Routing performance. For the configuration surface, see Routing.

Two stages, two different questions

A routing decision answers two questions, and AICP keeps them separate because they have different answers for different customers.

   your prompt
        │
        ▼
  ┌───────────────────────────────────────────┐
  │ 1 · Tier prediction                       │   "How capable does the
  │    prompt → small | medium | large        │    model need to be?"
  └───────────────────┬───────────────────────┘
                      ▼
  ┌───────────────────────────────────────────┐
  │ 2 · Model selection                       │   "Given that floor, which
  │    governance filter → strategy → winner  │    model wins under your
  └───────────────────┬───────────────────────┘    policy?"
                      ▼
              provider call
                      │
                      ▼
        response + X-Router-* headers
              + auditable decision record

Stage 1 is a property of the prompt. Stage 2 is a property of your organisation: your enabled providers, your regions, your budget ceilings, your strategy. Splitting them means your compliance rules never have to be encoded into a model, and the model never has to be retrained because you added a provider.

Stage 1: predicting the tier

The router turns each prompt into a calibrated probability across tiers, rather than a bare pick. That distinction is what makes the rest of the system controllable: a probability can be compared against a threshold, and a threshold can be tuned to your risk. A hard label cannot.

It then walks the tiers from cheapest upward, accumulating confidence as it goes, and stops at the first tier it is confident is sufficient:

confidence so far = 0
for tier in [small, medium, large]:      # cheapest first
    confidence so far += P(tier)
    if confidence so far >= quality bar:
        route here
otherwise: route to the largest tier      # nothing cleared the bar, play safe

Two things follow, and both matter:

• Failure escalates, never degrades. If no cheap tier clears the bar, the request goes to the strongest tier. Uncertainty costs you money, not quality.
• The middle tier actually gets used. An earlier design compared each tier's probability on its own, which almost never selected the middle tier and collapsed routing into cheap-versus- frontier. Accumulating confidence instead makes the middle tier the workhorse: that is where most of the saving comes from, because the frontier tier costs ~24× the middle one.

Traffic split at the deployed dial

67% of traffic → middle tier · 14% → cheap tier · 19% → escalated to frontier

The classifier is a compact model over a semantic representation of the prompt plus some lightweight structural signals. It runs in-process, adds ~12 ms at p50 to a call that takes seconds, and makes no network requests of its own: no LLM sits in the routing hot path. Prompts that are empty or degenerate short-circuit to the cheapest tier; if the classifier is unavailable for any reason, routing falls back to your configured strategy rather than failing the request.

The dial

One number controls how aggressively auto saves money.

Dial setting	Behaviour	Result
Low	Cheap tiers clear the bar easily	Cheapest, most risk
High	Few cheap tiers clear it, more escalation	Safest, dearest

We do not guess where to set it. The dial is calibrated: candidate settings are swept across held-out prompts and we take the cheapest setting that still meets a quality-retention target.

That sweep also produces a cost/quality frontier, which is really a menu. At an aggressive setting you keep roughly 95% of frontier-model quality for about 6% of frontier cost; at a cautious one you keep all of it for about 36%. The deployed default sits between them, at roughly 99% of frontier quality for 17% of frontier cost. The point on that curve is a decision, not a default we impose. See Routing performance for the full table and the baselines it should be judged against.

note

A tier is a quality floor, not a fixed model. Every model in a tier has been measured to clear that tier's floor, which is why the router can optimise freely inside a tier without putting quality at risk. How those floors were established is covered in Routing performance.

How the router is trained

The router is trained to predict adequacy, not difficulty. That distinction is the whole design, and it shows up in how the training labels are made.

Labels come from outcomes. Each training prompt is labelled with the cheapest tier whose reference model actually answered it acceptably, graded independently against a correctness threshold, not guessed from the prompt's appearance. Prompts that no tier answered acceptably are dropped rather than assigned a tier nothing earned. The target the router learns is therefore "which tier is sufficient here", not "how hard does this look". The two come apart constantly: a long prompt full of code can be trivially answerable, and a twelve-word question can need a frontier model. Raising the correctness threshold makes the labels stricter and shifts traffic upward; it is a deliberate choice, and we report the resulting mix rather than targeting one.

Training is offline and self-contained. Prompt representations are computed once and cached, so fitting makes no model calls at all. Nothing about training touches the serving path, and no customer traffic is used.

A slice is held back. The data is split with the tiers kept proportionally represented, and a test portion is locked away and never fitted on.

The model is chosen by the routing rule, not by classification accuracy. This matters more than it sounds. On traffic where most prompts are answerable by the cheapest tier, a classifier that always predicts "cheapest" scores impressively on accuracy while routing nothing useful. Candidate models are therefore judged on the cost and quality they actually realise under the rule described above, the same rule that runs in production.

The dial is calibrated last, against held-out prompts, once the model is fixed.

The result ships as a versioned artefact. It is retrained deliberately, when the tier reference models change or the catalogue moves enough to shift what each tier represents, not on a schedule, so a given release routes the same way every time.

Stage 2: choosing the model

Once a tier is fixed, AICP expands it into every (model, provider) pair you could legitimately be served by, then narrows and ranks.

First, a hard governance filter. This runs before selection and applies identically whether the tier came from the router or you named a model yourself. A candidate is excluded if the provider is disabled, failing health checks, not authorised for your organisation, or in a provider category you have not allowed; if the model is unavailable; if the provider is outside your permitted regions; if it lacks a capability the request needs; or if it would breach your configured latency or per-request cost ceiling. Exclusions are recorded with their reason, so "why wasn't this provider used?" is always answerable.

Then your strategy ranks what survives. cheapest, fastest and highest_quality rank on a single axis. balanced, the default, normalises cost, latency and quality across the surviving candidates and combines them under weights you control (by default weighted toward cost, then latency and quality evenly). Your preferred-provider order applies as a bounded nudge on top, so a preference can break a close call but cannot override a large difference. The five strategies are documented in Routing.

Latency comes from your traffic, not a benchmark

Every model starts with a benchmark-derived latency estimate. That estimate is a cold-start value only. Once a model has accumulated enough recent requests on your platform traffic, the observed figure replaces it. Routing then reflects how each model actually performs for you: your regions, your prompt shapes, that provider's real load.

Two details make the number trustworthy:

• It is normalised per output token. Raw end-to-end latency is dominated by how long the answer was, so a model that happens to receive short prompts would otherwise look faster than it is. Normalising removes that bias and makes streaming and non-streaming calls comparable.
• It never falls back to infinity. A model with too little recent traffic reverts to its estimate rather than being ranked last, so low-volume models stay in genuine contention instead of being quietly starved of the traffic they would need to prove themselves.

The switchover is continuous, not a one-way latch: a model whose traffic dries up drops back to its estimate automatically.

Using auto with streaming, tools and multi-turn

	Behaviour
Streaming	Supported. Routing is resolved before the stream opens, so a routing failure still returns a normal JSON error rather than a half-written stream.
Multi-turn	The router reads the most recent turns of the conversation, not just the latest message, so a short follow-up carries the context it depends on. The tier is decided per request, so a conversation can move between tiers as it gets harder or easier.
Tool calling	Supported. A request carrying tools is routed only to providers that declare function calling. The check is at the provider level: it does not rank how reliably an individual model invokes a tool once offered one, so on quality-sensitive tool traffic a higher tier or an explicit model is the safer choice.
Structured output	response_format is not supported yet. The field is ignored rather than honoured, and routing does not take it into account. To get JSON reliably today, name a model you know supports it and instruct the model in the prompt.

When things go wrong

Situation	What AICP does
No model in the predicted tier is eligible	Escalates upward to a stronger tier, never down to a weaker model. The response carries X-Router-Tier-Escalated.
Nothing is eligible at any tier	Returns a clear no_available_model error rather than serving something you excluded.
Your policy excludes every provider	Applies your fallbackBehavior: fail, or relax to commercial / global / cheapest. The relaxation is recorded on the decision.
A provider rate-limits or goes down mid-request	Re-routes to a different provider and retries, up to three attempts, excluding each failed provider as it goes.
Every provider is rate-limited	Returns 429 rather than hanging.

Two limits worth knowing up front. Cross-provider retry applies to non-streaming requests in auto mode. A streaming response that fails after the first token has already been sent is surfaced as a stream error rather than silently re-routed. And retry is a failover mechanism, not a load balancer: AICP reacts to a 429 rather than tracking per-provider rate-limit budgets in advance.

Everything is auditable

Routing that you cannot inspect is routing you cannot trust. Every request carries its decision back to you in headers:

Header	Meaning
X-Router-Tier	The tier chosen for the prompt (0 small · 1 medium · 2 large)
X-Router-Confidence	How confident the router was in that tier
X-Router-Model	The model actually served
X-Router-Estimated-Saving	Estimated fraction saved versus always using the large tier
X-Router-Tier-Escalated	Present only when the request was escalated

And every decision is persisted with its full candidate set, the exclusions and their reasons, the strategy applied and the resulting cost, all retrievable through routing history or the dashboard's request explorer.

See also

• Routing: configuring strategies, fallback and aliases
• Routing performance: how the router was evaluated, and what it delivers
• Routing in the JavaScript SDK · in the Python SDK