Routing requests to cheaper models without hiding quality losses

Two routing errors with different costs

Under-routing sends a request that needed the strong model to the cheap one. The user gets a worse answer, and nothing in the cost dashboard shows it. Over-routing sends an easy request to the strong model. The answer is fine and the saving is smaller than it could be. Most routing projects report only the second error, because it is the one that shows up on the invoice.

The two errors trade against each other through the router's threshold, so decide which one you can tolerate before tuning. For a coding assistant, a weak answer costs an engineer a few minutes. For a claims letter, it may cost a complaint.

Build the acceptance test before switching anything on

Take a recent sample of real requests and run each through both models. Have experts, or a model judge whose grades you have checked against experts, mark each answer from each model as acceptable or not against a reference or their own judgement. Do not assume the strong model is always right: requests where neither answer is acceptable are a prompt or task problem and belong outside the routing figures. Then run the router on the same sample. A hypothetical result for 1,000 requests, where the strong answer was acceptable on all of them and the cheap answer on 700:

• Routed cheap and acceptable: 610
• Routed cheap but not acceptable, the under-routed requests: 40
• Routed strong although cheap was acceptable, the over-routed requests: 90
• Routed strong and needed: 260

Assume the cheap model costs a tenth as much per request and ignore the router's own cost. Sending everything to the strong model costs 1,000 units, this routing costs 415, but 4 percent of users get a worse answer. If the agreed limit is 1 percent, raise the threshold, treating this sample as a calibration set. Suppose the new threshold sends 560 requests to the cheap model, 9 of them under-routed, for 496 units. Then confirm on a separate held-out sample the threshold has never seen. Small counts need an interval: 9 misses in 1,000 fits a true rate of roughly 0.5 to 1.7 percent at 95 percent confidence, so a larger final sample may be needed to show the limit holds.

Rules, learned routers and cascades

Rules by feature, task type or tenant are the easiest to audit and may capture much of the available saving. Learned routers predict per request: RouteLLM trains them on human preference data and reports cutting cost by more than half on some benchmarks without lowering response quality. Cascades call the cheap model first and escalate when a check fails. FrugalGPT reports large savings this way, but escalated requests pay for two calls and wait for both, and the check itself has to be reliable.

Treat published savings as evidence that routing can work, not as a forecast. They come from benchmark traffic and price gaps that differ from yours. Gateways now ship routers too: LiteLLM's auto routing, in beta at the time of writing, classifies requests into tiers with heuristics, an LLM classifier or your own plugin. Whichever you use, it goes through the same acceptance test.

Fallbacks protect availability, not quality

A fallback fires on errors. In LiteLLM, a call that still fails after its retries moves to the next model group in an ordered list, and separate lists can handle context-window and content-policy errors. None of this checks the answer. If the fallback target is a model your acceptance test never covered, an outage turns into an untested quality change. LiteLLM also warns that a fallback to another provider cannot read encrypted reasoning from earlier turns.

Run every fallback target through the acceptance test, and rehearse failover in a non-production environment by triggering a real provider error, as the LiteLLM documentation recommends.

Knowing when routing goes wrong in production

Start by logging which model actually answered. With LiteLLM's auto routing, the response body keeps the alias the client called, and the serving model is exposed in the x-litellm-model-id response header, so capture it. Its spend logs also record attempted_fallbacks and original_model_group for each request, which separates fallback traffic from routed traffic. Then watch these signals per tier:

• A small sample of cheap-tier requests also sent to the strong model and compared, so under-routing is measured rather than guessed
• Users retrying or rephrasing shortly after a cheap-tier answer
• The share of traffic per tier, which drifts when request mix or router inputs change
• Fallback rate per provider, kept apart from routing decisions