Changing model providers during an outage without changing the product's promises

Failover changes more than the endpoint

A second model differs from the first in more than its URL. Instruction following, tool-call formatting, structured output reliability, refusals, context limits, tokenisation and therefore cost and latency can all change. The fallback may run under a different provider contract, region and retention policy. If it goes through another gateway or API key, it can even carry different permissions. Each of these is a promise the product made without anyone writing it down.

Write a fallback contract for each route

• Quality: passes the feature's evaluation set at an agreed threshold, with its own prompt variant
• Output: structured results validate against the same schemas, and downstream code rejects anything that does not
• Tools and permissions: the same tool list and credential scope, with nothing extra enabled by the fallback's framework
• Data terms: provider, region and retention approved by your data owners for what this feature sends
• Capacity and cost: the fallback's rate limits at failover volume and the expected cost per request
• Notice: whether operators or users are told that a fallback is active

If no route meets the contract, the honest fallback is a degraded mode: the previous non-AI process, a queued request or a clear message.

What should trigger a switch

Not every error justifies failover. Anthropic's API reference, for example, separates rate limiting (429), internal errors (500) and temporary overload (529). Invalid requests and rejected credentials usually point to a problem on your side, though what a provider rejects can differ between providers. Classify each failure and configure the recovery for it, and do not fail over automatically on errors that a fallback would not fix or that need your own investigation. Google's SRE book adds the retry rules: separate retriable from permanent errors, use randomised exponential backoff, limit retries per request and avoid retrying at several layers, since three layers that each retry three times can turn one user action into 64 attempts. When errors or latency on the primary stay above a threshold for an agreed period, switch the whole feature rather than individual requests, so that one conversation does not bounce between models.

Prove behavioural equivalence before the outage

Run the evaluation set on every fallback route on the same schedule as the primary, and after every change to prompts, models or tools. Compare the kinds of failure, not only the pass rate. A fallback that refuses slightly more often may be acceptable; one that calls a write tool with wrong arguments is not. Keep a prompt variant per model in version control and test it against the same tool schemas.

A hypothetical example: on a 300-case set for a support assistant, the primary passes 282 cases and the fallback 270, but the fallback gets 6 of the 20 refund cases wrong by citing an outdated policy. The decision: during failover the refund tool is switched off and refund questions go to a person, while everything else stays on the fallback.

Exercise the fallback before you need it

Google's SRE book warns that a degraded mode rarely used in normal operation is a code path that often does not work, and suggests deliberately exercising it. For a model fallback, that means routing a small share of traffic, or shadow traffic, to it routinely, or running scheduled drills, so that keys, quotas, prompts and monitoring are known to work. Shadow runs must isolate, stub or suppress write tools so they never repeat a real refund, email or other business action, whatever the provider's data terms allow. Confirm in advance that the fallback's rate limits can absorb the volume it will receive.

Switching back, and models that are retired

Switch back deliberately: after the primary has been healthy for an agreed period, gradually, and while watching the same signals. Record which requests each route served, so that quality reviews and cost reports stay accurate. Planned changes need the same discipline. Anthropic's deprecation policy, for example, gives deprecated models a retirement date after which requests fail, and notes that partner platforms set their own schedules. Treat a deprecation notice as a scheduled failover and apply the same contract. A practical next step is to write the contract for one feature and run its evaluation set against the fallback route before your next release.

AI engineering and architecture