What an LLM gateway does, and when a company needs one

The problems an LLM gateway solves

A gateway solves coordination problems. They appear when model use spreads across teams, and no single application can fix them on its own. These are the five we hear most often, in the words people use:

• 'Every team has its own provider keys and nobody knows how many exist.' The gateway holds the provider keys and gives each team, service or developer its own gateway key, which can be limited and revoked on its own. In LiteLLM, for example, each virtual key carries its permitted models, its budget and the team that owns it.
• 'We get one invoice and cannot say which team caused it.' Each request is logged with its key, team, model and token count, so spend can be summed per team and per use case.
• 'Switching to another model means code changes in every application.' Applications call one endpoint with a model name, and the gateway maps that name to a provider deployment. Switching becomes a configuration change, although prompts and outputs still have to be tested against the new model.
• 'When the provider has an outage, our features go down with it.' The gateway retries and falls back to another deployment or provider by rules you set in advance.
• 'Which data may go to which model?' A gateway can enforce the answer, but it cannot reach it. It controls which keys and teams may call which models and records what was sent. It does not know by itself whether a prompt contains customer data. Some gateways offer PII detection or redaction as an optional guardrail, yet the classification rules come from your data protection owners, and each application still decides what it puts into a prompt.

If none of these hurts yet, a gateway is one more service to patch, monitor and keep available, and it can wait. That is our judgement. It is also the most common good reason to put one off.

LLM cost allocation by team, task and outcome

What a gateway does and what stays outside it

A gateway governs the traffic to models. It does not run them, decide what data an application may read, or tell you whether the answers are any good. The table shows the usual split.

The split matters when you plan the work. In our experience, a project scoped as 'install a gateway' leaves the fallback evaluation, the log retention decision and the guardrail policy without an owner, and those are the parts auditors and incident reviews ask about.

LLM model routing without hidden quality lossLLM provider failover without changing quality or access

The main LLM gateway options as of 7 November 2026

Most companies choose between a self-hosted open-source gateway, the AI features of an API management product they already run, and a hosted gateway. The table covers seven common options and lists only facts we checked on the vendors' own pages on 7 November 2026. Ownership, names and features in this market changed several times this year, so check again before you decide.

Many of the 'best LLM gateway' and 'LiteLLM alternatives' lists you will find are published by vendors of a competing gateway. They are useful for collecting names and less useful for choosing. Two of the seven options above were acquired or agreed to be acquired in 2026 and a third was renamed. That is a practical reason to keep your applications on a standard API format, so the gateway itself can be replaced.

Running LiteLLM securely in production

How to choose an LLM gateway

Choose by where the gateway must run and who will operate it. Then check provider coverage, licence and data rules. The feature lists are so similar that they rarely decide anything on their own.

• Self-hosted or hosted. A self-hosted gateway runs in your network or cloud account, so prompts and logs stay where you deploy it, but you patch, scale and monitor it. A hosted gateway takes that work away and adds another party that processes your prompts, whose terms your data protection owners have to accept.
• Existing API management. If you already run Azure API Management or Kong, check their AI features first. They may cover keys, limits and logging without adding a new component that someone has to learn.
• Provider and client coverage. List the providers and API formats you need now and next year, including models deployed in your own Azure or AWS account and any self-hosted models. Coding agents add a constraint: Claude Code works with any gateway that exposes a supported API format, and Anthropic notes that the gateway has to be kept up to date as Claude Code adds capabilities.
• Licence. Separate the features you need from those in the free or open-source edition. With LiteLLM, SSO for more than five users, audit logs and tag-based budgets need the enterprise licence, so decide early whether to buy it or to build those parts around the gateway.
• EU data rules. Write down, for each option, where prompts, responses and logs are processed and stored. For hosted routers the answer can differ by type of data: OpenRouter processes EU-routed prompts in Europe but states that account, billing and other operational records may be processed in the US. For cloud models, the region and deployment type at the model provider matter as much as the gateway.
• Operations effort. Count the parts besides the gateway itself. LiteLLM's documentation recommends Redis for any deployment with more than one worker or replica: without it, rate limits are counted per worker, so a limit of 100 requests per minute across four workers lets 400 through, and a revoked key stays usable on workers that did not process the revocation. Add a database for keys and spend logs, upgrades, and on-call cover for a component every AI feature depends on.

What we often see, though it is no rule: teams that have standardised on Azure API Management or Kong start there. Teams that need cloud models in their own AWS or Azure account as well as self-hosted models, and have people to run it, tend to end up with a self-hosted LiteLLM. Teams without operating capacity and with fewer data restrictions use a hosted gateway.

EU data residency with Azure OpenAI and Amazon Bedrock

Where the gateway sits in your AI platform

The gateway sits in the middle of the platform: everything that calls a model goes through it, and it is the only component that holds the provider credentials. This is the path from the caller to the model:

1. 1

   Callers

   Applications, agents and coding agents such as Claude Code. Each authenticates with its own gateway key or through your identity provider, never with a provider key.

2. 2

   Gateway

   Checks the key, the model permission and the budget, applies guardrails, picks a deployment and writes a log entry. It runs in your network, or with the hosting provider if you chose a hosted gateway.

3. 3

   Models

   Cloud models deployed in your own Azure or AWS account, direct provider APIs where they are approved, and self-hosted models served with vLLM on Kubernetes. Only the gateway can reach them.

4. 4

   Logs and monitoring

   Gateway logs and metrics flow into the monitoring you already run, where cost per team, error rates and latency per provider are reported and alerted on.

This only works if traffic cannot go around the gateway. Keep provider keys in a secrets manager that only the gateway can read, restrict outbound network access to model endpoints, and compare each provider's usage report with the gateway logs to find direct calls. Coding agents can be pinned as well: as of 7 November 2026, setting allowedProviders to customEndpoint in Claude Code's managed settings makes it refuse sessions pointed anywhere other than your gateway, from Claude Code v2.1.285.

Agent secrets and credentials: storage and rotationRouting coding agents through an LLM gateway

Rolling out a gateway step by step

Roll out a gateway one team and one model route at a time, with budgets, logging and fallbacks tested before the first production request goes through it.

1. 1

   Take stock of current use

   List every provider account, key and team that calls models today, and what each spends according to the invoices.

2. 2

   Agree the rules

   Decide which teams may use which models, the budgets, which routes may be cached, and whether prompts are logged and for how long. Your data protection owners sign off the last point.

3. 3

   Deploy and harden

   Run the gateway in your environment or set up the hosted service, with provider keys in a secrets manager, no public admin interface, pinned versions and alerts on errors and latency.

4. 4

   Test before real traffic

   Check that budgets stop requests, that revoked keys stop working on every replica, that fallbacks trigger on the right errors and that the logs contain what you agreed. Run your evaluation set through the fallback routes too.

5. 5

   Move one team first

   Switch one application or team to the gateway endpoint, compare gateway spend with the provider's own report for the same period, and explain any gap before moving the next team.

6. 6

   Close the bypass

   Once every team is on the gateway, revoke the old provider keys held by applications and block direct network access to the providers.

7. 7

   Operate it like production

   Patch on a schedule, review spend per team every month and repeat the fallback and key revocation tests after each upgrade.

Cloudsail sets up LiteLLM and the platform around it, including the tests above, in a client's own AWS or Azure account or on the client's own servers, and our engineers in Germany and Poland can run it afterwards if you do not want to.

LiteLLM gateway setup and operationLLM platform engineeringAI in your own cloud account