LLM platform engineering: one gateway for every model, run on Kubernetes

Without a gateway, keys sit in many applications, spend shows up as one invoice per provider, and switching a model means changing code in every team. A gateway gives every application one API, whichever model answers, and gives you one place to decide who may use which model.

• A key per team or application, revoked in one place.
• Budgets and rate limits before the invoice arrives.
• Cost per team and per application, not per provider.
• Fallback to another model when a provider is down.
• Rules for which application may use which model.

LiteLLM is an open-source proxy that speaks the OpenAI API format and forwards requests to commercial providers and to models you host yourself. It handles virtual keys, budgets, spend tracking, routing and fallbacks, and it runs in your own environment with its own database. We deploy it with infrastructure as code, connect it to your identity provider and logging, and write the routing rules with your teams.

Some features, such as single sign-on for more than five users, are part of LiteLLM's paid enterprise edition. We check which ones you actually need before you buy a licence. No vendor pays us, so if your cloud provider's own gateway or another product fits better, we use that instead.

Where data may not leave your infrastructure, or where one task runs often enough that own hardware is cheaper, we serve open-weight models with vLLM on GPU nodes in Kubernetes. Before go-live we measure answer times and throughput under your expected load, so the hardware is sized from measurements and not from a data sheet.

The platform is run like the rest of your infrastructure: in version control, deployed through pipelines and watched by the monitoring you already use.

• Infrastructure as code with Terraform and Helm, in your repositories.
• CI/CD for gateway configuration, routing rules and prompts.
• New model versions rolled out in stages, with regression tests from your own cases before each step.
• Monitoring of answer times, errors and cost per team, with alerts.
• Runbooks your team can follow without us.

We build on the Kubernetes you already run, on premise or as a managed service on AWS, Azure or the cloud you use. Commercial models can be reached through your cloud provider in an EU region, so requests stay under the contracts you already have. One gateway can sit in front of both, with rules for which application may use which model.

With every request passing the gateway, cost can be reported per team, application and task, and charged back if you want. Routing to a cheaper model is only switched on after the regression tests show that quality holds for that task, so savings do not come from answers getting worse.