Your own language models on vLLM and Kubernetes, with data in house and costs you can plan

Serving your own models makes two things possible that a public API cannot: data that may not leave your infrastructure can be used with AI, and a task that runs all day costs what the GPUs cost, however many requests it makes. You also keep control of the model version, so answers do not change because a provider updated its model.

Many companies are preparing for this. In a 2026 survey of 106 IT leaders at small and mid-sized companies in Germany, Austria and Switzerland, run by CIO and Computerwoche with Dell, 20% had started building local server infrastructure for AI. GPUs are a large purchase, so we measure on your workload before anything is bought.

Serving your own models is worth it when data may not leave your infrastructure, when one task runs at a steady high volume, or when you need full control over the model version. It is usually the wrong choice for a small group of users, for very uneven usage, or for tasks where only the largest commercial models are good enough.

What drives the cost is the GPU class and count, how busy the GPUs are, the model size and the effort to run the service. We compare these drivers with what the same work would cost through an API before you decide.

"How many GPUs do we need for 50 users?" has no answer without the details: how many requests arrive at the same time, how long the documents and answers are, which model size and quantisation pass your tests, and which answer times your users accept. With many concurrent requests, the GPU memory for the KV cache usually runs out before compute does.

So we measure first, on rented GPUs or hardware you already have, with your own prompts and documents. You buy servers once you have seen the numbers.

vLLM runs well on Kubernetes once the GPU-specific parts are handled.

• GPU node pools with the drivers and device plugin your cluster needs, on premise or on EKS or AKS.
• Model files cached close to the nodes, so a new replica does not download gigabytes before it can answer.
• Autoscaling on vLLM's own metrics, such as waiting requests and cache use, because CPU load stays low while the GPU does the work.
• Readiness checks that wait until the model is loaded, and rollouts that keep enough capacity while a new version starts.
• Dashboards and alerts for answer times, queue length, errors and GPU use, in the monitoring you already run.

vLLM is the serving engine. KServe adds a standard way to describe and roll out model services on Kubernetes, and llm-d is a newer project for spreading inference with vLLM across many GPUs. In our judgement, most companies start well with plain vLLM deployments behind a gateway, and add these layers when the number of models or GPUs makes them worth it.

A smaller or quantised model can be cheaper and faster but worse at your task. Before a model goes live, and before every upgrade, we run your regression tests against the model it would replace and only switch when quality holds.