What a company needs to run an on-premise LLM, from hardware to tested answers

When on-premise is the right choice and when a private cloud account is enough

On-premise is the right choice when a rule or a risk assessment says the data may not be processed by any outside provider, or when the systems the assistant must read are only reachable inside your network. If the requirement is that data is processed in the EU by a contracted processor and not used for training, a model deployment in your own cloud account usually meets it with much less operating work. Write the requirement down in one sentence before you choose. In our experience, most later disagreements go back to a requirement nobody wrote down.

As of 7 November 2026, both large clouds document EU options. Microsoft says Data Zone deployments in the EU process data within the Azure EU Data Boundary, and that, for models sold directly by Azure, prompts and completions are not available to model providers and are not used to train foundation models without your permission. On Amazon Bedrock, inference profiles tied to a geography such as the EU keep cross-region requests inside it, and a retention setting decides whether AWS may keep prompts and outputs for abuse prevention.

Self-hosted AI or a managed API: the full decisionEU data residency with Azure OpenAI and Amazon Bedrock

The layers of a working system, from GPU servers to agents

A working system has six layers, and data stays inside only if none of them calls out: GPU servers, a serving engine, an LLM gateway, data access and retrieval, the assistant or agent, and tests with monitoring.

Two details make the stack easier to change later. vLLM serves OpenAI-compatible APIs, including chat completions and embeddings, and LiteLLM can route to a vLLM server through its hosted_vllm provider, so applications talk to the gateway rather than to a particular model. LiteLLM's virtual keys and its budgets and rate limits per key, user or team need a Postgres database. Without one, budgets are not enforced (vLLM and LiteLLM documentation, as of 7 November 2026).

"Data never leaves the company" is a property of the network, so test it like one. List every outbound connection the stack makes: model downloads, container registries, package mirrors, telemetry and error reporting in each component, and any hosted monitoring service. Then block outbound traffic from the serving and gateway hosts by default and allow only what is on the list. For us this is a check before go-live. A sentence in the design document does not count. Retrieval needs the same care inside the company: an assistant that can read every file will answer anyone's question with anything, so results must be filtered by the permissions of the person asking before the model sees them.

What an LLM gateway doesRunning LiteLLM securely in productionDocument permissions in RAG

Which open-weight models are good enough in German, and how to test it

A model is good enough in German when it passes your acceptance tests on your own documents and questions. In our view, public leaderboards say little about German business text, so use them for a shortlist at most. Two filters come straight from the model card. The first is language support: Meta's card for Llama 3.3 70B Instruct lists eight supported languages including German, and Mistral's card for Mistral Small 3.1 lists German among its languages. The second is the licence: Llama 3.3 uses the Llama 3.3 Community License Agreement and Mistral Small 3.1 uses Apache 2.0. The conditions differ, so have both read before you invest in tests. We name these two only as examples of what cards state. Check the current cards when you start.

Then test the shortlist the same way you will later accept the system:

• Collect real requests from the people who will use the assistant, together with the documents they would need, including long and badly formatted ones
• Name the failure types that matter and how serious each one is: a wrong figure, an invented source, an answer in English, the wrong form of address, broken JSON in a tool call
• Run each candidate at the precision you will serve in production, because a quantised model can behave differently from the full one
• Have German-speaking domain experts grade the answers without knowing which model wrote them
• Run each case more than once, since the same prompt can produce different answers
• Keep the cases: the same set decides every later model change

In our experience, the differences between candidates show up in a few specific failure types, such as figures taken from tables or answers about long documents, more than in an average score. So report a pass rate per failure type instead of a single number.

Checking model quality after quantisationAcceptance criteria for AI systems

Sizing hardware without buying the wrong thing

Size the hardware from measurements of your own workload on the model you chose. A head count tells you little: the same number of users can need very different hardware depending on how long their requests are and how many arrive at the same moment. These are the drivers of memory and cost:

• Model size and precision. Weights take roughly the number of parameters times the bytes per parameter. Mistral's card, for example, says Mistral Small 3.1 with 24 billion parameters needs about 55 GB of GPU memory in bf16 or fp16, and fits a single RTX 4090 once quantised
• Context length and concurrency. The KV cache grows with the tokens each request holds and the number of requests in flight, so long documents and agents that carry their history use most of the memory
• Peak concurrency rather than total users: count the requests in flight during the busiest hour
• Redundancy: enough capacity to keep serving with one server out of service
• Other models: embedding and reranking models for retrieval need memory too
• Power, cooling, rack space and the people who maintain the machines

Measure before you order. Rent the GPU type you are considering, serve the chosen model with vLLM at the precision and context length you plan, and replay realistic requests. As of 7 November 2026, vLLM ships a benchmark command, vllm bench serve, for online serving throughput, and its /metrics endpoint reports vllm:num_requests_waiting, vllm:kv_cache_usage_perc and vllm:num_preemptions. The vLLM tuning guide explains that when the KV cache runs short, requests are preempted and recomputed later, which hurts latency. Its remedies are more memory for the cache or fewer concurrent sequences per server. Find the load one server carries at your latency target, divide your peak by it and add a spare.

Planning GPU capacity, with a worked example

What it takes to run it after go-live

After go-live you run a service, so someone must own updates, monitoring, regression tests and the document index, with time set aside for each. On-premise means all of these are yours. Nobody upstream patches the serving engine or watches the GPUs for you.

Three operating models work: an internal platform team that already runs Kubernetes, the IT operations team with a named owner for the AI service, or an external partner under a written operating agreement. In our experience, the systems that fail are the ones nobody was assigned to once the project team moved on.

Regression testing for LLM applicationsWho owns an AI system after the pilot

Data protection, works council and the EU AI Act

Running the model yourself means there is no outside processor for the model. The data protection officer, the works council and the AI Act still ask what the system does with people's data. Below is a checklist for those conversations. It is not legal advice, so check the current text at the official sources linked below.

Data protection

Article 35 GDPR requires a data protection impact assessment before processing, in particular processing that uses new technologies, that is likely to result in a high risk to people's rights and freedoms. Expect the data protection officer to ask which personal data goes into prompts, indexed documents and logs, how long logs are kept and who can read them, and how a deletion request reaches both the index and the logs.

Works council

Under section 87(1) no. 6 of the German Works Constitution Act (BetrVG), the works council co-determines the introduction of technical devices intended to monitor employees' behaviour or performance. In its decision of 16 July 2024 (1 ABR 16/23), the Federal Labour Court repeated that a device counts as intended for monitoring when it is objectively suitable for it, whatever the employer's intent. In our reading, a gateway that logs prompts per user is suitable in that sense. Plan a works agreement early and design the logs for it: pseudonymised user IDs, short retention, named people with access, and a written rule that logs are not used to assess performance.

EU AI Act

As of 7 November 2026, the AI Act, Regulation (EU) 2024/1689, has been amended by the Digital Omnibus on AI, Regulation (EU) 2026/1744 of 8 July 2026. According to the Council's timeline, obligations for stand-alone high-risk systems now apply from 2 December 2027 and those for high-risk systems embedded in products from 2 August 2028. The amended Article 4 requires providers and deployers to take measures to support AI literacy, without requiring them to guarantee a specific level for each person. In our reading, an internal assistant for documents is usually not a high-risk system, but if it will feed decisions about employees or applicants, ask counsel to check it against the high-risk list in Annex III before you build it.

From first use case to production, step by step

Take one use case to production in a fixed order. Each step produces what the next one needs: a written requirement, a test set, a measured model, a sized platform and a tested system.

1. 1

   Pick one use case and write the data requirement

   Choose a task with clear users and documents, and write in one sentence where its data may be processed. That sentence decides between on-premise and a private cloud account.

2. 2

   Build the test set first

   Collect real requests, the expected behaviour and the failure types with their severity and required pass rate, agreed with the people who will sign off.

3. 3

   Shortlist and test models

   Filter by licence and German support, then run the candidates at the precision you would serve, on rented GPUs, against the test set.

4. 4

   Measure capacity and size the hardware

   Benchmark the chosen model with realistic traffic at your latency target, then size for peak load plus one spare server.

5. 5

   Build the platform with outbound traffic closed

   Set up the serving engine, the gateway, retrieval with permission filters and monitoring, then block outbound traffic and allow only what you have listed.

6. 6

   Clear data protection and the works council

   Complete the impact assessment, agree the log design and the works agreement, and plan AI literacy measures before real users arrive.

7. 7

   Run the acceptance tests, then a limited rollout

   Release only when every criterion passes. Start with one group of users and watch the metrics and the failures they report.

8. 8

   Hand over operations

   Name the owner, write the runbooks and fix the routine for updates, regression tests and capacity reviews.

We build and run these systems on a client's own servers or in the client's own cloud account, from the GPU serving layer to the acceptance tests before go-live.

On-premise AIvLLM on KubernetesLLM and agent platform