Deciding whether an AI feature should run on your own infrastructure

Name the constraint that raised the question

Self-hosting comes up for different reasons, and each one has its own test:

• Data handling: a contract or policy restricts where data may be processed. Test whether any managed option, under your contract terms, meets it.
• Model choice: you need an open-weight or fine-tuned model. Test whether a managed service can serve your weights.
• Latency: the feature has a strict response-time target. Measure the 95th percentile on the managed endpoint with real prompts before assuming it fails.
• Model lifecycle: providers retire models on their own schedule, and Anthropic notes that partner platforms set their own retirement dates. Count how often you would be forced to migrate and what each migration costs you.
• Cost at volume: spend grows with traffic. Test it with the worksheet below.

Read what the managed terms already settle

Teams sometimes assume a managed endpoint keeps their data indefinitely. Read the current terms instead. OpenAI's API documentation, for example, states that API data is not used for training unless the customer opts in, that abuse monitoring logs can contain prompts and responses and are kept for up to 30 days by default unless law or harm prevention requires longer, that some features store application state, and that eligible customers can apply for Zero Data Retention or Modified Abuse Monitoring. Other providers publish different terms by product and plan. Whether any of them is acceptable is for your data protection and legal owners to decide; the engineering task is to describe the data flow precisely enough for that decision.

What you take on when you serve the model yourself

• Capacity sized for peak traffic with headroom, paid for while idle unless someone builds scaling for it
• Serving metrics someone watches: vLLM, for example, exposes Prometheus metrics such as time to first token, end-to-end request latency and KV cache usage
• Upgrades of models, serving software and drivers, each checked against your evaluation set
• Security of the hosts, the model weights and the endpoint's credentials
• On-call cover for failures outside office hours

The last item is easy to miss in a cost comparison. Google's SRE book derives a minimum of eight engineers for a single-site team to sustain a round-the-clock rotation with a primary and a secondary, under its own rule that no more than a quarter of an engineer's time goes to on-call. Your rules may differ, and an internal tool may only need support in office hours, but the calculation shows that 24/7 operation is a staffing decision as much as a hardware one.

A worksheet for measuring the workload

• Requests per day and how they spread over the day and week, including peaks
• Input and output tokens per request, measured from logs rather than estimated
• The latency target and whether requests can be batched or queued
• An evaluation set that any self-hosted model must pass
• Managed cost per month, taken from invoices
• Self-hosted cost per month: compute for peak with headroom, storage and networking, plus engineering and on-call time

A hypothetical example shows how the pieces combine. Say an internal document classifier has steady daytime traffic and uses a usage-billed managed API costing M per month. An open-weight model that passes the evaluation set needs two GPU instances for redundancy at G per month each, plus an assumed quarter of one engineer's time for upgrades and monitoring, costing E. Self-hosting is cheaper only if 2G plus E is below M. If traffic stops at night, the instances sit idle unless someone builds scale-down, while the usage-billed API charges only for tokens used. Provisioned or dedicated managed capacity is billed for capacity as well, so it enters the comparison the way GPUs do. The balance can shift when traffic is high and steady, or when a smaller fine-tuned model passes the same evaluation at a lower serving cost, so measure both.

Options between the two

The choice is rarely binary. A provider's models offered through your cloud account and region, dedicated capacity from a provider, or routing where one high-volume task goes to a small self-hosted model and the rest stays managed each move part of the operating work to you. Each also has its own terms and lifecycle, so the checks above apply per option. A practical next step is to fill in the worksheet with one month of real logs before anyone sizes hardware.

AI engineering and architectureCustom model training