Planning GPU capacity for a model your team serves

What limits a single replica

GPU memory holds the model weights and the KV cache, which grows with the context length and number of sequences in flight. Once weights are loaded, the KV cache sets how many requests a replica can batch, and batching sets throughput. The PagedAttention paper behind vLLM showed that wasted KV cache memory had been limiting batch sizes, and that managing it in pages raised throughput at the same latency.

In vLLM, gpu_memory_utilization decides how much memory the instance takes, 0.92 by default in the current documentation, and max_model_len and max_num_seqs cap context and concurrency. When the KV cache runs out, vLLM preempts requests and recomputes them, which its documentation warns can hurt latency. Quantised weights or a smaller KV cache data type free memory for more concurrent requests, at a quality cost you have to test.

Describe the workload before you benchmark

A benchmark on synthetic prompts of fixed length answers a question nobody asked. Before measuring, write down:

• Arrivals per second by hour and weekday, with the busiest five minutes rather than the hourly average
• The distribution of input and output tokens per request, because a few long requests consume a large share of the KV cache
• Latency targets per feature, such as p95 time to first token and time per output token
• Which work is interactive and which can wait, such as overnight document processing

Benchmark one replica at your latency target

vLLM's serving benchmark can replay a custom dataset of your prompts at a chosen request rate, with adjustable burstiness and an optional concurrency cap, and report goodput: the requests that met the time-to-first-token, per-token and end-to-end targets you set. Step the rate up until goodput falls below the share you need. That rate is the replica's capacity for this workload. The offline throughput benchmark answers a different question, maximum tokens per second with no latency target, which is useful for sizing batch jobs.

While the benchmark runs, watch the server's own metrics, but treat them as clues. Rising queue time with KV cache usage near full suggests memory pressure, and slowing generation with cache to spare suggests compute, yet both can occur together and scheduler settings affect each. Before naming the limit, check preemption counts, GPU compute and memory-bandwidth utilisation and the number of running requests, then rerun with controlled changes to concurrency and context length to see which one moves the result.

From one replica to a fleet: a worked example

All figures here are hypothetical. An internal assistant peaks at 18 requests per second on weekday mornings and drops to 2 at night and on weekends. On real prompts, one single-GPU replica keeps 99 percent goodput up to 5 requests per second; at 6, queue time pushes goodput to 93 percent.

1. 1

   Peak replicas

   18 divided by 5 is 3.6, so 4 replicas carry the peak.

2. 2

   Failure headroom

   One more replica covers a failed node or a rolling deployment, making 5.

3. 3

   Quiet hours

   At 2 requests per second, 1 replica suffices, plus 1 spare.

4. 4

   Compare schedules

   Five replicas around the clock use about 3,650 GPU hours a month. Five for 12 hours on 22 weekdays and 2 for the remaining hours use about 2,250, roughly 38 percent fewer.

The schedule only works if replicas are ready before the morning ramp. Loading a model is not instant, so scale up on a timetable rather than waiting for queues to grow. vLLM documents ways to shorten repeated boots, such as reusing its compile cache. Overnight batch work can then run on the spare replica instead of a separate fleet.

Decide what absorbs traffic above plan

No plan covers every spike. Choose in advance between three options. A bounded queue keeps latency predictable and rejects excess load early. A hosted model as overflow keeps answers flowing, provided it passed the same quality tests and your data rules allow it. Deferring non-urgent work moves batch jobs out of the peak. Each choice has a cost, and agreeing it before launch prevents the reflex of buying GPUs for a spike that happens twice a year.