Finding where an AI response spends its time

Split one slow request into spans

Steps that run one after another add up. Steps that overlap do not, and only the longest chain through them, the critical path, sets the elapsed time. Instrument the request path so that each step is its own span with a start and end time, then study the slow requests rather than the typical ones. A useful split:

• Waiting before work starts: gateway queue, rate-limit backoff and retries
• Retrieval: embedding the query, vector or keyword search, reranking
• Each model call, split into time to first token and generation time
• Tool and API calls the model triggers, and the round trips between them
• Post-processing: parsing, validation, moderation and any repair call

Give retries their own counter. LiteLLM's proxy, for example, retries a failed call on the same model group before falling back, three times by default according to its documentation, so an unstable provider can look like slowness rather than errors.

A hypothetical latency budget

Suppose a support assistant's p95 is 6.1 seconds and the product owner wants 3. In this hypothetical trace every step runs in sequence, so the spans add up to the total:

• Gateway wait and one retry: 0.4 s
• Query rewrite by a large model: 0.9 s
• Retrieval and reranking: 0.7 s
• Answer model, time to first token with a 9,000-token prompt: 1.0 s
• Answer model, generating 520 tokens: 2.6 s
• Order-status tool call, made after retrieval and before the answer: 0.5 s

Generation is the largest term, so shorter answers come first: the replies run to three paragraphs where one would do. The query rewrite is a well-defined task that a smaller model may handle, and the order lookup does not depend on the answer, so it can run alongside retrieval. Measure each change separately, because each can also change answer quality. Once the lookup runs in parallel, it leaves the critical path, and shortening it further gains nothing.

What usually moves the number

OpenAI's latency guide is direct about the order. Generating tokens is almost always the slowest step, and cutting output by half may cut latency by roughly half. Halving the prompt may gain only 1 to 5 percent unless the context is very large. Every sequential request adds a round trip, so combine steps that can share a call and run independent steps in parallel.

Two further levers change what users perceive more than what the clock shows. Streaming shows the first words while the rest is generated, and visible progress through tool steps tells the user that work is happening. Neither reduces end-to-end time, so keep measuring both numbers.

Prefix caching shortens the time to first token for requests that share a long, stable prefix such as instructions and tool definitions. The vLLM documentation notes that it speeds up the prefill phase only and does nothing for generation.

Extra signals when you serve the model yourself

With your own inference server, the model span splits further. vLLM exposes Prometheus metrics for each request's queue time, prefill time, decode time, time to first token and time per output token, next to server-level gauges such as the share of KV cache in use. Its documentation frames the relationship well: server-level metrics explain the request-level ones.

One pattern to look for is rising queue time while the KV cache is full. When cache space runs out, vLLM preempts requests and recomputes them later, which its documentation warns can hurt end-to-end latency. The remedies are more memory per replica, fewer concurrent sequences or more replicas, and each has a cost. Chunked prefill settings trade one latency against another: a smaller token budget per batch favours steady streaming, a larger one favours time to first token.

Measure under load and at the tail

Latency measured by one engineer on a quiet afternoon says little about Monday morning. Replay a sample of real prompts at realistic arrival rates. vLLM's benchmark command can send requests at a set rate with adjustable burstiness, cap concurrency, report chosen percentiles of first-token, per-token and end-to-end latency, and count goodput: the requests that met the service-level targets you specify.

Set the target per feature as a percentile, such as p95 time to first token and p95 end-to-end time, and check it after every release. A change that improves the median but worsens the tail is a regression for the users who notice delays.

When no single span is the problem

Sometimes every span looks reasonable and the sum is still too slow, for example because the design chains five model calls where two would do. OpenAI's guide suggests asking whether each step needs a language model at all: confirmations can be hard-coded, constrained choices precomputed, and ordinary caching or search can beat another call. That redesign takes more effort than changing a setting, and it is often where the larger gain lies.