Measuring what prompt caching actually saves

Why the headline number overstates your saving

When Anthropic launched prompt caching in 2024, it described cost reductions of up to 90 percent for long prompts, with cache reads at 10 percent of the base input price and cache writes at 25 percent more. Its current documentation lists different read rates for some newer models, and OpenAI publishes its own rates. Either way, the discount applies to cached input tokens only.

A hypothetical bill shows the effect. Input is 60 percent of spend and output 40 percent. Of the input tokens, 70 percent are read from cache at 0.1 times the base price, 10 percent are written at 1.25 times, and 20 percent stay uncached. Input now costs 0.395 of what it did, a 60 percent cut. The whole bill falls by about 36 percent, well short of 90.

Seven common reasons requests miss the cache

The details below come from Anthropic's and OpenAI's caching documentation at the time of writing. Check the current pages for your models, because minimums and lifetimes differ by model.

• The prefix is below the minimum cacheable length. Anthropic processes such requests without caching and returns no error, so both cache fields read zero.
• Something changes before the breakpoint: a timestamp, a user name, reordered or edited tool definitions. Anthropic's cache follows the order tools, system, messages, and a change at one level invalidates everything after it.
• The breakpoint sits on content that differs every request, so each request pays for a write and none ever reads it.
• Entries expire between requests. Anthropic's default lifetime is five minutes, refreshed on each use. OpenAI's default for its newest models is 30 minutes. Longer lifetimes cost more to write.
• Parallel requests start together. Anthropic notes that an entry becomes available only after the first response begins, so a burst of identical requests all write.
• Traffic is split where caches are not shared. Anthropic isolates caches per workspace on its own API. OpenAI routes by prefix and, for older models, recommends a prompt_cache_key and warns that busy keys above about 15 requests per minute can overflow to other machines.
• A conversation grows past the lookback window. Anthropic checks at most 20 blocks back from a breakpoint, so long turns can skip past the last cache entry unless you add a second breakpoint.

Measure from usage fields, request by request

Each response reports what happened. Anthropic returns cache_read_input_tokens, cache_creation_input_tokens and input_tokens, where input_tokens counts only what follows the last breakpoint, so total input is the sum of all three. OpenAI reports cached_tokens, which on models before GPT-5.6 is rounded down to a multiple of 128 while newer models report the exact matched boundary, and cache_write_tokens on its newer models. Log these fields with the feature, model and tenant for every request, then compute three numbers per feature:

1. 1

   Read share

   Cached input tokens divided by all input tokens. This is the hit rate that matters, because it is weighted by tokens rather than requests.

2. 2

   Write share

   Cache write tokens divided by all input tokens. A high write share with a low read share means you are paying a premium for entries nobody reuses.

3. 3

   Net saving

   What all input tokens would have cost at the base rate, minus what they cost at the read, write and base rates actually applied, divided by total spend including output.

Reconcile with the provider's report

Your own logs can miss requests, and your price table can be out of date. Anthropic's Usage and Cost API breaks token usage down into uncached input, cached input, cache creation and output, groupable by model, workspace and API key, and its cost report gives amounts in US dollars. OpenAI offers a prompt caching dashboard for read hit rates. Compare monthly totals from your logs with these reports. A gap points to unlogged traffic, a wrong price or a feature tagged under the wrong name.

On your own vLLM servers there is no per-token discount. Automatic prefix caching skips recomputing a shared prefix, which shortens prefill and frees GPU time, and vLLM counts prefix cache queries and hits. The saving becomes money only when that freed capacity lets you run fewer GPUs.

Fixes that usually raise the read share

• Put stable content first: tool definitions, instructions and reference material, with anything per user or per request after the breakpoint
• Freeze tool order and serialisation so the same tools produce byte-identical prefixes
• Send one warm-up request before a parallel burst that shares a prefix
• Choose a longer lifetime only where the request interval exceeds the default and the write premium is recovered