Prompt Caching

Prompt caching is a technique where the static, repeated portion of a prompt — a long system prompt, a list of tool definitions, a reference document — is processed once and stored, so subsequent requests sharing that exact prefix pay a small fraction of the normal input-token cost and respond faster, instead of reprocessing the same content from scratch on every call.

Any application that resends a large shared context on every request — a big system prompt, a fixed set of tool schemas, a support bot's knowledge-base excerpt, an agent's steadily growing conversation history — is paying full price to reprocess content that hasn't changed since the last call. Prompt caching turns that recurring cost into something closer to a one-time cost.

How it works

The provider computes a hash of the prompt content up to a marked cutoff point. The first request containing that prefix pays a write premium — slightly more than a normal pass, since storing the state costs something — and the cached state is kept for a short time-to-live window. Every following request that presents the identical prefix within that window reads from the cache at a steep discount instead of paying full price for those tokens. Because this is a strict prefix match, changing even one token earlier than the cutoff invalidates the cache for everything after it — a timestamp or a randomly ordered field inserted into the middle of an otherwise-stable prompt silently defeats the whole mechanism.

Why it matters

For any workload that repeats a large shared context across many calls — a customer-support bot processing a burst of tickets against the same system prompt, an agent loop resending its growing history every turn, a batch job running the same instructions against many inputs — caching is often the difference between an approach being affordable at scale and not. It requires no change to what the requests actually ask; the savings come purely from not re-paying for content that was already processed moments ago.

A worked example

A 10,000-token shared prefix — a system prompt plus a set of tool definitions — gets reused across 100 requests inside a five-minute burst, each carrying its own 200-token question, at $5 per million input tokens. Paying full price on every request costs $5.10 total. With caching, the first request pays a 1.25x write premium on the 10,000-token prefix, and each of the remaining 99 requests reads that same prefix at roughly a tenth of the normal price — for a total of $0.66, an 87% reduction, without changing a single word of what any request actually asks.

How miatz teaches it

Learners compute the cache-versus-no-cache cost comparison above themselves, for a workload of their own design, before adding caching to any real request — so the 87% figure in the worked example is a number they derived, not one they were handed. The agents module then has them deliberately break their own cache — inserting a timestamp into the middle of a stable prompt — and watch the savings disappear, so the prefix-match invariant is a lesson learned by breaking it, not just reading about it.