Cost Per Token

Cost per token is the unit price a model API charges for text processed, quoted separately for input tokens (what you send) and output tokens (what the model generates) — and the two rates are rarely equal, with output usually priced several times higher than input.

Every LLM API bill, no matter how it is summarized on an invoice, reduces to this one arithmetic fact: tokens in times the input rate, plus tokens out times the output rate. Everything else — a "per request" mental model, a flat monthly estimate — is an approximation of that underlying multiplication, and a surprisingly leaky one once request sizes vary.

How it works

Providers typically quote price per million tokens, separately for input and output — something like $3 per million input tokens and $15 per million output tokens is a realistic order of magnitude for a mid-tier model. The asymmetry is not arbitrary pricing: processing input tokens the model has already been given is comparatively cheap, parallelizable work, while generating output tokens happens one at a time, sequentially, each new token requiring its own full forward pass conditioned on everything before it — output is simply more compute per token to produce.

Some providers add a third rate, usually a steep discount, for cached input tokens — content repeated identically across requests, like a long system prompt or a retrieved document reused across a conversation, that the provider can avoid reprocessing from scratch. That cache discount is often the single highest-leverage cost lever available to an application with a large, stable system prompt or context, since it turns a recurring full-price cost into a much smaller ongoing one.

Why it matters

Cost-per-token arithmetic is what turns a demo into a viable product, or doesn't. A feature that looks free in a five-message test conversation can become the largest line item on the infrastructure bill once it runs across ten thousand daily users, and the asymmetric input/output pricing means the fix is rarely "use a cheaper model" alone — capping output length, trimming unnecessary context, and caching repeated content usually move the number more than a model downgrade does.

A worked example

At $3 per million input tokens and $15 per million output tokens, a single request using 5,000 input tokens and 800 output tokens costs 5,000 divided by 1,000,000 times $3, or $0.015, for input, plus 800 divided by 1,000,000 times $15, or $0.012, for output — $0.027 total, under three cents.

That looks negligible until it runs at scale: 10,000 such requests a day costs $270 a day, or roughly $8,100 a month — and note that the output tokens, despite being under a sixth of the token count of the input in this example, contribute nearly as much dollar cost, because the output rate is five times higher per token. A team that only tracks total token count, without separating input from output, will consistently underestimate its bill.

How miatz teaches it

Cost math is a required line item in every capstone project's technical writeup: learners must show input and output token counts per request, the blended cost at scale, and at least one lever, caching, context trimming, output limits, they used to bring it down, because in the AI era, an architecture that cannot state its own unit economics is not considered finished.