Context Window

A context window is the maximum number of tokens a language model can hold in view for a single request — every system instruction, retrieved document, prior turn of conversation, and generated token all draw from one shared, finite budget.

Unlike a program that can request more memory mid-task, a context window is a hard architectural ceiling set at training and inference time. Feed the model one token past the limit and something has to give: most APIs simply reject the request, while some client libraries silently truncate the oldest messages, which is its own kind of bug. There is no "almost fits" — you are either inside the window or you are not.

How it works

Transformer models compute self-attention across every pair of tokens in the input, which is why context length was historically expensive to grow: naive attention cost grows quadratically with sequence length, so doubling the window roughly quadruples the compute for that layer. Techniques like sliding-window attention, grouped-query attention, and position-embedding schemes designed for length extrapolation are what let modern models offer windows of 128,000 tokens and beyond without the compute bill growing at the same rate.

Size is not the whole story. Research on long-context recall — often called "lost in the middle" — has repeatedly found that models are more reliable at using information near the start or end of the context than information buried in the middle, even when the window is nominally large enough to hold everything. A bigger window raises the ceiling; it does not guarantee the model will notice what you put in it.

Why it matters

Every architectural decision in an LLM application is really a context-window budgeting decision. A system prompt, retrieved chunks, tool definitions, conversation history, and the model's own output all compete for the same tokens. Run out of room and the cheapest fix — truncating old messages — is also the one most likely to silently drop the instruction or fact that mattered.

A worked example

Say your model's context window is 128,000 tokens. Your system prompt runs 800 tokens. A retrieval step pulls back three chunks at 1,200 tokens each — 3,600 tokens. The conversation has run long enough that history sits at 15,000 tokens. That is 19,400 tokens spent before the user's question or the model's answer exist at all — leaving roughly 108,600 tokens of headroom, comfortably above the 4,096-token output cap most APIs default to.

Let that same conversation run for another hour with no history management, and the math changes fast: history alone can eat 80,000 or 100,000 tokens, at which point the fixed 800-token system prompt is a rounding error competing against a wall of old messages for the model's attention — a "lost in the middle" setup even before the true ceiling is hit.

How miatz teaches it

Context-window math is a standing drill in the miatz prompting module: learners are handed a fixed token budget and a pile of candidate context — system rules, retrieved docs, history — and have to decide what stays, what gets summarized, and what gets cut, before ever touching a model. Running out of context in a War Room drill teaches the lesson faster than any diagram: the window is a budget, and budgets get spent.