Token

A token is the smallest unit of text a large language model processes — typically a word, part of a word, or a punctuation mark, averaging about 3-4 characters of English.

Language models never see letters or words the way you do. Before your prompt reaches the model, a tokenizer splits it into tokens and maps each one to an integer ID from a fixed vocabulary — usually 50,000 to 200,000 entries. The model's entire world is sequences of these IDs. Every price you pay, every context limit you hit, and a surprising number of model failures trace back to tokenization.

How it works

Modern tokenizers use algorithms like Byte Pair Encoding (BPE). BPE starts from raw bytes and repeatedly merges the most frequent adjacent pairs found in training data, so common strings become single tokens while rare strings shatter into many. The word "the" is one token. A common word like "understand" is often one token too. But "unfathomable" might split into "un", "fath", "omable" — three tokens for one word.

This is why models historically struggled to count the letters in "strawberry": the model never sees letters, only token IDs. It is also why non-English text, code with unusual identifiers, and long numbers cost more — they fragment into more tokens per character.

Why it matters

Tokens are the billing and capacity unit of the entire LLM economy. API pricing is per token, in and out. Context windows are measured in tokens: a 200,000-token window is roughly 150,000 English words, but far less if you are feeding it minified JSON or Chinese prose. Latency scales with output tokens, since models generate one token at a time.

If you build with LLMs and cannot estimate token counts, you cannot estimate cost, latency, or whether your retrieval results will fit in the window. Token literacy is the arithmetic of AI engineering.

A worked example

Take the sentence: "Miatz trains T-shaped engineers." A typical BPE tokenizer produces something like:

"Mi" + "atz" + " trains" + " T" + "-shaped" + " engineers" + "."

Seven tokens for a five-word sentence. "Miatz" splits in two because it is a rare string; " trains" is one token including its leading space; the hyphenated compound splits at the hyphen. Paste the same sentence into a different tokenizer and you will get a different split — vocabularies differ per model family, which is why token counts from one provider do not transfer to another.

How miatz teaches it

Tokenization is the first stop in the miatz AI Labs: a free interactive tokenizer where you paste any text and watch it split into colored tokens in real time, compare English against other languages, and see why "strawberry" trips models up. Learners run tokenizer drills before they ever call an API, because you cannot reason about cost or context until you can see what the model sees.