Every prompt you have ever written was shredded into tokens before the model read it. Paste anything below and watch it happen.
Tokens are the units a language model actually reads and writes — not letters, not words, but pieces from a fixed vocabulary learned by byte-pair encoding (BPE). 'Deploying' might be one token; an emoji might be three; a Hindi word might be six. Every limit you hear about models — context windows, pricing, rate limits, max output — is denominated in tokens, and this lab makes them visible.
Token blindness is the most common source of surprise in LLM engineering: prompts that silently truncate, features that cost 10× the estimate, latency that scales with output length nobody measured. An engineer who can eyeball text and estimate its token weight designs better prompts, budgets context windows properly, and never gets ambushed by the invoice.
Models don't read your words — they read tokens, and everything you pay for, fit in a window, or wait on is counted in them.
loading tokenizer…
API pricing is per million tokens — input (your prompt) and output (the reply) are billed at different rates.
How much of a model's context window would this text alone consume?
Every message in a chat — plus system prompts, retrieved documents and the model's own replies — shares this one budget. That's why a long conversation with a few big pasted documents can suddenly hit the limit: the window fills up token by token, not message by message.
Tokens are the model's alphabet: byte-pair encoding (BPE) merges frequent byte sequences into single vocabulary entries, so common English words often cost just one token. Indic scripts like Devanagari and Telugu — and emoji — appear far less often in the training data behind the vocabulary, so they fragment into many byte-level tokens, inflating both cost and context usage for the same meaning. This lab uses OpenAI's cl100k_base vocabulary as a representative tokenizer; other models' tokenizers differ slightly but show the same pattern.
A token is the unit a language model actually reads — a frequent chunk of characters (often a word piece) mapped to a number. Models never see letters or words, only token IDs.
Because the model sees token IDs, not characters. If 'strawberry' arrives as two tokens, the model never looks at individual letters unless it reasons its way there.
Model pricing and context limits are counted in tokens. The same paragraph can be 30% more tokens in one language or format than another — which is real money at scale.
Deeper dive: token, defined properly
The Chunking Lab shows how RAG systems slice documents — the step that decides what an AI can and cannot find.