Token Shredder

Token Shredder

Type any word and watch it shred into tokens — the units a language model actually reads. Tokens are not words and not letters: long words shatter into ~4-character pieces, digits and punctuation stand on their own, and a leading space fuses into the next piece exactly the way a production tokenizer encodes " word" as one token. You get a live token count, a words-to-tokens ratio, a chars-per-token readout, and a side-by-side letter view so the gap between what you read and what the model receives is visible, not asserted. Runs entirely in your browser (0 uploads, works offline).

How to use it

1. Type or paste text into the box, or load a preset — strawberry, a long word, or a sentence.
2. Read the stat row: tokens, words, characters, plus tokens / word and chars / token.
3. Scan the token strip (what the model reads) against the letter view (what you read) — the two counts rarely match.
4. Toggle Show token IDs to see each token as the opaque number the model actually operates on.
5. Open the X-ray panel to see, on your own word, why the model can't reliably spell or count letters.

What this clears up (the fundamentals)

• A token is not a word or a letter. "strawberry" is one word and ten letters, but three tokens. The unit the model reasons over is its own — and this tool shows you exactly where the cuts fall.
• Long and rare words cost more tokens. Short common words are usually one token; long compounds shatter into pieces, so tokens / word climbs and your context (and bill) fills faster.
• The leading space belongs to the token. Real tokenizers encode " the" as a single token, so spacing changes the split. This is why prompt formatting quietly shifts token counts.
• This is why AI "can't spell." The model never sees your letters — only opaque token IDs. Counting the R's in a word it received as chunks like stra · wber · ry is guessing at a spelling it was never shown.

Where it's used

A top-of-funnel AI-literacy check and the fastest way to feel why a famous failure mode (miscounting letters) happens — a mechanism, not a meme. It's a miatz build-lab concept playable: play it here, then learn to build the heuristic tokenizer and the "why AI can't spell" reveal yourself in AIE-101's opening tokens-and-vocabulary lesson.

FAQ

What exactly is a token?

The atomic unit a language model reads and predicts. Modern models use subword tokens: common words are a single token, while rarer or longer words are split into pieces. The model never sees raw characters — it sees a sequence of token IDs.

Why is the token count different from my word and character counts?

Because tokens sit between words and characters. Whitespace-word count and character count are shown alongside so you can watch a single word become several tokens, and several characters collapse into one.

Why does this prove AI can't spell?

If the model receives "strawberry" as the tokens stra · wber · ry, the individual letters are already gone. Asking it to count R's forces it to reconstruct a spelling it was never given — which is exactly where models slip. The tool shows this on your word, not a screenshot of someone else's.

Is this a real GPT tokenizer?

No — it's a documented heuristic approximation so it can run instantly, fully offline, with no vocabulary download. The cut points won't match GPT-4, Llama, or Claude exactly, but the lesson (tokens != words != letters, long words fragment, spaces fuse) is the real behavior.

Is anything uploaded?

No. Your text is shredded entirely in your browser — nothing is transmitted, stored, or logged. Turn off your Wi-Fi and it still works.

Limits

This is a teaching heuristic, not a real BPE vocabulary. It splits long ASCII words into fixed ~4-character pieces rather than using a learned merge table, so exact token counts and IDs are illustrative — a real tokenizer would split differently and its IDs are model-specific. Non-Latin scripts (Hindi, Tamil, emoji) are treated as punctuation here; real tokenizers fragment them heavily, which is its own well-known "non-English token tax." Use it to build intuition, not to price a production prompt to the token.

Related

Part of the miatz playgrounds. Explore the rest from the playground catalog.

Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.