Subword Tokenization

Subword tokenization is the family of algorithms — byte pair encoding (BPE), WordPiece, and Unigram/SentencePiece are the three in common use — that builds a vocabulary of frequent character sequences so any string, however rare or misspelled, can be represented as a sequence of known pieces.

Before subword methods, tokenizers worked at the word level, and word-level vocabularies have a fatal flaw: language is open-ended, so no fixed word list ever covers everything, and a model hits an unknown-word token the moment it sees a name, a typo, or a word coined after training ended. Subword tokenization sidesteps the problem by working below the word: build the vocabulary from pieces small enough that everything decomposes into something already seen.

How it works

BPE, the most common variant, learns its vocabulary directly from data. Start with every individual character as a token. Count every adjacent pair of symbols across the training corpus and merge the single most frequent pair into one new symbol. Repeat, thousands of times: each round the vocabulary gains one entry and the corpus gets a little more compressed. Frequent strings collapse into single tokens early; rare strings stay fragmented for the algorithm's entire run.

The classic illustration, from the original BPE paper, trains on four words with their frequencies: "low" (5), "lower" (2), "newest" (6), "widest" (3), each spelled out as characters. Run the merge loop and the earliest merges are exactly what frequency predicts:

merge 1: e + s -> es (9 occurrences, in newest + widest)

merge 2: es + t -> est (still 9)

merge 3: est + </w> -> est</w>

merge 4: l + o -> lo (7 occurrences, in low + lower)

merge 5: lo + w -> low (7 occurrences)

merge 6: n + e -> ne (6 occurrences, in newest)

After six merges, "low" is already a single token while "widest" is still split across three pieces, because "low" and its relatives appeared more often in this tiny corpus. WordPiece runs a similar merge loop but picks pairs by likelihood gain rather than raw frequency; Unigram works backward, starting from a large candidate vocabulary and pruning down to the pieces that best explain the data under a probabilistic model.

Why it matters

Subword tokenization is why modern models have no true "unknown word" token: worst case, an unfamiliar string decomposes into individual characters, which are always in vocabulary. It is also why vocabulary size is a real design trade-off — a bigger vocabulary means fewer tokens per sentence, cheaper and faster, but a bigger embedding table and softmax to compute; a smaller vocabulary is the reverse. And it explains why every model family tokenizes differently: the merge rules are learned from that model's specific training corpus, so a token count from one model's tokenizer never transfers to another's.

A worked example

The merge table above is the worked example: it is the toy run from the original 2016 BPE paper, reproduced by running the algorithm on that four-word corpus. Notice the mechanism is entirely frequency-driven and has no notion of "words" at all — it would happily merge across a syllable boundary if that pair showed up often enough. That is also why subword splits sometimes look linguistically odd to a human reader even when they are statistically optimal for the model.

How miatz teaches it

The tokenizer lab in the miatz AI Labs playground includes a from-scratch BPE trainer: learners feed it a small corpus, step through merges one at a time exactly like the table above, and watch a vocabulary emerge symbol by symbol, before ever loading a production tokenizer that hides the process behind a single function call.