Tokenizer Architecture

Before a transformer processes a single word, the tokenizer has already decided how much of that word survives as usable signal — and that decision outlives every downstream choice this theme covers, from attention pattern to context-window budget. It is the layer every other topic in Transformer & Attention Internals treats as a fixed input, yet it is also where cost, coverage, and quality problems most often originate rather than surface. A tokenizer trained mostly on English text does not just under-serve other languages abstractly — it charges them more per request, in tokens billed and context consumed.

How to read tokenizer architecture: mechanism, limits, then market

Start with how BPE, WordPiece, and Unigram encode text for LLMs — it lays out the pipeline every tokenizer runs (pre-tokenization, vocabulary, merge rules, byte fallback) and gives you the vocabulary to read any tokenizer’s config file. Follow it immediately with glitch tokens, fertility gaps, and the unsolved limits of subword tokenization: it explains why some vocabulary entries never fire correctly and why the same sentence costs more tokens in some languages than others — the two failure modes that surface first in production.

Once the mechanism and its limits are clear, the guide to training and choosing a custom tokenizer with tiktoken, SentencePiece, and HF Tokenizers turns that knowledge into a build decision — when an existing vocabulary is enough and when it is not. For where the field is moving, the 262K-vocabulary race behind SuperBPE and LiteToken tracks vocabulary design becoming a compute lever rather than a fixed default. Close with the hidden bias in tokenizers — the fertility gap from the second article, followed to its economic conclusion.

How tokenizer architecture differs from the transformer and attention layers above it

Two neighbours in this theme get credited with decisions that are actually the tokenizer’s.

Tokenizer architecture is not transformer architecture. The tokenizer fixes what a token is — a one-time vocabulary decision made before training starts; the transformer decides what happens to a token once it is inside the network, layer after layer. Swapping a tokenizer means rebuilding the vocabulary and re-embedding everything; swapping between decoder-only and encoder-decoder transformer layouts is a separate decision made after the vocabulary is already fixed.

Tokenizer architecture is not the attention mechanism either. Attention weighs tokens against each other once they already exist as discrete units; it has no visibility into how a word got split in the first place. A fertility problem — one word costing five tokens instead of one — never shows up as an attention weight. It shows up earlier, as a sequence that is already too long or already missing signal before attention runs at all.

Common questions about tokenizer architecture

Q: Do I need to train a custom tokenizer, or can I reuse an existing one? A: Reuse one whenever your vocabulary was trained on languages and domains close to yours — training is expensive, and tiktoken cannot do it, only encode. Build a custom vocabulary with HF Tokenizers or SentencePiece when coverage validation shows your target languages or domain vocabulary are underrepresented.

Q: Is a bigger tokenizer vocabulary always the safer choice? A: No — a larger vocabulary buys shorter sequences and lower inference compute, which is why designs like SuperBPE and LiteToken pushed vocabularies from 32K to 262K, but every added entry also grows embedding-table size and training cost. The right size depends on model scale and language targets, not a single trend line.

Q: Can a tokenizer break production even when the model’s code is correct? A: Yes — glitch tokens are vocabulary entries that rarely or never appeared during training, so the model has no reliable behavior for them even though nothing in the inference code is wrong. The failure lives in the vocabulary, not the stack trace.

Q: Should I worry about tokenizer bias if my product only serves English users today? A: Worry about it before you expand, not after. The hidden bias in tokenizers shows the fertility gap is structural, not incidental — the moment you add a second language, that language starts paying a token tax your English users never see.

Part of Transformer & Attention Internals · closest neighbour: Transformer Architecture. New to this from a software background? Start with the story: Transformer Internals for Developers: What Maps, What Breaks.