Tokenization

Tokenization is the process of breaking text into smaller units called tokens — subwords, characters, or bytes — that transformer models convert into numerical representations for processing language.

What It Is

Every language model faces the same fundamental problem: computers work with numbers, not words. Tokenization is the bridge between human language and the math that powers transformers. Without it, models like Claude or GPT would have no way to read your prompts or generate responses.

Think of tokenization like a postal sorting system. Just as a post office breaks addresses into zip codes, streets, and house numbers to route mail efficiently, a tokenizer breaks text into standardized pieces that a model can route through its attention layers and matrix multiplications. Each piece — called a token — gets assigned a number from the model’s vocabulary, and that number maps to a learned embedding vector that captures meaning.

The dominant approach is Byte-Pair Encoding (BPE), which works by starting with individual characters and repeatedly merging the most frequent pairs into larger units. The word “understanding” might become [“under”, “stand”, “ing”] — three tokens instead of one long word. This subword strategy strikes a balance: common words stay intact as single tokens, while rare or unfamiliar words get split into recognizable pieces. According to Wikipedia, this lets BPE handle words the model has never seen before by decomposing them into known subword units.

Vocabulary size — the total number of unique tokens a model recognizes — typically ranges from 32,000 to 256,000 tokens depending on the model, as noted by Wikipedia. Larger vocabularies mean fewer tokens per sentence (faster processing, more text fits in the context window), but they also require more memory. This is a direct trade-off that model designers weigh carefully.

After tokenization, each token maps to an embedding vector. According to D2L, these learned vectors have a fixed dimension (called d_model) and encode semantic relationships — tokens with similar meanings end up closer together in vector space. These embeddings are what feed into the positional encoding and attention layers that make transformers work.

How It’s Used in Practice

If you’ve ever hit a “context length exceeded” error in ChatGPT or Claude, you’ve run into tokenization directly. The limits you see — like 128K or 200K tokens — are measured in tokens, not words. A rough rule of thumb: one token equals about three-quarters of a word in English. So a 100K-token context window fits roughly 75,000 words.

Tokenization also explains why AI tools sometimes behave oddly with code, URLs, or non-English text. A single Chinese character might consume two or three tokens, while a common English word like “the” is just one token. When developers build prompts for AI coding assistants or chatbots, understanding token counts helps them stay within limits and control costs, since API pricing is usually per token.

Pro Tip: Before sending long documents to an API, run your text through a tokenizer library like tiktoken (for OpenAI models) or HuggingFace Tokenizers to count actual tokens. This avoids surprise truncation and helps you estimate costs upfront. The token count is almost always different from word count.

When to Use / When Not

Common Misconception

Myth: Tokenization splits text into whole words, so word count and token count are roughly the same. Reality: Most modern tokenizers use subword splitting. The word “transformers” becomes multiple tokens (e.g., [“transform”, “ers”]), while common short words stay as one token. English averages about 1.3 tokens per word, but code, technical jargon, and non-Latin scripts can push that ratio much higher. Always count tokens directly rather than estimating from word count.

One Sentence to Remember

Tokenization is where human language becomes numbers — and the way text gets split into tokens directly affects how much a model can read, how fast it runs, and how much each API call costs. If you work with any language model, understanding tokens gives you practical control over context limits, pricing, and prompt design.

FAQ

Q: How many tokens does one word equal? A: In English, roughly 1.3 tokens per word on average. Code, technical terms, and non-Latin scripts often require more tokens per word, sometimes two to four times as many.

Q: Does tokenization affect model accuracy? A: Yes. Poor tokenization can split meaningful words into fragments that lose context, especially in languages with complex morphology. Better tokenizers preserve more semantic meaning per token.

Q: Can I see how a model tokenizes my text? A: Yes. Tools like OpenAI’s tiktoken library and HuggingFace Tokenizers let you inspect exact token splits. OpenAI also offers an online tokenizer tool for quick checks.

Sources

• HuggingFace Docs: HuggingFace Tokenizers Documentation - reference documentation for the widely-used Rust-based tokenizer library supporting BPE, WordPiece, and Unigram algorithms
• Wikipedia: Byte pair encoding - overview of the BPE algorithm that underpins most modern language model tokenizers