Glitch Tokens

Glitch tokens are anomalous entries in a language model’s vocabulary that produce erratic behavior — nonsensical outputs, hallucinations, or refusals — because the model never learned meaningful patterns for them during training.

What It Is

When you train or select a custom tokenizer, you’re building the vocabulary your model uses to read and write text. Glitch tokens are what happen when that vocabulary includes entries the model’s training data rarely or never contained. The tokenizer learned these tokens from one dataset, but the model trained on a different one — leaving gaps. When one of these orphaned tokens reaches the model during inference — the moment it generates a response — the result is unpredictable: gibberish, repeated phrases, hallucinated content, or outright refusal to respond.

Think of it like a phrasebook for a language you’ve never heard spoken. The words are listed, the spelling looks right, but you have no idea how to use them in a sentence. When someone asks you to say one of those phrases in context, you freeze or improvise badly. That’s what a language model does when it encounters a glitch token — it has a slot in its vocabulary but no learned understanding of what belongs there.

The phenomenon gained attention in January 2023 when researchers Jessica Rumbelow and Matthew Watkins discovered that feeding the token “SolidGoldMagikarp” — a Reddit username — into GPT-3.5 triggered bizarre outputs. The model would repeat the word, hallucinate stories, or refuse to acknowledge what it had been told. According to Li et al. 2024, the root cause is a token present in the vocabulary but absent or extremely rare in the model’s training corpus (its training dataset), producing an undertrained embedding — the internal numeric representation for that token — that destabilizes output.

Detection is harder than it sounds. You can’t scan a vocabulary list and flag suspicious entries by eye. According to Li et al. 2024, their GlitchHunter method uses iterative clustering in embedding space — grouping tokens by how similar their internal representations look — to spot tokens that cluster abnormally, a signal they never received enough training to develop stable meanings.

Newer tokenizers have reduced the problem. According to LessWrong, GPT-4’s tokenizer splits “SolidGoldMagikarp” into five normal subtokens, eliminating the glitch behavior entirely. But the underlying risk persists whenever a tokenizer vocabulary and model training data diverge — a direct concern for anyone building a custom tokenizer with tools like tiktoken, SentencePiece, or HF Tokenizers.

How It’s Used in Practice

Most practitioners encounter glitch tokens not by looking for them, but by bumping into unexplained model failures. A prompt that should produce a clean summary instead outputs repetitions or nonsense. The first instinct is to blame the model or the prompt, but the actual culprit can be a single token in the input that the model cannot process coherently.

For teams building custom tokenizers, awareness of glitch tokens changes how you validate your vocabulary. After training a tokenizer on your domain-specific corpus, you need to cross-check: does every token in the resulting vocabulary appear frequently enough in the data your model will actually train on? If your tokenizer trains on a web scrape but your model fine-tunes on curated documents, the gap between those datasets is exactly where glitch tokens hide.

Pro Tip: After training a custom tokenizer, run your model’s most common prompts through it and inspect the token IDs. If any token ID appears in the output but has fewer than a handful of occurrences in your training corpus, flag it. Removing or merging rare tokens before model training is far cheaper than debugging mysterious inference failures after deployment.

When to Use / When Not

Common Misconception

Myth: Glitch tokens are bugs in the model’s neural network that disappear with more training. Reality: They’re a vocabulary-level problem, not a weight-level problem. The model’s weights are fine — the issue is that certain tokens in the tokenizer’s vocabulary never had enough training examples to develop stable representations. The fix is updating the tokenizer’s vocabulary or retraining it on a corpus that matches the model’s training data, not running more training epochs.

One Sentence to Remember

If your tokenizer’s vocabulary includes tokens your model never learned from real data, those tokens become landmines — unpredictable in output and invisible until someone triggers them.

FAQ

Q: How did the “SolidGoldMagikarp” glitch token get discovered? A: In January 2023, researchers Rumbelow and Watkins found that this Reddit username, present in GPT-3.5’s tokenizer but nearly absent from its training data, caused nonsensical and erratic model outputs when prompted.

Q: Can glitch tokens appear in any language model? A: Yes. Any model whose tokenizer was trained on a different corpus than the model itself can have glitch tokens. The problem is not specific to one vendor or architecture.

Q: How do you detect glitch tokens in a custom tokenizer? A: Cross-reference your tokenizer’s vocabulary against your training corpus. Tokens with very low or zero frequency in training data are candidates. Automated methods like embedding-space clustering can also flag them.

Sources

• Li et al. 2024: Glitch Tokens in LLMs: Categorization Taxonomy and Effective Detection - Academic study categorizing glitch tokens across multiple LLMs and proposing detection methods
• Alignment Forum: SolidGoldMagikarp (plus, prompt generation) - Original discovery and analysis of glitch tokens in GPT-3.5