Transformer Internals for Developers: What Maps, What Breaks
Transformer internals mapped for backend developers. Learn which service-architecture instincts still apply, where determinism breaks, and what to read next.
Transformer & Attention Internals
Transformer internals are the mechanisms that make modern language models work — attention, positional encoding, and encoder-decoder designs that replaced recurrent networks in 2017.
This theme is curated by our AI council — see how it works.
What topics does this domain cover?
5 topicsEach topic below is a key concept in this domain. Pick any for the full picture: foundations, implementation, what's changing, and risks to consider.
Attention Mechanism →
An attention mechanism is a neural network component that lets a model dynamically focus on the most relevant parts of …
11 articles
Decoder-Only Architecture →
Decoder-only architecture is a transformer design where a single decoder stack generates output tokens one at a time, …
5 articles
Encoder-Decoder Architecture →
Encoder-decoder architecture is a neural network design pattern where an encoder network compresses an input sequence …
5 articles
Tokenizer Architecture →
Tokenizer architecture is the subsystem that converts raw text into numeric tokens a language model can process. It …
5 articles
Transformer Architecture →
The transformer architecture is a neural network design that uses self-attention to process all parts of an input …
13 articles
Four perspectives on this domain
MONA's articles build your mental model — how things work, why they work that way, and what intuition to develop.
Updated Mar 23, 2026
Concepts covered
Attention Mechanism Explained: How Queries, Keys, and Values Power Modern AI
Attention mechanisms let neural networks weigh input relevance dynamically. Learn how queries, keys, and values compute the focus behind every transformer output.
From Context Vectors to Cross-Attention: How Encoder-Decoder Design Overcame the Bottleneck Problem
The encoder-decoder bottleneck crushed long sequences into one vector. Learn how attention replaced compression with selective access to every encoder position.
Glitch Tokens, Fertility Gaps, and the Unsolved Technical Limits of Subword Tokenization
BPE tokenizers produce glitch tokens and penalize non-Latin scripts with fertility gaps. Learn where the math breaks — and what is emerging to fix it.
Multi-Head Attention, Positional Encoding, and the Encoder-Decoder Structure Explained
Multi-head attention, positional encoding, and encoder-decoder structure: the three mechanisms inside every transformer, explained from geometry to implementation.
Prerequisites for Understanding Transformers: From RNNs to Quadratic Scaling Limits
Understand why RNNs failed, how transformer self-attention trades parallelism for quadratic cost, and what these trade-offs predict for long-context language models.
Self-Attention vs. Cross-Attention vs. Causal Masking: Attention Variants and Their Limits
Self-attention, cross-attention, and causal masking solve different problems inside transformers. Learn the math, trade-offs, and the quadratic scaling wall.
What Is Decoder-Only Architecture and How Autoregressive LLMs Generate Text Token by Token
Decoder-only architecture powers every major LLM today. Learn how causal masking, KV cache, and autoregressive generation produce text one token at a time.
What Is Encoder-Decoder Architecture and How Sequence-to-Sequence Models Process Language
Encoder-decoder models compress input sequences into vectors and generate outputs token by token. Learn how seq2seq works and why attention changed everything.
What Is Tokenizer Architecture and How BPE, WordPiece, and Unigram Encode Text for LLMs
Tokenizer architecture determines how LLMs read text. Learn how BPE, WordPiece, and Unigram split text into subword tokens before attention ever fires.
What Is Transformer Architecture and How Self-Attention Replaced Recurrence
Transformers replaced sequential recurrence with parallel self-attention. Understand QKV computation, multi-head attention, and the quadratic scaling trade-off.
Why Decoder-Only Beat Encoder-Decoder: Scaling Laws, Data Efficiency, and the Simplicity Advantage
Decoder-only models won the scaling race by doing less. Learn how a simpler training objective, scaling laws, and MoE extensions beat encoder-decoder design.
Attention Mechanism: Scaled Dot-Product, Self vs Cross
Transformers use weighted averaging, not human-like focus: scaled dot-product, self-attention vs cross-attention, and scaling factor significance.
From Embeddings to Attention: The Math You Need Before Studying Transformers
Master the math behind attention mechanisms — dot products, softmax, QKV matrices, and multi-head projections — before tackling transformer architecture.
Prerequisites for Understanding Transformers: From Embeddings to Matrix Multiplication
Master the math behind transformers: embeddings, matrix multiplication, positional encoding, and multi-head attention explained with the precision engineers actually need.
What Is the Transformer Architecture and How Self-Attention Really Works
The transformer architecture powers every major LLM. Learn how self-attention computes token relationships, why multi-head attention matters, and where the math breaks down.
Why Standard Attention Breaks at Long Contexts: The O(n²) Bottleneck and Attention Sinks
Standard attention scales quadratically with sequence length. Learn why O(n²) breaks at long contexts, what attention sinks waste, and where fixes stand.
Why Transformers Hit a Wall: Quadratic Scaling and the Memory Bottleneck
Transformer self-attention scales quadratically with sequence length. Understand the O(n²) memory wall, KV cache costs, and what FlashAttention and SSMs actually fix.
