Transformer & Attention Internals Cheat Sheets

Attention Mechanism cheat sheet

Every token is weighed against every other one, which is why a longer context costs far more compute.

#Transformer & Attention InternalsAnimated

→ Topic: Attention Mechanism

Decoder-Only Architecture cheat sheet

The model reads prompt and reply as one stream and adds one token per pass, so the causal mask decides what each token may see.

#Transformer & Attention InternalsAnimated

→ Topic: Decoder-Only Architecture

Encoder-Decoder Architecture cheat sheet

One half reads the whole input, the other writes the output token by token, and cross-attention lets the writer look back at every input position.

#Transformer & Attention InternalsAnimated

→ Topic: Encoder-Decoder Architecture

Mixture of Experts cheat sheet

Each token wakes only a few experts, so compute follows active parameters while the router has to keep the load balanced.

#Transformer & Attention InternalsAnimated

→ Topic: Mixture of Experts

Multimodal Architecture cheat sheet

Every input is projected into the token space the language model already reads, so lost detail starts at the connector.

#Transformer & Attention InternalsAnimated

→ Topic: Multimodal Architecture

State Space Model cheat sheet

Cost per token stays flat at any length, because the past is compressed into one state instead of compared token by token.

#Transformer & Attention InternalsAnimated

→ Topic: State Space Model

Tokenizer Architecture cheat sheet

A model only ever sees token IDs; the subword algorithm decides how text splits, and changing it later means retraining.

#Transformer & Attention InternalsAnimated

→ Topic: Tokenizer Architecture

Transformer Architecture cheat sheet

Positional encoding restores word order, attention links every token at once, and the same block repeats layer after layer.

#Transformer & Attention InternalsAnimated

→ Topic: Transformer Architecture

Vision Transformer cheat sheet

Self-attention over patches drops the built-in assumptions of convolution, so data or a pretrained backbone has to supply them.

#Transformer & Attention InternalsAnimated

→ Topic: Vision Transformer