Inference Optimization Cheat Sheets

Continuous Batching cheat sheet

Finished requests free their slot at once, so the GPU stops waiting for the slowest request in the batch.

#Inference OptimizationAnimated

→ Topic: Continuous Batching

Inference cheat sheet

Each token waits on the one before it, so more GPUs serve more requests rather than one response faster.

#Inference OptimizationAnimated

→ Topic: Inference

Quantization cheat sheet

Fewer bits barely move the average, but the loss lands unevenly, so test your own task.

#Inference OptimizationAnimated

→ Topic: Quantization

Temperature and Sampling cheat sheet

Temperature only changes which words get picked from the same probabilities; the model's knowledge stays the same.

#Inference OptimizationAnimated

→ Topic: Temperature and Sampling

What is a KV cache? cheat sheet

A KV cache is the memory an LLM keeps while it writes. Learn what it stores, what it saves, and how its size grows with text length and users.

#Inference Optimization

→ Read the article

What is LLM inference? cheat sheet

LLM inference is a trained model writing its answer, in most LLMs one token at a time. Learn why it is slow and what you can change.

#Inference Optimization

→ Read the article