Quantization

A 70-billion-parameter model that refuses to fit on a single workstation GPU is usually a memory problem, and quantization is the lever built to solve it: shrink the precision each weight is stored in, and the same model needs a fraction of the space to run. That tradeoff sits at the compression tier of the inference optimization stack, the step most teams reach for once the decoding loop itself is understood. The tradeoff is never free — every bit removed is a decision about which errors you can tolerate — and this topic exists to make that decision deliberate rather than accidental.

Reading the compression tradeoff in order

Start with how FP32-to-INT4 compression makes LLMs run on consumer hardware — it is the topic’s own definition, and every later article assumes you already know why shrinking a number’s precision shrinks its memory footprint. Once the mechanism is set, GPTQ vs AWQ vs GGUF vs bitsandbytes maps the actual format decision — which one you pick determines which serving stack, and which hardware, you are committing to. Read the cost of getting it wrong before you ship: accuracy collapse, task-specific degradation, and the hard limits of sub-4-bit quantization is the honest ceiling on how far the compression goes before quality, not memory, becomes the constraint.

When you’re ready to deploy, the AWQ, GGUF, and vLLM deployment guide walks the full path from checkpoint to serving, including which older tools are already dead. For where the field is moving, BitNet, FP8 native, and the 1-bit frontier tracks the tier that skips post-training compression altogether. Close with the hidden costs of quantized AI if you want the question the benchmark tables don’t ask: who ends up running the degraded copy.

Where quantization gets confused with its neighbours

A quantization format is also a serving-engine decision, which the memory-savings pitch skips. A model quantized to GGUF serves through llama.cpp; pointing the same checkpoint at vLLM’s continuous batching engine simply does not work — it is a different runtime. Picking a format because it saves the most memory, then discovering it cannot reach your batching stack, is the deployment mistake practitioners hit most.

A quantized model producing bad output and a badly sampled one look identical from the outside, but the fix is different. Quantization degrades the weights, so the same distortion shows up on every request regardless of settings. Temperature and sampling changes which token gets picked from an otherwise-unchanged distribution, so its effect varies request to request even on identical weights. If the same prompt fails the same way every time, suspect the compression; if it fails differently on every run, suspect the sampling configuration.

Quantizing a model and training it natively in low precision are not the same step. Everything above compresses a model that was already trained at high precision — post-training quantization, applied once before deployment. FP8-native training and BitNet skip that step: the model trains in low precision from the start, so there is no higher-precision version to lose in the first place.

Common questions about quantization

Q: Does quantizing a model always make it faster, not just smaller? A: Usually, but not guaranteed — lower precision cuts memory traffic, which is often the real bottleneck, so decoding speeds up alongside the footprint. How FP32-to-INT4 compression works explains why memory traffic, not raw arithmetic, is what most quantized models are trading against.

Q: Why do quantization tutorials from 2024 keep failing when I follow them today? A: Because the tooling underneath them changed — AutoAWQ and AutoGPTQ are deprecated, replaced by llm-compressor and GPTQModel, so an old walkthrough often points at packages that no longer install cleanly. The deployment guide flags the current toolchain.

Q: How do I know if my use case can tolerate sub-4-bit quantization? A: Test the specific task, not the average — accuracy degrades unevenly per task and language, so a model that holds up in chat can still collapse on structured reasoning or lower-resource languages at the same bit-width. The sub-4-bit limits research documents which task types go first.

Q: If I’m starting a new deployment today, should I wait for BitNet-style native low-precision models instead of quantizing? A: Not yet, for most teams — native low-precision models are still narrow in ecosystem support and model choice, while post-training quantization works on almost anything already trained. The 1-bit frontier read tracks whether that gap closes.

Q: Who actually loses out when a lab ships a quantized model instead of the full-precision one? A: Not an evenly distributed cost — the users routed to the compressed copy are usually the ones with the least leverage to demand the better one, and the degradation is easy to miss without dedicated evaluation. The hidden costs of quantized AI traces who bears it.

Part of the inference optimization stack · closest neighbour: continuous batching. New to inference work from a software background? Start with the story: Inference Optimization for Developers: What Transfers and What Breaks.