What Is LLM Inference? How a Model Writes Its Answer Token by Token

A large language model has two stages. First it is trained, once: it reads huge amounts of text and sets its parameters (weights), the billions of numbers that hold what it learned. Then it is used, on every request: that is LLM inference, and the parameters stay fixed.

Making that use faster and cheaper is inference optimization. It starts with what this article explains: how the model writes an answer and where the time goes. Its next steps are quantization (weights stored at lower precision) and serving (running the model for many users at once).

Reading the prompt once, then writing one token at a time

The model cuts your prompt into tokens. Our example, a customer-support email thread plus “summarise this”, came to 508. In prefill, it reads all 508 at once. Reading ends with the model choosing what comes next: that is the first token of the answer.

In decode, it writes the rest one token at a time: it picks the next token from everything so far, adds it and repeats, until it writes an end-of-answer token or reaches a length limit.

The model does not recompute the earlier tokens at each step. It keeps the results it already calculated for them in memory (this store is the KV cache), so only the newest token goes through the model. How big that store gets is the subject of our KV cache explainer.

Why every token of a single answer costs a full read of the model

The weights are numbers, stored in memory that the GPU, the chip that does the arithmetic, reads from.

Our model has 1.54 billion parameters of 2 bytes each (16-bit numbers), so its weights take about 3.1 GB. Writing one token means running the numbers of the newest token through all of these weights, so, when one request runs alone, all about 3.1 GB is read again for each token.

Each weight is used only once per token, in about one small multiplication. So the chip spends its time fetching weights from memory rather than multiplying: it calculates faster than memory delivers, and each step mostly waits. NVIDIA calls decode memory-bound for this reason.

Prefill avoids this: the 508 prompt tokens share one read of the weights. Decode steps cannot overlap: token 150 of our answer needs token 149 first.

When the wait starts to hurt

A long prompt makes the pause longer. Every prompt token must go through prefill before the first word can appear, so more tokens mean more work first; the wait is called time to first token.

A long answer makes the typing longer. Each token is one more step; a step’s time is called time per output token.

What you can change without writing code

• Send a shorter prompt. Less to read means a shorter prefill.
• Ask for a shorter answer. Fewer tokens mean fewer steps.
• Use a smaller or compressed model. Fewer bytes to read per token. Storing weights at lower precision is quantization; it can cost accuracy.

If you only wanted to know what LLM inference is, you can stop here. The rest is for developers.

Three numbers to read, and the lever that moves each

We use the definitions from NVIDIA’s benchmarking docs.

Many users at once make each answer slower, but the total larger. A busy service runs several users’ requests as one batch, together in the same step.

Batching spreads the memory cost of the weights over all requests in the batch (NVIDIA), so total tokens per second (throughput) rises. But each step has more to do: more KV cache to read and more calculation, so each answer slows.

Expect batching to raise a busy service’s total output, not the speed of your own answer; continuous batching is one such technique.

An inference engine is software built to run LLM inference efficiently; vLLM, SGLang and TensorRT-LLM are examples. Our deployment guide by Max covers them.

What we measured on a laptop

On 10 November 2026 we timed Qwen2.5-1.5B-Instruct (1.54 billion parameters, about 3.1 GB in float16) on the GPU of an Apple M3 laptop with 24 GB of memory (macOS 15.7.9, Python 3.13.14, Transformers 5.22.1, PyTorch 2.15.0), with no inference server.

The main test used the 508-token email and a 200-token answer, 10 runs. Two more tests changed the prompt length and the number of requests at once, 5 runs each. The laptop was busy, so we show ratios, not seconds.

These numbers come from one laptop and a small model, not from ChatGPT, Claude or a data-center GPU. Production servers use other code and mix prompt lengths; our batch test did not.

Where the picture stops holding

When it breaks: “One read of all the weights per token sets the pace” describes small batches. In a big batch, a step’s main calculations wait for arithmetic, not memory. The authors of How to Scale Your Model, a free Google DeepMind book, explain this switch-over.

Long contexts (the prompt plus the answer so far) add a second case. Each step reads the KV cache as well as the weights, and by the formula in the same book the cache part grows with batch size.