Temperature and Sampling

Every generation call an LLM makes ends at the same fork: dozens of candidate next tokens, ranked by probability, and one rule that decides which one gets typed. That rule is temperature and sampling — the cheapest lever in the inference optimization stack, because it costs no extra hardware, only judgment. Get it wrong and a perfectly capable model looks unreliable, either locked into safe repetition or drifting into nonsense; get it right and the same weights serve chatbots, coding assistants, and structured extraction pipelines with entirely different personalities. This is where an engineer’s instinct to blame “the model” gets replaced with the more useful question — which sampling rule decided.

The reading order: from the probability distribution to production defaults

Start with how softmax scaling controls text generation randomness — it opens the one mechanism every later article assumes you already understand: temperature reshapes the distribution before a single token is chosen. From there, the full sampling method comparison adds the filters that act after that reshaping — top-k, top-p, min-p, and beam search each keep a different slice of the ranked candidates. Read the hard limits of sampling parameter tuning next, before touching a production config — it shows exactly where the knobs stop helping and start producing loops or hallucination.

Once the mechanics are set, the per-use-case configuration guide turns them into settings for code, RAG, and creative workloads. For what is shifting under those settings, the piece on locked temperatures and min-p adoption tracks how providers are changing the defaults. Close with the argument about who controls those defaults if you want the stakes beyond your own config file.

Where sampling parameters end and the other inference levers begin

Sampling gets confused with the other two levers in this theme, and each confusion sends debugging in the wrong direction.

• Sampling is not all of inference. Inference is the entire autoregressive decoding loop — memory management, scheduling, the forward pass. Sampling is the very last step of that loop: the rule that turns a finished probability distribution into one chosen token. Slow inference is rarely a sampling problem.
• Sampling is not quantization. Quantization changes the model’s weights before generation ever starts; sampling changes the selection rule at generation time. They compose — a heavily quantized model still needs a sampling strategy — but degenerate or repetitive output right after quantizing is usually a precision problem, not a temperature problem, and vice versa.
• Sampling is not continuous batching. Batching decides which requests share a GPU pass; it does not touch how a token is chosen within any one of them. The catch: batch composition can still affect output consistency at a fixed setting, which is why testing sampling only at low concurrency before shipping is a common blind spot.

Common questions about temperature and sampling

Q: Does setting temperature to 0 make an LLM’s output fully deterministic? A: Not automatically — temperature at 0 collapses selection to the single highest-probability token, but production serving conditions such as batching can still introduce variation at the same setting. How softmax scaling controls text generation randomness explains what zero temperature does and does not guarantee.

Q: Should temperature and top-p be tuned at the same time? A: No — treat them as one dial, not two. Setting both aggressively is a documented misconfiguration: the two filters compete over the same candidate pool, producing unpredictable swings between flat and erratic output. The per-use-case configuration guide recommends picking one control per request.

Q: Can a provider lock or remove a sampling parameter my application already depends on? A: Yes, and it is already happening — some proprietary APIs are locking reasoning-model temperature while open-source runtimes move toward min-p as the smarter default. The 2026 shift in sampling defaults tracks which knobs are disappearing and what replaces them.

Q: Does lowering temperature always fix repetitive or unreliable output? A: No — pushed too low, sampling can trap generation in loops instead of fixing them; the failure mode looks similar but the cause and the correction differ. The hard limits of sampling parameter tuning diagnoses both extremes.

Part of the inference optimization theme · closest neighbour: inference. Coming to sampling from a software background? Start with the story: Inference Optimization for Developers: What Transfers and What Breaks.