
What Are Scaling Laws and How Power-Law Curves Predict LLM Performance
Scaling laws predict LLM performance from model size, data, and compute via power-law curves. Learn the math behind Kaplan, Chinchilla, and Densing Law.
If you’re choosing a pretrained transformer model — decoder-only or otherwise — the first question is usually “how big does it need to be?” Scaling laws give you a data-driven answer instead of a guess. They describe the repeatable, mathematical relationship between three inputs (parameters, training tokens, and compute) and one output (model quality, measured as cross-entropy loss). Cross-entropy loss is simply a number that captures how surprised the model is by the next word — lower loss means better predictions.
Think of it like a recipe yield curve. If you know that doubling flour and doubling oven time produces 1.8 times more bread (not double), you can plan your bakery’s output without baking every possible batch first. Scaling laws do the same for model training: they let you predict how good a model will be at a given size without actually running the full experiment.
The concept took shape in two studies. According to Kaplan et al., the 2020 paper “Scaling Laws for Neural Language Models” showed that loss decreases as a smooth power law when you increase any of the three factors independently, and that larger models are more sample-efficient — they reach lower loss with fewer training tokens relative to their size. This finding accelerated the trend toward building ever-larger models.
Then came a significant correction. According to Wikipedia, the 2022 “Chinchilla” paper by Hoffmann et al. at DeepMind demonstrated that many large models were under-trained relative to their parameter count. The study proposed a roughly 20-to-1 ratio of training tokens to parameters for compute-optimal training. A model with one billion parameters, for example, should see about twenty billion tokens during training. This single insight shifted industry practice away from simply making models larger and toward balancing size with the volume of training data.
For anyone selecting a decoder-only transformer today, scaling laws explain why you can look at published benchmarks and make informed trade-offs: a smaller model trained on the right amount of data can match or outperform a larger but under-trained model, often at a fraction of the inference cost.
The most common place you’ll encounter scaling laws is during model selection. When a team evaluates pretrained decoder-only transformers — deciding between a smaller model trained with more data versus a larger model trained with less — scaling laws provide the framework for that comparison. Instead of running expensive head-to-head experiments, engineers plot expected loss against their available compute budget and pick the model that lands on the most favorable point of the curve.
This matters outside the lab, too. Product managers reviewing vendor pitches will often see claims like “trained at optimal compute” or “Chinchilla-optimal.” These phrases reference the scaling law research directly, meaning the model was trained with a token-to-parameter ratio that maximizes quality per dollar of compute spent.
When comparing two pretrained models of different sizes, don’t assume the bigger one wins. Check how many tokens each was trained on relative to its parameter count. A model that followed compute-optimal scaling (roughly twenty tokens per training parameter) often outperforms a model with twice the parameters but half the recommended training data.
Start with how power-law curves predict LLM performance — it establishes the mechanism every later article assumes: loss falls predictably as parameters, data, and compute grow, until it doesn’t. Read diminishing returns, data exhaustion, and the hard limits of neural scaling next to see exactly where those curves bend and why data, not compute, is the tighter constraint. When a real budget is on the table, applying scaling laws and Chinchilla-optimal ratios to training decisions turns the theory into a defensible ratio. For how the frontier is responding as the classic formula strains, DeepSeek-v3, OpenAI o3, and the data wall tracks the fragmentation into competing levers. Close with the scaling tax — the same curves that justify a bigger run also justify who gets to run it.

Scaling laws are the prediction; pre-training is the run they predict. A scaling law tells you what loss to expect from a given mix of parameters, tokens, and compute before you spend a single GPU-hour; pre-training is the industrial process that actually consumes that budget. Confusing the two leads teams to treat a scaling estimate as a training plan, when it is only the input to one.
A second confusion sits inside the laws themselves: “scaling” is not one axis. Parameter scaling, data scaling, and inference-time scaling behave differently and hit different ceilings — a model can be data-starved while still having compute to spare, or the reverse. Treating all three as interchangeable is why some teams add parameters to a problem that was actually a data problem.
A third: a compute-optimal training ratio is not a deployment plan. The Chinchilla 20:1 figure balances training-time compute against model quality; it says nothing about how many inference requests that model will serve once shipped. Teams that lock in a training ratio and only then look at serving costs are optimizing half the bill.
| Scenario | Use | Avoid |
|---|---|---|
| Choosing between pretrained models of different sizes | ✅ | |
| Estimating compute budget for a new training run | ✅ | |
| Predicting exact benchmark scores on a specific task | ❌ | |
| Comparing models trained on very different data distributions | ❌ | |
| Planning hardware procurement for a training cluster | ✅ | |
| Fine-tuning a small model on a narrow domain dataset | ❌ |
Q: Why do experts warn against following the Chinchilla 20:1 ratio literally? A: Because it answers a training-compute question, not a deployment one — and the original coefficients behind it were later corrected. The Chinchilla-optimal ratios guide walks through working backward from deployment constraints instead of copying the ratio as-is.
Q: Do scaling laws mean the largest model you can afford is always the best choice? A: No. Gains follow a power curve that flattens, so beyond a point extra parameters or compute buy a shrinking loss improvement rather than proportional new capability — the technical limits of neural scaling traces exactly where that bend happens.
Q: If pre-training gains are slowing, should teams shift budget toward inference-time compute instead? A: Increasingly, yes — the frontier no longer competes on one scaling axis. DeepSeek-v3, OpenAI o3, and the data wall shows pre-training scale, inference-time compute, and post-training work such as RLHF now competing for the same budget.
Q: If scaling laws show diminishing returns, why do labs keep chasing bigger training runs? A: Partly because the economics reward scale itself, not only the marginal quality gain — access to enough compute and data is its own competitive moat. The scaling tax traces how that incentive concentrates power even as the curves flatten.
Part of the LLM training theme · closest neighbour: pre-training. New to this from a software background? Start with the story: LLM Training for Developers: Which Instincts Help, Which Mislead.
Scaling laws reveal predictable patterns in how neural networks improve with size. Understanding these power-law relationships explains why certain capabilities appear only beyond specific thresholds.
Concepts covered

Scaling laws predict LLM performance from model size, data, and compute via power-law curves. Learn the math behind Kaplan, Chinchilla, and Densing Law.

Scaling laws predict how AI models improve with compute, but power-law exponents guarantee diminishing returns. Learn where the ceilings are — and why.
Practical guides cover how to use scaling curves for compute-optimal training decisions and where standard predictions break down in real-world resource allocation.
Tools & techniques

Apply scaling laws and Chinchilla-optimal ratios to real LLM training decisions. Compute budgeting, model sizing, and inference-aware trade-offs for 2026.
The relationship between scale and capability is actively shifting as new training paradigms challenge established assumptions.
Models & benchmarks
Updated March 2026

Scaling laws split in 2025 along three axes. DeepSeek proved efficiency, o3 proved inference-time compute, and the data wall is forcing the post-Chinchilla rethink.
Uncritical faith in scaling can concentrate power among the few organizations able to afford massive compute, while masking diminishing returns and environmental costs.
Risks & metrics

Scaling laws promise better AI through more compute, but the energy, water, and capital costs concentrate power among the few. Who pays the scaling tax?