Scaling Laws

Authors 5 articles 49 min total read
Definition
Empirical power-law relationships showing how a language model’s performance predictably improves as you increase model size, training data, or compute budget, enabling teams to forecast results before committing resources.
Also called: neural scaling laws, compute scaling, Chinchilla scaling

What scaling laws are

If you’re choosing a pretrained transformer model — decoder-only or otherwise — the first question is usually “how big does it need to be?” Scaling laws give you a data-driven answer instead of a guess. They describe the repeatable, mathematical relationship between three inputs (parameters, training tokens, and compute) and one output (model quality, measured as cross-entropy loss). Cross-entropy loss is simply a number that captures how surprised the model is by the next word — lower loss means better predictions.

Think of it like a recipe yield curve. If you know that doubling flour and doubling oven time produces 1.8 times more bread (not double), you can plan your bakery’s output without baking every possible batch first. Scaling laws do the same for model training: they let you predict how good a model will be at a given size without actually running the full experiment.

The concept took shape in two studies. According to Kaplan et al., the 2020 paper “Scaling Laws for Neural Language Models” showed that loss decreases as a smooth power law when you increase any of the three factors independently, and that larger models are more sample-efficient — they reach lower loss with fewer training tokens relative to their size. This finding accelerated the trend toward building ever-larger models.

Then came a significant correction. According to Wikipedia, the 2022 “Chinchilla” paper by Hoffmann et al. at DeepMind demonstrated that many large models were under-trained relative to their parameter count. The study proposed a roughly 20-to-1 ratio of training tokens to parameters for compute-optimal training. A model with one billion parameters, for example, should see about twenty billion tokens during training. This single insight shifted industry practice away from simply making models larger and toward balancing size with the volume of training data.

For anyone selecting a decoder-only transformer today, scaling laws explain why you can look at published benchmarks and make informed trade-offs: a smaller model trained on the right amount of data can match or outperform a larger but under-trained model, often at a fraction of the inference cost.

Scaling laws forecast loss before a run starts; balancing data with size beats adding parameters alone. · More AI cheat sheets
  • Scaling laws let you predict a model’s loss before training it — but the classic Chinchilla 20:1 ratio optimizes training compute only, not the inference cost that now dominates real budgets.
  • Gains follow a power curve that flattens: past a point, more compute buys shrinking improvements, not proportional capability.
  • By 2026 the frontier race split across three levers — pre-training scale, inference-time compute, and post-training optimization — not one formula anymore.
  • The same curves that predict better loss also predict concentrated compute costs and data demands few organizations can match.

How scaling laws are used in practice

The most common place you’ll encounter scaling laws is during model selection. When a team evaluates pretrained decoder-only transformers — deciding between a smaller model trained with more data versus a larger model trained with less — scaling laws provide the framework for that comparison. Instead of running expensive head-to-head experiments, engineers plot expected loss against their available compute budget and pick the model that lands on the most favorable point of the curve.

This matters outside the lab, too. Product managers reviewing vendor pitches will often see claims like “trained at optimal compute” or “Chinchilla-optimal.” These phrases reference the scaling law research directly, meaning the model was trained with a token-to-parameter ratio that maximizes quality per dollar of compute spent.

When comparing two pretrained models of different sizes, don’t assume the bigger one wins. Check how many tokens each was trained on relative to its parameter count. A model that followed compute-optimal scaling (roughly twenty tokens per training parameter) often outperforms a model with twice the parameters but half the recommended training data.

How to read the scaling-laws ladder

Start with how power-law curves predict LLM performance — it establishes the mechanism every later article assumes: loss falls predictably as parameters, data, and compute grow, until it doesn’t. Read diminishing returns, data exhaustion, and the hard limits of neural scaling next to see exactly where those curves bend and why data, not compute, is the tighter constraint. When a real budget is on the table, applying scaling laws and Chinchilla-optimal ratios to training decisions turns the theory into a defensible ratio. For how the frontier is responding as the classic formula strains, DeepSeek-v3, OpenAI o3, and the data wall tracks the fragmentation into competing levers. Close with the scaling tax — the same curves that justify a bigger run also justify who gets to run it.

MAX asks: 'My team only has budget for one training run — is the Chinchilla ratio still the right target?' MONA answers: 'That ratio optimizes training compute only. If your inference volume is high, work backward from serving cost instead.' — comic dialog.
The 20:1 ratio was never built to answer your inference bill.

How scaling laws differ from the training run they govern

Scaling laws are the prediction; pre-training is the run they predict. A scaling law tells you what loss to expect from a given mix of parameters, tokens, and compute before you spend a single GPU-hour; pre-training is the industrial process that actually consumes that budget. Confusing the two leads teams to treat a scaling estimate as a training plan, when it is only the input to one.

A second confusion sits inside the laws themselves: “scaling” is not one axis. Parameter scaling, data scaling, and inference-time scaling behave differently and hit different ceilings — a model can be data-starved while still having compute to spare, or the reverse. Treating all three as interchangeable is why some teams add parameters to a problem that was actually a data problem.

A third: a compute-optimal training ratio is not a deployment plan. The Chinchilla 20:1 figure balances training-time compute against model quality; it says nothing about how many inference requests that model will serve once shipped. Teams that lock in a training ratio and only then look at serving costs are optimizing half the bill.

When scaling laws fit — and when they don't

ScenarioUseAvoid
Choosing between pretrained models of different sizes✅
Estimating compute budget for a new training run✅
Predicting exact benchmark scores on a specific task❌
Comparing models trained on very different data distributions❌
Planning hardware procurement for a training cluster✅
Fine-tuning a small model on a narrow domain dataset❌

Common questions about scaling laws

Q: Why do experts warn against following the Chinchilla 20:1 ratio literally? A: Because it answers a training-compute question, not a deployment one — and the original coefficients behind it were later corrected. The Chinchilla-optimal ratios guide walks through working backward from deployment constraints instead of copying the ratio as-is.

Q: Do scaling laws mean the largest model you can afford is always the best choice? A: No. Gains follow a power curve that flattens, so beyond a point extra parameters or compute buy a shrinking loss improvement rather than proportional new capability — the technical limits of neural scaling traces exactly where that bend happens.

Q: If pre-training gains are slowing, should teams shift budget toward inference-time compute instead? A: Increasingly, yes — the frontier no longer competes on one scaling axis. DeepSeek-v3, OpenAI o3, and the data wall shows pre-training scale, inference-time compute, and post-training work such as RLHF now competing for the same budget.

Q: If scaling laws show diminishing returns, why do labs keep chasing bigger training runs? A: Partly because the economics reward scale itself, not only the marginal quality gain — access to enough compute and data is its own competitive moat. The scaling tax traces how that incentive concentrates power even as the curves flatten.

Part of the LLM training theme · closest neighbour: pre-training. New to this from a software background? Start with the story: LLM Training for Developers: Which Instincts Help, Which Mislead.

Sources

1

Understand the Fundamentals

Scaling laws reveal predictable patterns in how neural networks improve with size. Understanding these power-law relationships explains why certain capabilities appear only beyond specific thresholds.

2

Build with Scaling Laws

Practical guides cover how to use scaling curves for compute-optimal training decisions and where standard predictions break down in real-world resource allocation.

4

Risks and Considerations

Uncritical faith in scaling can concentrate power among the few organizations able to afford massive compute, while masking diminishing returns and environmental costs.