Loss Function

A loss function is a mathematical formula that measures how wrong a model’s predictions are, providing the core feedback signal that drives training and reveals when scaling more data or compute stops improving results.

What It Is

Every neural network needs a way to know whether it’s getting better during training. A loss function provides that feedback. It calculates a single number — the “loss” — that represents the gap between what the model predicted and what the correct answer actually was. The lower the loss, the closer the model is to getting things right.

Think of it like a golf score. A lower number means better performance, and the entire training process is an attempt to drive that score down as far as possible. Each time the model processes a batch of training data, the loss function evaluates the output, and the training algorithm (called an optimizer) adjusts the model’s internal parameters to reduce that score on the next pass. This loop — predict, measure loss, adjust — repeats billions of times during training.

This concept becomes especially important when discussing neural scaling. Scaling laws — the empirical relationships between model size, training data, and performance — are measured primarily through loss. When researchers say that doubling compute produces predictable improvements, they’re tracking loss curves. And when those curves start to flatten, the loss function is the instrument that signals diminishing returns. A model that has saturated its training data will show a loss that stops decreasing meaningfully no matter how much additional compute goes into training.

The most common loss function in large language models is cross-entropy loss, which measures how well the model’s predicted probability distribution over the next token matches the actual next token in the training data. For regression tasks — predicting continuous values like temperature or price — mean squared error is the standard choice. The selection of a loss function shapes what the model optimizes for, and by extension, what kinds of errors it learns to avoid first.

Loss functions also interact directly with techniques like fine-tuning and reinforcement learning from human feedback (RLHF). During fine-tuning, a modified loss function narrows the model’s focus to a specific task or domain. In RLHF, a reward model acts as a proxy loss function, replacing the standard training objective with a human-preference signal that steers the model toward outputs people rate as more helpful or accurate. The choice and design of these loss signals fundamentally determines what the model gets good at — and what it ignores.

How It’s Used in Practice

When ML teams train or fine-tune a model, the loss value is the primary metric they monitor. Training dashboards display a loss curve — a graph that plots loss against training steps — and the shape of that curve tells the story of whether training is working.

A healthy loss curve drops steeply at first, then gradually levels off. If the curve flatlines early, the model architecture might be too small for the task, or the learning rate might be wrong. If training loss keeps dropping but validation loss (measured on held-out data the model hasn’t seen) starts rising, the model is memorizing training examples rather than learning general patterns — a problem called overfitting.

In scaling research, loss curves are the primary evidence for claims about compute-optimal training. The Chinchilla scaling work showed that many large models were undertrained: their loss curves hadn’t flattened yet when training stopped, meaning performance was being left on the table. This insight reshaped how labs allocate compute budgets between model size and training duration.

Pro Tip: When evaluating vendor claims about model improvements, ask whether the improvement was measured by loss reduction or by benchmark scores. A lower loss doesn’t always translate to better performance on the specific tasks you care about — benchmark results on your actual use case matter more than abstract loss numbers.

When to Use / When Not

Common Misconception

Myth: A lower loss always means a better model. Reality: Loss measures how well a model fits its training objective, not how useful it is in practice. A model with extremely low training loss may have memorized its data (overfitting) and perform poorly on new inputs. Loss values also depend on the specific function chosen — comparing loss numbers across different functions or datasets is meaningless. What matters is the trend: consistent loss reduction on both training and validation data, with the gap between them staying small.

One Sentence to Remember

Loss is the training signal that tells a model how wrong it is — and when that signal stops improving despite more data or compute, you’ve found the ceiling of what scaling alone can deliver.

FAQ

Q: What is the difference between a loss function and a cost function? A: In practice, the terms are interchangeable. Some textbooks define loss as the error on a single training example and cost as the average across the full dataset, but this distinction rarely matters outside academic papers.

Q: Why does loss flatten during training even with more data? A: Loss flattens when the model has extracted most learnable patterns from the available data. Further training yields smaller improvements because the remaining errors are harder to correct or represent noise the model shouldn’t learn.

Q: Can you switch the loss function after training has started? A: Technically yes, but it resets the optimization direction. Switching loss functions mid-training is uncommon and usually counterproductive. The standard approach is to select the right loss function before training begins and keep it fixed throughout.