Learning Rate

Learning rate is a hyperparameter that controls the size of weight updates during model training, determining how quickly a neural network adapts to new data during fine-tuning.

What It Is

Every time you fine-tune a language model — whether through LoRA adapters, full parameter updates, or reinforcement learning from human feedback — the model adjusts its internal weights to better match your training data. The learning rate decides how large each adjustment is. Set it too high, and the model overshoots useful patterns, forgetting what it already knew. Set it too low, and training crawls forward, burning compute time without meaningful improvement. This single number has more influence over fine-tuning outcomes than most other settings combined.

Think of it like turning a volume knob on a stereo. A small turn (low learning rate) makes barely noticeable changes — safe, but painfully slow. A big turn (high learning rate) shifts everything fast, but you risk blowing past the sweet spot and distorting the output entirely. The goal is finding the right turn size without overshooting.

In technical terms, the learning rate is a scalar value that multiplies the gradient — the direction of improvement — calculated during backpropagation (the process where the model works backward through its layers to figure out how to improve). Each training step updates the model’s weights by this scaled amount. The gradient tells the model where to move; the learning rate tells it how far to step in that direction.

For fine-tuning pre-trained models, learning rates are generally much smaller than those used when training from scratch. The model already carries strong general knowledge in its weights. The goal is to nudge it toward your specific task — not to rewrite its entire understanding. This is why fine-tuning platforms typically default to rates one to two orders of magnitude below standard training values.

Learning rate also interacts with other training decisions. Batch size, number of epochs, weight decay, and the choice between full fine-tuning versus parameter-efficient methods like LoRA all influence which learning rate works best. Changing one often means reconsidering the others.

How It’s Used in Practice

When teams fine-tune models on platforms that handle infrastructure and optimization, learning rate is one of the first hyperparameters they configure. Most fine-tuning interfaces expose it as a single field, sometimes with a recommended default. Users training custom models for classification, summarization, or domain-specific generation typically start with the platform’s suggested value and adjust based on whether the training loss decreases smoothly.

The most common workflow uses a learning rate schedule — starting at one value and adjusting it as training progresses. Warmup schedules begin with a very small learning rate for the first few steps, ramp up to the target rate, then gradually decay. This prevents early instability when the optimizer hasn’t yet estimated reliable gradient directions, and it helps the model settle into a stable minimum during later training.

Pro Tip: If your fine-tuning loss spikes or oscillates wildly in the first few hundred steps, cut the learning rate in half before changing anything else. Loss that stays flat, on the other hand, usually means the rate is too low — multiply by three to five and watch for smooth downward progress.

When to Use / When Not

Common Misconception

Myth: There is one “correct” learning rate for each model, and finding it guarantees successful fine-tuning. Reality: The optimal learning rate depends on dataset size, batch size, model architecture, fine-tuning method, and the specific task. A rate that works for sentiment classification will likely fail for code generation on the same model. Learning rate is always tuned in context — it is discovered through experimentation, not looked up in a table.

One Sentence to Remember

Learning rate controls the pace of model adaptation — too fast destroys existing knowledge, too slow wastes compute — and getting it right is the single most impactful hyperparameter decision in any fine-tuning job.

FAQ

Q: What happens if the learning rate is too high during fine-tuning? A: The model’s weights change too aggressively, causing training loss to spike or oscillate. In severe cases, the model forgets its pre-trained knowledge entirely — a problem called catastrophic forgetting.

Q: Should I use a different learning rate for LoRA versus full fine-tuning? A: Yes. LoRA updates fewer parameters, so each individual update carries proportionally more weight. LoRA adapters typically need learning rates several times higher than full fine-tuning to compensate for the smaller parameter set being adjusted.

Q: What is a learning rate scheduler? A: A scheduler automatically adjusts the learning rate during training, usually starting higher and decaying over time. Common approaches include cosine decay, linear warmup, and step-based reduction, all designed to balance fast early progress with stable later convergence.