LLM Training & Pre-Training

LLM training is the staged process that turns raw internet text into a model that follows instructions: a massive pre-training pass teaches language itself, fine-tuning adapts the result to a task or domain, and preference-based alignment shapes how it behaves. The theme spans that whole ladder — data curation, scaling laws, adaptation methods, and the reward models that steer alignment — and this page maps the reading order through it.

Why LLM training matters for engineers moving into AI

You will probably never pre-train a model — but every model you call was shaped by these stages, and their fingerprints show up in your product: what the model knows and when its knowledge stops (pre-training), why a small fine-tune can beat a bigger base model (adaptation), and why it refuses some requests while over-complying with others (alignment). Understanding the ladder turns “the model is being weird” from a mystery into a diagnosis: you can ask which training stage produced a behavior, and which lever — prompting, fine-tuning, or a different base model — actually addresses it.

Start here: pre-training and scaling laws, the foundations of LLM training

Two concepts explain why every modern LLM looks the way it does, and neither requires touching a GPU to understand.

The first is pre-training itself: the unsupervised phase where a model learns language by predicting the next token across trillions of words. What pre-training is and how LLMs learn language from raw text at scale is the single best first read in this theme, because every later stage — fine-tuning, alignment, everything — edits a model this phase created. When you want to see the machinery, from data curation to checkpoints walks the pipeline stage by stage; and for the honest ceiling, scaling walls, data exhaustion, and the technical limits of pre-training explains why the era of “just add more text” is closing.

The second foundation is the math that decides how big those runs get. Scaling laws are the empirical curves that predict model performance from parameters, data, and compute — how power-law curves predict LLM performance is the orientation read, and diminishing returns and the hard limits of neural scaling covers where the curves bend. If your team ever weighs training or fine-tuning budgets, the Chinchilla-optimal ratios guide turns the theory into decisions, and DeepSeek-v3, OpenAI o3, and the data wall shows how frontier labs are responding as the classic curves flatten.

With these two, every headline about a new model’s parameter count or token budget becomes legible — and the next tier, where developers actually get their hands on the process, makes far more sense.

The core of applied LLM training: fine-tuning and RLHF

This is the layer where a developer’s decisions live: you will never own a pre-training run, but you may well own a fine-tune — and you already live with the consequences of alignment every time you call an API.

Fine-tuning adapts a pre-trained model to your task by continuing training on a curated dataset. Start with how gradient updates adapt pre-trained LLMs to specific tasks for the mental model, then read LoRA vs. QLoRA vs. full fine-tuning — the decision read that determines whether you need a data center or a single GPU. When you are ready to build, the Hugging Face PEFT, Unsloth, and Axolotl guide covers the current tooling; and before you ship, catastrophic forgetting and overfitting names the failure modes that surface only after deployment.

Fine-tuning teaches a model what to do; RLHF — reinforcement learning from human feedback — shapes how it behaves. How human preferences train LLMs to follow instructions is the entry point, and from reward modeling to KL penalties walks every stage of the pipeline; hands-on readers can train a model with OpenRLHF and TRL. The technique has sharp edges — reward hacking, mode collapse, and the unsolved limits of RLHF explains why aligned models still misbehave — and a human cost that rarely makes the changelog: the hidden human cost of RLHF alignment.

Everything above treats the reward signal as a given. The last tier opens that box.

Advanced LLM training: inside the reward model

The most consequential — and least visible — component of the alignment stack is the model that scores the model. A reward model is trained on human preference pairs to grade LLM outputs, and it becomes the optimization target for the whole RLHF loop: whatever the reward model prefers, the LLM learns to produce. How Bradley-Terry scoring shapes LLM alignment is the orientation read; from loss functions to reward hacking lists what you should already know and where these models break. Builders can train and evaluate a reward model with OpenRLHF, TRL, and RewardBench 2, and the reward model race in 2026 tracks how the field is pivoting toward LM-as-a-judge designs. Whose preferences those models encode is not a technicality — whose preferences count traces how reward-model bias decides what LLMs refuse to say.

How pre-training, fine-tuning, and RLHF differ

The confusion that costs teams the most is treating the three training stages as variations of one thing. They answer different questions, consume different data, and fail differently.

Three finer distinctions trip people just as often:

• Full fine-tuning vs parameter-efficient methods. LoRA and QLoRA update a small adapter instead of every weight — usually the right default, but not free of trade-offs; the methods comparison maps when full fine-tuning still wins.
• RLHF vs the reward model. RLHF is the process; the reward model is the component it optimizes against. Debugging alignment behavior usually means asking what the reward model rewarded, not what the process intended.
• PPO vs its successors. The original RLHF recipe used PPO, but DPO- and GRPO-style methods now compete with it — how RLHF alternatives reshaped alignment tracks the shift and what it means for training cost.

Common questions

Q: Where should I start learning LLM training as a software developer? A: Read the two foundations first — pre-training explains what a base model is, and scaling laws explain why it is the size it is. Only then move to fine-tuning and RLHF, which are edits to that base and make little sense without it.

Q: Do I need to fine-tune, or is prompting enough? A: Prompt first; fine-tune when prompting plateaus on a well-defined task with examples you can collect. Fine-tuning buys consistency of format and domain behavior, not new knowledge — what fine-tuning actually changes draws that line before you spend the budget.

Q: What should I read before attempting an RLHF run? A: The whole core tier, then the reward-model prerequisites — the loop optimizes against a learned scorer, so a weak reward model quietly corrupts everything downstream. From loss functions to reward hacking lists what to have in place first.

Q: Why did my fine-tuned model get worse at things it used to do? A: That is catastrophic forgetting: gradient updates on your narrow dataset overwrote broader capabilities. It is the most common surprise in first fine-tunes, and the hard technical limits of LLM fine-tuning covers how to detect it and which mitigations actually help.

Q: Is the data wall real, and does it change what I should build on? A: The limit is real enough that frontier labs are redesigning around it — synthetic data, heavier curation, and post-training gains instead of ever-larger corpora. Scaling walls and data exhaustion separates the evidence from the hype so you can judge the trajectory yourself.

Q: Who actually decides how an aligned model behaves? A: In practice, the annotators and the reward model that aggregates their preferences — their biases become the model’s refusals and defaults. Whose preferences count follows that chain from annotation guidelines to what the model will not say.