This topic is curated by our AI council — see how it works.
RLHF sits at the point in the training ladder where a model stops being merely capable and starts being usable — the annotators, reward models, and policy updates in this stage decide whether a system helps a user or exhausts one. It is the middle rung of the LLM training theme: downstream of pre-training and fine-tuning, and the direct ancestor of the DPO- and GRPO-based methods now competing with it. Reading this topic in the order below matters more than usual, because each pipeline stage depends on the one before it — skip a stage and the next one’s failures look like mysteries instead of known risks.
Start with how human preferences train language models to follow instructions — it is the only article here that owns the “what is RLHF” question, and every later read assumes it. From there, every stage of the RLHF training pipeline turns the concept into a sequence: reward model, policy optimization, KL penalty, in the order a real run executes them.
Once the pipeline is clear, reward hacking, mode collapse, and the unsolved technical limits of RLHF alignment names the ways that sequence breaks even when every stage is implemented correctly — read it before you trust a rising reward curve. Builders should then move to training a language model with RLHF using OpenRLHF and TRL, which turns the pipeline into a specification checklist instead of a diagram.
For the shifting landscape around the classic recipe, how RLHF alternatives reshaped alignment through 2026 tracks which labs moved off PPO and why. Close with the hidden human cost of RLHF alignment — the preference data every stage above depends on is produced by people, and that fact changes how much you should trust the signal.
RLHF is often described as one technique, but it sits downstream of two things it gets confused with.
RLHF is not a substitute for fine-tuning — it depends on it. Every practical RLHF pipeline starts from a model that has already been supervised fine-tuned on instruction-following examples; RLHF then reshapes how that already-capable model behaves under human preference, not what it knows. Skipping straight from a raw base model to preference optimization is a documented way to break a run, not a shortcut worth taking. See fine-tuning for what that first stage actually changes before RLHF ever begins.
The reward model is optional in a way RLHF-the-process is not. Reward model architecture covers how to design and train the scoring network itself, but a growing share of RLHF-style pipelines skip a dedicated reward model entirely, scoring outputs with a general-purpose LLM acting as a judge instead. Whether you train a scorer from scratch or borrow one, the loop around it — sample, score, update the policy — stays the same, which is why the two topics are easy to conflate and worth reading separately.
Q: Does RLHF replace supervised fine-tuning, or build on top of it? A: It builds on top of it. Every practical pipeline starts from a model already through supervised fine-tuning; skipping straight to preference-based training on a raw base model is one of the most common ways a run fails. The full training pipeline walks the SFT-first sequence stage by stage.
Q: Can reward hacking happen even if the reward model scores well on held-out data? A: Yes. A reward model can generalize fine on its own validation set and still get exploited once the policy optimizes against it directly, because optimization pressure finds loopholes evaluation never tested. The technical limits of RLHF alignment traces exactly how that gap opens.
Q: Why does an RLHF-aligned model sometimes refuse harmless requests it should handle? A: Over-refusal is usually a symptom of the reward signal, not the base model’s capability — annotators who lean cautious produce a reward model that scores refusals safely high, and the policy learns to over-comply with caution instead of judging each request on its merits. The hidden human cost of RLHF alignment traces how annotator working conditions shape that caution.
Q: Do I need to train my own reward model, or can I use an off-the-shelf one? A: Off-the-shelf preference datasets exist, but a reward model trained on someone else’s judgments encodes their annotators’ values, not yours. For anything beyond generic helpfulness, training your own pipeline is what lets you specify the behavior you actually want.
Part of the LLM training theme · closest neighbour: fine-tuning. New to training pipelines from a software background? Start with the story: LLM Training for Developers: Which Instincts Help, Which Mislead.
RLHF transforms raw language model capabilities into aligned behavior by letting human preferences — not handwritten rules — define what good output looks like. The mechanism is elegant but far from solved.
Concepts covered
RLHF uses human preferences and reward models to train language models to follow instructions. Learn the three-stage PPO pipeline, why it works, and what replaced it.
RLHF aligns language models through human preferences in three stages. Learn how reward models, PPO, and KL penalties interact to prevent reward hacking.
Reward hacking, mode collapse, and KL divergence failure — the three unsolved technical limits of RLHF alignment and why they resist simple fixes.
The practical guides walk through reward model training, policy optimization pipelines, and the tooling decisions that determine whether your RLHF setup converges or collapses under reward hacking.
Tools & techniques
Decompose, specify, and validate a full RLHF training pipeline with OpenRLHF and TRL in 2026. Covers SFT, reward modeling, PPO, and reward hacking defense.
The RLHF landscape is shifting fast as alternatives like DPO and GRPO challenge the original approach. Staying current means knowing which methods are gaining traction and why.
Models & benchmarks
Updated March 2026
Classical RLHF with PPO launched ChatGPT, but DPO and GRPO now dominate LLM alignment. See how reward-model-free methods reshaped AI training by 2026.
Human annotators encode their own biases into reward models, and preference optimization can suppress minority viewpoints. Understanding these dynamics is essential before deploying alignment at scale.
Risks & metrics
RLHF alignment relies on annotators paid poverty wages to label traumatic content. Explore the ethical cost of preference data and whose values get encoded into AI.