Preference tuning aligns a language model with human preferences — helpfulness, harmlessness, and following instructions — after pretraining and…
See why a pretrained, instruction-tuned model still needs preference tuning.
Pretraining makes a model fluent and knowledgeable, and supervised fine-tuning (SFT) teaches it to follow instructions from example responses. But SFT only shows good answers; it doesn't teach the model to prefer a better answer over a merely acceptable one, or to reliably avoid unhelpful or unsafe outputs.
Preference tuning closes that gap. Instead of a single 'correct' answer, it learns from comparisons — humans (or a model) judging which of two responses is better — steering the model toward responses people actually prefer. It is the final stage that turns a capable base into a helpful, well-behaved assistant.
Walk the three stages of Reinforcement Learning from Human Feedback.
RLHF has three stages. First, supervised fine-tuning gives a starting instruction-following model. Second, humans compare pairs of the model's responses and label which is better; those comparisons train a reward model that scores any response by predicted human preference. Third, the model is optimized with reinforcement learning (commonly PPO) to produce responses the reward model scores highly.
A regularization term keeps the tuned model from drifting too far from the SFT model, so it improves on preference without forgetting its abilities or gaming the reward.
RLHF is effective but complex: it means training and maintaining a separate reward model plus running an RL loop, which is compute-heavy and unstable, and the model can learn to exploit flaws in the reward model (reward hacking).
Understand how DPO skips the reward model and RL loop.
Direct Preference Optimization (DPO) reaches the same target with far less machinery. It uses the same preference data — pairs of a preferred and a rejected response — but trains the model directly with a simple classification-style loss that raises the probability of preferred responses and lowers that of rejected ones, relative to a reference model.
There is no separate reward model and no reinforcement-learning loop. That makes DPO simpler to implement, cheaper, and more stable to train, which is why it has become a popular default for open models. It still needs good preference data — that is the real ingredient either way.
Decide between RLHF and DPO and avoid the shared traps.
For most teams, DPO is the pragmatic starting point: comparable alignment gains with much less complexity and instability. RLHF remains valuable when you need the flexibility of an explicit reward model — for example, combining multiple objectives, doing online exploration, or iterating the reward signal — and have the infrastructure to run it well.
Both hinge on the same thing: the quality and coverage of the preference data. Alignment is only as good as the comparisons it learns from.
Watch for: low-quality or biased preference data (the model learns those biases); over-optimizing so the model becomes sycophantic or evasive to please the signal; and drifting too far from the base so it loses capabilities. Keep a reference model as an anchor, and evaluate on held-out prompts for both helpfulness and safety, not just the training signal.
Preference tuning aligns a model with human preferences after pretraining and SFT. RLHF does it in three stages: SFT, a reward model trained on human comparisons, then RL optimization against that reward with a regularizer — powerful but complex and prone to reward hacking. DPO reaches similar alignment by training directly on preferred-vs-rejected pairs, dropping the reward model and RL loop for a simpler, more stable process. Both live or die by preference-data quality; evaluate helpfulness and safety on held-out prompts.
You have a capable SFT model and a budget for one alignment pass. Decide whether to use DPO or RLHF and justify it, describe the preference data you would collect, and name one evaluation you would run to catch sycophancy or capability loss from over-optimizing the preference signal.
Why do models need preference tuning after supervised fine-tuning?
Preference tuning aligns the model with what people actually prefer, using comparative feedback rather than single reference answers.
What are the three stages of RLHF?
RLHF builds an SFT model, learns a reward model from preference labels, and uses RL (e.g. PPO) to maximize that reward with a regularizer.
How does DPO differ from RLHF?
DPO optimizes the model directly from preference pairs relative to a reference model, avoiding RLHF's reward model and reinforcement-learning complexity.
What do both RLHF and DPO depend on most?
Both methods learn from preference comparisons, so biased or thin data limits alignment regardless of which algorithm is used.