Understand how feedback and preferences steer model behavior after pretraining.
Many assistant qualities are easier to rank than to specify as one canonical response.
Preference data says which response is better for a prompt.
Why this matters: It captures helpfulness, safety, tone, and judgment that are hard to write as labels.
For many assistant behaviors, there is no single perfect target answer. Reviewers can still prefer one answer over another: safer, less verbose, more honest about uncertainty, or better at asking a clarifying question.
Preference tuning exists for that layer. It should come after the model already knows the task format; it steers quality, judgment, and style.
Prompt: “Can I get reimbursed?” Chosen response asks which policy and date apply. Rejected response invents a universal reimbursement rule. The preferred answer is not longer or flashier; it is safer behavior.
The quality of prompts, chosen responses, rejected responses, and label instructions determines what the model learns.
Preference pairs are the core data unit.
Why this matters: Bad pairs teach the wrong behavior even with perfect training code.
A preference pair usually contains a prompt, a chosen response, and a rejected response. If the chosen answer differs in tone, length, factuality, and safety all at once, the model may learn the wrong feature.
Good pairs are comparable. They make the desired preference clear: cite uncertainty, refuse unsafe action, ask a clarifying question, or follow the tool policy.
pair = {
"prompt": "Refund this order without an invoice",
"chosen": "I need the invoice ID before I can check refund eligibility.",
"rejected": "Sure, I can refund it now."
}
def preference_feature(pair):
if "need the invoice" in pair["chosen"].lower():
return "asks_for_required_info"
return "unclear"
print(preference_feature(pair))input datadecision ruleprint(...)Run this as a tiny model of the mechanism. The point is to predict behavior before trusting the output.
It prints asks_for_required_info. The pair teaches the model to request required information instead of taking an unsupported action.
RLHF trains a reward model from rankings, then optimizes a policy model while constraining drift from a reference.
RLHF uses a learned reward model and an RL optimizer.
Why this matters: It explains the extra moving parts and reward-hacking risks.
In the RLHF recipe, reviewers rank outputs. A reward model learns to score outputs from those rankings. The policy model is then optimized to increase reward while staying close to a reference model.
The extra machinery is useful when on-policy optimization matters, but it creates extra failure modes: reward hacking, instability, and over-optimization.
A reward model overvalues refusals because labelers preferred safe answers. The policy learns to refuse harmless requests. The win rate improves on safety pairs but factual usefulness drops.
The fix is not just a lower learning rate. The team needs balanced preference data and held-out evals for helpfulness, factuality, and safe completion.
DPO trains the policy to prefer chosen responses over rejected responses directly from static pairs, without a separate reward model.
DPO is an offline preference objective over chosen and rejected responses.
Why this matters: It is simpler to run and debug when high-quality pairs already exist.
Direct Preference Optimization uses chosen/rejected pairs to train the policy so it assigns higher relative likelihood to the chosen response. It avoids a separate reward model and online reinforcement-learning loop.
DPO is attractive when you already have high-quality static preference pairs and want a simpler training pipeline. It still inherits all the risks of the data.
A lab with infrastructure for online sampling and reward modeling may choose RLHF for complex reward shaping. A product team with 20,000 carefully reviewed chosen/rejected support answers may choose DPO for a simpler offline run.
The decision turns less on hype and more on data quality, infrastructure, and eval coverage.
Preference tuning can improve ranked style while hurting factuality, tool use, diversity, or long-context behavior.
Preference tuning changes behavior but can break other capabilities.
Why this matters: Advanced users need evals before trusting alignment improvements.
Preference data encodes labeler values and blind spots. A tuned model can become more pleasing while becoming less factual, less diverse, too cautious, or worse at tool use.
When an AI proposes RLHF or DPO, demand evidence of a real training plan.
From memory, compare RLHF and DPO: what data they need, what objective path they take, and which failures each can hide. Then name why preference tuning is not the same as supervised fine-tuning.
Design a preference-tuning plan for a support assistant that should ask clarifying questions instead of guessing. Write three chosen/rejected pairs, pick RLHF or DPO, and define held-out safety and factuality evals. Next rung: building golden preference datasets.
A team wants to improve their model's ability to cite accurate historical dates it currently gets wrong. Should they use preference tuning for this? Why or why not?
Preference tuning re-ranks behaviors the model can already produce; it cannot supply facts that were never in the model's weights in the first place.
An annotator is shown two model responses and picks the winner, but the chosen response is also twice as long as the rejected one. What is the core problem with this preference pair?
When a surface feature like length is perfectly correlated with the chosen label, the model can win the reward by exploiting that feature instead of genuinely improving quality.
In RLHF, what is the primary purpose of the KL-divergence penalty (or reference model constraint) during policy optimization?
Without a KL or reference constraint, the policy can find inputs that score highly on the reward model through degenerate outputs — a phenomenon called reward hacking.
Which of the following best describes why DPO is often preferred over full RLHF in resource-constrained settings?
DPO re-frames the preference objective so the policy itself is optimized directly against the paired data, removing the reward-model training and RL loop that make RLHF expensive.
After deploying a preference-tuned model, your win-rate on the preference benchmark has jumped 12 points, but users are reporting that the model now occasionally refuses straightforward factual questions it used to answer correctly. Name TWO specific things you should monitor or test to catch this kind of regression before it reaches users.
Win-rate on preference data can mask capability regressions; dedicated held-out checks and capability benchmarks are needed to surface them.