DPO and Preference Optimization.

Picture two ways of teaching someone to write better essays. First way: hire a grader. Train the grader to score essays on a 1-to-10 scale by showing it many essays and many human-judgment scores. Then hand the grader to a student and tell the student to write essays the grader will score highly. The student writes a draft, the grader scores it, the student adjusts, the grader scores again. Over thousands of rounds, the student learns to write essays the grader likes — which, if the grader was trained well, are essays humans would like too.

That is RLHF. The grader is the reward model. The student is the policy. The rounds of "write, score, adjust" are on-policy rollouts. The whole loop has two separate training stages, two models in memory, and a feedback channel between them that has to be carefully constrained so the student does not learn to game the grader.

Now the second way. Skip the grader entirely. Hand the student pairs of essays — one labeled "better," one labeled "worse" — and tell the student: learn to assign higher probability to the better essay than to the worse one. No grader, no scores, no rollouts. The student reads pairs and adjusts.

That is Direct Preference Optimization. The reward model has not gone away exactly. It has been folded into the loss function itself, where it acts as an implicit signal that never gets written down as a separate model. The clever trick is that the math of RLHF with a KL constraint has a known closed-form solution, and once you write the closed-form out, you can train against it with a single classification-style loss on preference pairs. The reward model is in there. It is hiding inside the policy.

I keep coming back to a mental picture for this. The reward model in RLHF is a separate ledger you maintain alongside your policy. DPO does not throw the ledger away. It writes the ledger entries directly into the policy's margin notes, then rips the cover off. Same accounting, fewer books.

The rest of the chapter is about understanding why that works. The mechanism is not magic. It is a derivation that takes about ten minutes if you go through it carefully, and the derivation is the entire chapter. Once you walk through it once, every other preference-optimization method (KTO, ORPO, SimPO, IPO) is a small variant on the same idea.

Most explanations of DPO I read while drafting this chapter were honestly terrible. They open with the loss function and motivate it backwards from there, which is exactly the part that does not teach you anything. The loss is two lines of code. The derivation is the chapter.

A few terms before we go further.

A preference pair is a triple (prompt, chosen, rejected). The prompt is something a user might ask, like "Explain how photosynthesis works." Chosen is the response a labeler preferred. Rejected is the one they did not. Datasets come from human annotators clicking thumbs-up or thumbs-down on pairs of model outputs, or from synthetic comparisons generated by stronger models. The two well-known public datasets are Anthropic's HH-RLHF (Helpful and Harmless, ~170,000 pairs) and UltraFeedback from OpenBMB (~64,000 pairs scored by GPT-4).

A reference policy is a frozen copy of your model from before alignment. In the standard recipe, this is your Project 21: Fine-Tuning and Instruction Tuning SFT model. The reference stays fixed throughout DPO training. The model you are training is the policy, and DPO compares the policy's probabilities against the reference's for every preference pair.

The KL constraint is a leash. The trained policy should not drift too far from the reference. Without this leash, RL-style training pushes the model to extreme outputs that score high on whatever signal you provided, even when those outputs are gibberish. D_KL measures the distance between the two distributions. KL of zero means identical; large KL means they have diverged.

The implicit reward is the term doing the heavy lifting in DPO. Not a separate model. It is an algebraic expression, log pi(y|x) - log pi_ref(y|x) scaled by a constant, that the math tells us behaves exactly like a reward function would, given certain assumptions.

Beta (β) controls how tight the KL leash is. Large beta keeps the policy close to the reference. Small beta lets it drift. Tuning beta is most of the practical work of getting DPO to behave well; we look at it in Step 8.

The sigmoid function is σ(z) = 1 / (1 + exp(-z)). It squashes any real number into (0, 1). DPO turns "did the policy prefer chosen over rejected?" into a sigmoid of a difference, then maximizes the log of that probability. If you have used logistic regression, this will look familiar: DPO is logistic regression on preference pairs, with a feature engineering choice that connects it back to RL.

With those terms in place, here is the one-paragraph version of why DPO works. RLHF wants the policy that maximizes expected reward subject to a KL penalty against the reference. That problem has a known closed-form optimum: the optimal policy is proportional to the reference times the exponential of the reward, divided by beta. Solve that for the reward, and you get a formula for reward in terms of the policy and the reference. Substitute it back into the Bradley-Terry preference model (the probability that humans prefer A over B is the sigmoid of the reward difference) and the reward model disappears. What is left is a loss function written entirely in terms of the policy and the reference. That loss is the DPO loss. Train against it with ordinary supervised methods and you have, in effect, run RLHF.

If that paragraph went past quickly, do not worry. We walk through it slowly in the build section, equation by equation.

The RLHF pipeline you built in Project 23: Reward Models and RLHF works, but it is not a system you want to maintain. Too many parts can fail independently. Your reward model can be miscalibrated, your policy can collapse onto reward-hacking exploits, your value function can diverge from the policy, your KL coefficient can be too tight or too loose. The on-policy sampling step requires generation infrastructure that doubles your compute budget relative to plain fine-tuning. And if anything goes wrong, you have four candidate culprits and a feedback loop between them that makes attribution hard.

This is a familiar shape to anyone who has run distributed production systems. In a past role on a multi-state SaaS platform, we kept a cloud-warehouse migration at 99.99% across hundreds of thousands of customer records, and the rule that kept it there was the same: every additional moving piece is another thing that fails in the middle of the night. Most of the engineering value of DPO is exactly that subtraction.

DPO removes most of those failure modes by removing most of those parts. No reward model to train and miscalibrate. No on-policy sampling, so the policy only sees the labeled pairs from disk. No value function, no separate reward-model checkpoint. The loss is one forward pass through the policy, one through the frozen reference, and a tiny algebraic combination of the four log-probabilities involved. You can implement the whole thing in about fifteen lines of PyTorch.

The first time I ran a DPO loop end-to-end, I checked the loss curve three times before believing it. It looked like SFT. It is, structurally, SFT on a folded log-ratio target, and the moment that clicked the chapter wrote itself.

On most public benchmarks the results are roughly comparable to RLHF. Sometimes DPO wins by a few points, sometimes RLHF wins. The numbers move with dataset, model size, and evaluation methodology. What does not move is the engineering reality: DPO is much easier to ship. You get most of the value at a fraction of the system complexity.

There is a second reason this chapter sits where it does. Alignment research has produced a small zoo of preference-optimization methods, and they all share the same skeleton: preference data, a frozen reference (or sometimes not), a sigmoid loss, a beta-style temperature. KTO drops pairs and works with single labels. ORPO folds in SFT. SimPO drops the reference policy entirely. IPO swaps the sigmoid for a different shape more stable when one side of the pair is much more likely than the other. Once you understand DPO, you understand the family. The chapter ends with a comparison table that lets you pick the right method.

There is a third reason. DPO has a failure mode that RLHF does not, and it deserves respect. With no reward model in the loop, the labels in your preference dataset are the only signal. If those labels are wrong, the model learns the wrong thing as faithfully as it would have learned the right thing. RLHF has a similar problem, but the reward model adds a layer of generalization that absorbs some label noise. DPO has no such layer. The BREAK IT experiment makes this concrete, and it is the moment I tell readers to slow down. The first time I ran an inverted-label sweep, I went looking for the bug for an hour before I accepted there wasn't one. The training was healthy. The labels were the bug.