Training
When to use preference tuning instead of supervised fine-tuning
When should I use DPO instead of supervised fine-tuning?
Use preference tuning when you can recognise a better answer but cannot write the ideal one — voice, tone, helpfulness, which of two plausible responses is preferable. Use supervised fine-tuning whenever you can specify the target directly, and usually do it first.
The distinguishing question
Can you write down the ideal answer? If yes, supervised fine-tuning is simpler, cheaper and needs half the memory. If you can only say which of two answers is better, that is what preference tuning is for.
Brand voice is the canonical case. Guidelines say things like "confident but not arrogant", which a model cannot act on and a reviewer recognises instantly. You never have to define good — only to keep choosing it.
Do supervised training first
The usual sequence is supervised fine-tuning to establish format and basic competence, then preference tuning to refine judgement on top of it.
Preference tuning applied to a model that has not yet learned the format spends its capacity teaching structure rather than preference, which is an expensive way to do a cheap job.
The pairs probably already exist
Any workflow where somebody chose one draft over another has been producing preference pairs for years without labelling them as such. Editorial review, A/B copy tests, support macro selection, code review between two proposed fixes.
What does not work is manufacturing pairs by degrading a good answer. The model learns to detect the degradation rather than to prefer the voice, and it will score beautifully in evaluation and behave generically in production.
Budget the memory
Preference tuning holds a frozen reference copy of the model alongside the one being trained, to measure and penalise drift. That roughly doubles the memory requirement.
The practical consequence is that a model which fitted supervised training on a mainstream card needs the next class up. It also puts preference tuning out of reach on the largest models entirely.
How many pairs
One to three thousand genuine pairs. Preference tuning is more sensitive to pair quality than to volume, and a thousand honest comparisons beat ten thousand synthetic ones.
The two responses should be plausible alternatives. A comparison between a good answer and an obviously broken one carries almost no information about what makes the good one good.
Frequently asked questions
Do I need a reward model?
Not for direct preference optimisation — that is the point of it. The preference is optimised directly from the pairs, without training an intermediate model first.
Can I use preference tuning for correctness?
You can, but if correctness is checkable a programmatic reward is better: it is objective, cheaper and much harder to game than a learned preference.
Terms used here
More on training
LoRA, QLoRA or full fine-tuning: how to choose
What is the difference between LoRA, QLoRA and full fine-tuning?
Why fine-tuned models fail, and how to tell which failure you have
Why did my fine-tuned model get worse instead of better?
Using reinforcement learning to improve reasoning
How does GRPO work and when should I use reinforcement learning to fine-tune?
Last verified 6 August 2026.
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.