---
title: When to use preference tuning instead of supervised fine-tuning
description: The shape of problem preference tuning solves, where the pairs come from, and the memory cost people discover too late.
url: https://www.onrup.com/guides/when-to-use-preference-tuning
category: Training
published: 2026-08-06
updated: 2026-08-06
source: Onrup
---

# When to use preference tuning instead of supervised fine-tuning

**When should I use DPO instead of supervised fine-tuning?**

Use preference tuning when you can recognise a better answer but cannot write the ideal one — voice, tone, helpfulness, which of two plausible responses is preferable. Use supervised fine-tuning whenever you can specify the target directly, and usually do it first.

## The distinguishing question

Can you write down the ideal answer? If yes, supervised fine-tuning is simpler, cheaper and needs half the memory. If you can only say which of two answers is better, that is what preference tuning is for.

Brand voice is the canonical case. Guidelines say things like "confident but not arrogant", which a model cannot act on and a reviewer recognises instantly. You never have to define good — only to keep choosing it.

## Do supervised training first

The usual sequence is supervised fine-tuning to establish format and basic competence, then preference tuning to refine judgement on top of it.

Preference tuning applied to a model that has not yet learned the format spends its capacity teaching structure rather than preference, which is an expensive way to do a cheap job.

## The pairs probably already exist

Any workflow where somebody chose one draft over another has been producing preference pairs for years without labelling them as such. Editorial review, A/B copy tests, support macro selection, code review between two proposed fixes.

What does not work is manufacturing pairs by degrading a good answer. The model learns to detect the degradation rather than to prefer the voice, and it will score beautifully in evaluation and behave generically in production.

## Budget the memory

Preference tuning holds a frozen reference copy of the model alongside the one being trained, to measure and penalise drift. That roughly doubles the memory requirement.

The practical consequence is that a model which fitted supervised training on a mainstream card needs the next class up. It also puts preference tuning out of reach on the largest models entirely.

## How many pairs

One to three thousand genuine pairs. Preference tuning is more sensitive to pair quality than to volume, and a thousand honest comparisons beat ten thousand synthetic ones.

The two responses should be plausible alternatives. A comparison between a good answer and an obviously broken one carries almost no information about what makes the good one good.

## Frequently asked questions

### Do I need a reward model?

Not for direct preference optimisation — that is the point of it. The preference is optimised directly from the pairs, without training an intermediate model first.

### Can I use preference tuning for correctness?

You can, but if correctness is checkable a programmatic reward is better: it is objective, cheaper and much harder to game than a learned preference.