Preference tuning (DPO)
On-brand copy generation
How do I fine-tune a model to write in our brand voice?
Train on copy your brand team approved, then use preference tuning on pairs they ranked. Voice is a preference, not a rule, and preference tuning is the objective built for exactly that shape of feedback.
| Objective | Preference tuning (DPO) |
|---|---|
| Dataset shape | Prompt with a chosen and a rejected response. Your existing review process already generates these. |
| Metric that decides it | Blinded preference rate against the current model, judged by the people who own the brand. |
| Cheapest starting point | Mistral Small 3 24B on A40$0.42 per GPU-hour |
Brand voice resists specification. Guidelines say things like "confident but not arrogant", which a model cannot act on and a reviewer can recognise instantly. That gap is the case for preference tuning: you do not have to say what good is, only which of two options is better.
The data usually already exists. Any review process where someone picked one draft over another has been generating preference pairs for years without labelling them as such.
Budget for the memory. Preference tuning holds a frozen reference model alongside the one being trained, so a pair that would fit on a 24GB card for supervised training needs a 48GB class here.
The mistake to avoid
Constructing rejected examples artificially by degrading a good one. The model learns to detect the degradation rather than to prefer the voice, and it will look excellent in evaluation and generic in production.
Models to start from
In order. Start at the top and move down the list only if the evaluation gate says you have to — the smallest model that clears your bar is the one you pay for on every request forever.
Before you deploy it
Whatever the metric above says in isolation, the deployment decision is comparative: is this better than what is running today? Gate the candidate against the incumbent, blinded, on held-out cases from your own traffic. A fail is a cheap answer to an expensive question.
Frequently asked questions
Do I need supervised fine-tuning first?
Usually yes. Supervised training establishes the format and the basic competence; preference tuning refines the judgement on top of it.
How many preference pairs?
One to three thousand real ones. Preference tuning is more sensitive to pair quality than to volume — a thousand honest pairs beat ten thousand synthetic ones.
Start with Mistral Small 3 24B
Upload your dataset, read the validation report, and forecast the run before anything is leased.