Training
Preference tuning
Also called DPO, Direct Preference Optimisation.
What is preference tuning?
Preference tuning trains a model on pairs of responses where one has been marked better than the other, rather than on a single correct answer. Direct Preference Optimisation is the common form: it optimises the preference directly, without training a separate reward model first.
It suits problems where quality is recognisable but not specifiable. Brand voice, helpfulness, tone and the general question of which of two plausible answers is better all fall into this category, and all resist being written down as a rule.
The data usually exists already. Any workflow where a person chose one draft over another has been producing preference pairs, and those real pairs are worth far more than synthetic ones constructed by deliberately degrading a good answer — a model trained on those learns to detect the degradation instead of the preference.
Budget extra memory. Preference tuning holds a frozen reference copy of the model alongside the one being trained, so the memory requirement is roughly double that of supervised training on the same model.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.