Training
Reference model
What is reference model?
A reference model is a frozen copy of the starting model, held during preference tuning to measure how far the trained model has drifted. It is what stops optimisation from destroying general capability in pursuit of the preference signal.
The training objective includes a penalty for diverging too far from the reference, which keeps the model recognisably itself while shifting its preferences.
It roughly doubles memory requirements, which is why preference tuning needs a larger GPU class than supervised training on the same model.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.