Training
Distillation
What is distillation?
Distillation trains a smaller model to reproduce the behaviour of a larger one. In practice it usually means generating outputs from the large model and fine-tuning the small one on them, which makes it a special case of supervised fine-tuning.
It works best on narrow tasks. Distilling general capability requires enormous coverage and usually disappoints; distilling one well-defined behaviour frequently succeeds.
The teacher’s mistakes are inherited along with its strengths, so filtering the generated data to known-good examples is the step that separates a good distillation from a small model that is confidently wrong.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.