Training
Learning rate
What is learning rate?
The learning rate controls how large a step the optimiser takes on each update. It is the single most consequential hyperparameter in fine-tuning: too high destabilises training and erodes existing capability, too low means the model barely changes.
Fine-tuning uses much lower rates than pretraining, because the model is being adjusted rather than built. Adapter-based methods tolerate higher rates than full fine-tuning, since the frozen base cannot be damaged by a large step.
A warmup period at the start, ramping from near zero, avoids a large early step wrecking the model before the optimiser has any sense of the gradient landscape. A decay schedule afterwards lets it settle rather than oscillate.
If a run diverges — loss spiking or becoming undefined — the learning rate is the first thing to halve.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.