Training
Warmup
What is warmup?
Warmup is a short period at the start of training during which the learning rate ramps from near zero to its full value. It prevents a large, badly-directed early update from damaging the model before the optimiser has accumulated any useful gradient statistics.
Adaptive optimisers estimate gradient moments from history. At step one there is no history, so the estimates are poor and a full-size step taken on them is close to random.
A few per cent of total steps is usually enough. Excessive warmup on a short fine-tuning run simply wastes a meaningful fraction of the training budget at an ineffective learning rate.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.