Training
Optimiser state
What is optimiser state?
Optimiser state is the additional data an optimiser maintains per trainable parameter — typically running estimates of gradient mean and variance. For a standard adaptive optimiser it occupies several times the memory of the parameters themselves.
This is the reason full fine-tuning is so much more expensive than the parameter count suggests. A model whose weights occupy sixteen gigabytes may need well over a hundred once optimiser state, gradients and activations are all resident.
Parameter-efficient methods collapse this cost because optimiser state is only kept for the small number of trained parameters. It is the single largest source of their memory advantage.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.