Training
Gradient
What is gradient?
A gradient is the derivative of the loss with respect to each trainable parameter — the direction and magnitude in which changing that parameter would most reduce the error. Training is repeated small movement in the direction the gradients indicate.
Gradients are computed by backpropagation, which walks the network backwards applying the chain rule. This backward pass is roughly twice the cost of the forward pass, which is why inference is much cheaper than training per token.
Gradient clipping caps the magnitude of an update to prevent one anomalous batch from taking a destructively large step. It is cheap insurance and is on by default in most practical setups.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.