Training
Gradient checkpointing
Also called Activation checkpointing.
What is gradient checkpointing?
Gradient checkpointing reduces memory by discarding intermediate activations during the forward pass and recomputing them during the backward pass. It typically cuts activation memory substantially in exchange for roughly a third more compute time.
Activations — the intermediate values at every layer — dominate memory during training at long sequence lengths, often exceeding the weights themselves. Recomputing rather than storing them is a direct and predictable trade.
It combines with gradient accumulation, and together they will usually bring a model that overflows a GPU class back inside it. Whether that is cheaper than the next class up depends on the rate difference against the time penalty.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.