Training
Gradient accumulation
What is gradient accumulation?
Gradient accumulation simulates a large batch on limited memory by processing several small batches, summing their gradients, and applying a single weight update at the end. The effective batch size is the small batch multiplied by the number of accumulation steps.
It trades time for memory, and does so almost exactly: four accumulation steps take roughly four times as long as one and use roughly a quarter of the peak activation memory.
This is the standard way a model that "does not fit" is made to fit. Before moving to a larger and more expensive GPU class, it is usually worth checking whether accumulation solves the problem at a lower hourly rate.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.