Training
Training step
What is training step?
A training step is one weight update: a forward pass over a batch, a backward pass to compute gradients, and an application of those gradients by the optimiser. Total steps is the training budget, and cost is very nearly linear in it.
Steps rather than epochs is the more useful unit for reasoning about cost, because it is what the hardware actually does. Total steps is roughly the dataset size divided by the effective batch size, multiplied by epochs.
Throughput in steps per second is the number that determines wall time, and it varies with sequence length, batch size, model size and whether memory-saving techniques are in use.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.