Cost
How much VRAM does fine-tuning actually need?
How much GPU memory do I need to fine-tune a model?
For half-precision LoRA, budget roughly two to two and a half gigabytes per billion parameters. For four-bit QLoRA, closer to one and a half. For full fine-tuning, around sixteen. Preference tuning roughly doubles whichever figure applies, because a frozen reference model is held alongside.
Four things compete for the same memory
This is why an adapter method saves so much: it collapses two of the four terms almost entirely. It is also why a run that fitted yesterday fails today when someone raised the sequence length.
- Weights — the model itself. Two bytes per parameter in half precision, half a byte in four-bit.
- Gradients — one value per trainable parameter. Near zero for adapters, enormous for full fine-tuning.
- Optimiser state — several values per trainable parameter. The dominant term in full fine-tuning and negligible for adapters.
- Activations — intermediate values held for the backward pass. Scales with batch size and sequence length, and it is the one that surprises people.
Rough figures per billion parameters
These are working approximations, not guarantees. The exact figure depends on sequence length, batch size and architecture, which is why the model catalogue publishes a specific admission threshold per model rather than a formula.
| Approach | Roughly, per 1B params | 8B model | 70B model |
|---|---|---|---|
| LoRA, half precision | ~2.2 GB | ~18 GB | ~140 GB |
| QLoRA, four-bit | ~1.7 GB | ~14 GB | ~48 GB |
| Full fine-tuning | ~16 GB | ~128 GB | Multi-node |
| Inference, half precision | ~2 GB | ~16 GB | ~140 GB |
Preference tuning doubles it
Direct preference optimisation holds a frozen reference copy of the model alongside the one being trained, to measure and penalise drift. Two models resident means roughly twice the memory.
This catches people out because supervised training on the same model fitted comfortably. A 14B pair that needs 28 GB for supervised training needs a 48 GB class for preference tuning.
Making a run fit without paying more
Before moving to a larger class, three levers are usually available and are worth trying in this order.
- Reduce sequence length to something the data actually needs. Setting it to the model maximum by default is the most common source of waste — check the token length percentiles in the validation report first.
- Turn on gradient checkpointing. It discards activations and recomputes them, cutting the largest variable term for roughly a third more time.
- Use gradient accumulation. Several small batches summed into one update give the same effective batch size at a fraction of the peak memory.
Together these will usually bring a run back inside a class. Whether that is cheaper than the next class up depends on the rate difference against the time penalty, and it is an arithmetic question rather than a judgement one.
Frequently asked questions
Why does training need so much more than inference?
Inference holds weights and a cache. Training additionally holds gradients, optimiser state and activations for the backward pass. For full fine-tuning those extra terms dominate.
Does a longer context need more memory at inference too?
Yes — the attention cache grows with sequence length and concurrency. At long context and high concurrency, that cache can exhaust memory with the weights comfortably resident.
Terms used here
More on cost
Last verified 6 August 2026.
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.