LoRA, QLoRA or full fine-tuning: how to choose

All three change model behaviour. They differ in how many parameters they train and at what precision, and everything else follows from that.

Full fine-tuning updates every weight, which means optimiser state for every weight — several times the size of the parameters themselves. LoRA trains small injected matrices and freezes the rest, so optimiser state exists only for a fraction of a per cent of the model. QLoRA does the same thing with the frozen base stored in four bits instead of sixteen.

Check whether the model fits half-precision LoRA on the GPU class you want. If it does, use LoRA. If it does not, use QLoRA on that class or LoRA on a larger one, and pick whichever is cheaper for the run length.

That second comparison is worth doing rather than assuming. QLoRA is slower per step because weights are dequantised on the fly, so a cheaper class at lower throughput sometimes costs more overall than a dearer class that finishes sooner.

Rarely, and the honest cases are narrow: a new language, an output format unlike anything the model has seen, a domain whose vocabulary barely overlaps with the pretraining corpus. In those, the adapter’s limited capacity genuinely binds.

For ordinary task adaptation — format, tone, classification, extraction, following your conventions — the measured difference against a well-configured adapter is small, and the memory difference is an order of magnitude.

Full fine-tuning also produces a complete model rather than a small file, which means it cannot share a base with siblings at serving time. If you expect to run several variants, that alone often settles it.

The instinct is to raise the rank. Usually the problem is data: too few examples, or inconsistent ones. Raising rank adds capacity to memorise a small dataset, which is the opposite of what is needed.

Extending the adapter to the feed-forward layers rather than attention alone is a better second move, because that is where more of the model’s substance sits.