Training
LoRA
Also called Low-Rank Adaptation.
What is lora?
LoRA, or Low-Rank Adaptation, is a fine-tuning method that freezes the original model weights and trains small pairs of low-rank matrices injected alongside them. It typically trains under one per cent of the parameters while reaching close to full fine-tuning quality on most tasks.
The insight behind it is that the weight change induced by fine-tuning has low intrinsic rank — the adaptation a task requires can be expressed in far fewer dimensions than the model has. So instead of learning a full update matrix, LoRA learns two thin matrices whose product approximates it.
The practical consequences are large. Memory drops because optimiser state is only kept for the small matrices. The output is a file of megabytes rather than gigabytes. And because the base model is untouched, many adapters can share one copy of it in memory at serving time.
The main lever is the rank. Higher rank means more capacity to learn and more memory; ranks between 8 and 64 cover most tasks, and going higher rarely helps unless the task is genuinely far from the base model’s existing behaviour.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.