Training
Why fine-tuned models fail, and how to tell which failure you have
Why did my fine-tuned model get worse instead of better?
Almost always one of five causes: too many epochs, inconsistent data, silent truncation, a missing end-of-sequence token, or evaluating on contaminated data. Each has a distinctive symptom, and four of the five are visible before training starts.
The model repeats itself or will not stop
Symptom: correct answers that keep going, or output that loops until it hits the length limit.
Cause: the end-of-sequence token is missing from your training targets, so the model never learned where to stop. This is a conversion bug rather than a training one, and it is the single most common cause of a fine-tune that looks broken.
Answers stop mid-sentence
Symptom: responses that cut off partway through, consistently on longer inputs.
Cause: truncation. Your sequence length is shorter than your examples, and the end of the target was silently discarded during training. Check the token length percentiles in the validation report — the 95th percentile against your sequence length tells you immediately.
Excellent on training data, poor on anything else
Symptom: training loss beautifully low, held-out performance flat or worse.
Cause: overfitting, usually from too many epochs. Fine-tuning wants one to three; runs configured for ten are paying to memorise. Reduce epochs before touching anything else — it is the cheapest lever and the most commonly mis-set.
The model hedges, or is inconsistent
Symptom: vague answers, or the same question answered differently on different runs.
Cause: inconsistent training data. Two examples giving different answers to substantively the same question teach the model that the task is ambiguous, and hedging is the correct response to ambiguity.
This is a data problem and no hyperparameter fixes it.
Better at the task, worse at everything else
Symptom: the target metric improves and unrelated capability quietly degrades — including safety refusals.
Cause: catastrophic forgetting. It is invisible in training loss, which measures only the new data, so it is only detectable with an evaluation set covering tasks you are not training on.
Adapter methods reduce the risk substantially because the base weights are frozen. Full fine-tuning at a high learning rate over many epochs is where it appears worst.
Great evaluation, disappointing production
Symptom: the gate passed comfortably and users disagree.
Cause: usually contamination — evaluation data that overlapped with training data, often because deduplication ran after splitting rather than before. Sometimes the evaluation set was simply drawn from cleaner data than production contains.
Frequently asked questions
Which should I check first?
Truncation and the end-of-sequence token, because both are visible in the validation report before you spend anything, and between them they explain a large share of confusing failures.
The loss curve looked fine. Does that mean training worked?
No. Training loss measures fit to the training data only. A model losing general capability, overfitting, or learning from truncated targets can all produce a perfectly healthy-looking curve.
Terms used here
More on training
LoRA, QLoRA or full fine-tuning: how to choose
What is the difference between LoRA, QLoRA and full fine-tuning?
When to use preference tuning instead of supervised fine-tuning
When should I use DPO instead of supervised fine-tuning?
Using reinforcement learning to improve reasoning
How does GRPO work and when should I use reinforcement learning to fine-tune?
Last verified 6 August 2026.
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.