Training
Catastrophic forgetting
What is catastrophic forgetting?
Catastrophic forgetting is the loss of previously learned general capability when a model is trained heavily on a narrow task. The model gets better at what you trained it on and measurably worse at everything else, often without any warning in the training metrics.
Training loss will look excellent throughout. Loss measures performance on the new data only, so a model quietly losing its general ability shows no sign of it in the one number most people watch.
Parameter-efficient methods reduce the risk substantially because the original weights are frozen — the base capability is still physically there. Full fine-tuning at a high learning rate over many epochs is where it appears most severely.
The defence is an evaluation set that includes tasks you are not training on. If general capability is not measured, its loss is invisible until a user finds it.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.