Evaluation
Critical regression
What is critical regression?
A critical regression is a failure on a case the model must never get wrong — a safety refusal, a compliance requirement, a known past incident. It is treated as a blocking failure regardless of how well the model scores on average.
Averages hide these by construction. A model that improves on ninety-nine per cent of cases and breaks the one that once caused an outage has a superb aggregate score and must not ship.
The practical form is a small, explicitly maintained set of cases with binary pass or fail outcomes, evaluated separately from the aggregate metric and given veto power over it.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.