Evaluation
Statistical significance
What is statistical significance?
Statistical significance is the judgement that an observed difference is unlikely to be an artefact of sampling. In model evaluation it is what separates a genuine improvement from noise, and small evaluation sets rarely have enough power to establish it.
A hundred-case evaluation cannot reliably detect a few percentage points of difference. Teams routinely deploy on differences their evaluation set was never capable of measuring.
Significance is also not importance. A difference can be statistically real and operationally irrelevant, and a gate should encode which magnitudes are worth acting on rather than treating any detectable difference as a reason to ship.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.