Evaluation
Confidence interval
What is confidence interval?
A confidence interval expresses the uncertainty around a measured result given the sample size. For model evaluation it converts a bare win rate into a statement about what the true rate plausibly is, which is what a deployment decision actually needs.
Bootstrapping is the usual method: resample the evaluation results many times with replacement, recompute the metric each time, and take the spread of those values as the interval. It makes no distributional assumptions and is easy to implement correctly.
When the interval spans the break-even point, the honest conclusion is that the evaluation did not resolve the question. That is the situation an "inconclusive" verdict exists to represent.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.