Evaluation
Win rate
What is win rate?
Win rate is the proportion of evaluation cases where the candidate’s output was judged better than the baseline’s. It is the headline number of a comparative evaluation and is only meaningful with a confidence interval attached.
A win rate of 55 per cent on two hundred cases and the same figure on twenty thousand are very different claims. Without an interval, the number invites a decision the sample size does not support.
Ties should be reported rather than distributed. A high tie rate means the models are behaving similarly, which is important information — often it means the evaluation set is too easy to discriminate between them.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.