Evaluation
Held-out set
Also called Test set.
What is held-out set?
A held-out set is data deliberately excluded from training and reserved for final measurement. It is the only honest estimate of how a model will perform on data it has never seen, and it must not be used to make tuning decisions.
The distinction from a validation split matters. Validation guides decisions during development and is therefore gradually fitted to, however careful you are. The held-out set is touched once, at the end, precisely so that it stays uncontaminated.
It should look like production. A held-out set drawn from the same clean, curated pool as the training data will overstate performance, because production contains the messy cases that were filtered out.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.