---
title: Designing an evaluation set that actually decides things
description: Sizing, sampling and the critical-case set — plus why most evaluation sets are too small to support the decisions made from them.
url: https://www.onrup.com/guides/designing-an-evaluation-set
category: Evaluation
published: 2026-08-06
updated: 2026-08-06
source: Onrup
---

# Designing an evaluation set that actually decides things

**How do I build an evaluation set for a fine-tuned model?**

Draw cases from real production traffic including the awkward ones, size it so the confidence interval is narrower than the difference you care about, and keep a separate small set of critical cases with veto power over the aggregate.

## Sample from production, not from the clean pool

An evaluation set drawn from the same curated data as your training set will overstate performance, because production contains the malformed inputs, the ambiguous requests and the long tail that got filtered out during preparation.

The cases that matter most are the ones where the current system already struggles. Those are also the ones people are least inclined to include, because they make the numbers worse.

## Size it for the difference you care about

A hundred cases cannot reliably detect a few percentage points of difference. Teams routinely deploy on differences their evaluation set was never capable of measuring, and then wonder why production disagrees.

The practical test is to bootstrap: resample your results, recompute the metric, and look at the spread. If the interval spans break-even, the evaluation did not resolve the question — which is what an inconclusive verdict is telling you.

## Keep a critical set separate

Aggregates hide the failures that matter most. A model that improves on ninety-nine per cent of cases and breaks the one that caused an outage last year has an excellent score and must not ship.

Maintain a small, explicit set of cases with binary outcomes — safety refusals, compliance requirements, known past incidents — evaluated separately and given veto power. This set grows every time something goes wrong, which is the right growth mechanism.

## Grade the right things separately

- Schema validity and content correctness. A valid envelope with wrong values and a broken envelope are different problems with different fixes.
- Per-category recall rather than one aggregate, wherever the categories have different costs of failure.
- Latency and output length alongside quality. A better model that is twice as slow is a trade, not a win.

## Blind it, or do not bother

Judges favour whichever output they believe is new, and this applies to model judges as much as human ones. Randomise the aliases and randomise the presentation order — position bias is real and will otherwise be recorded as quality.

Seed the case ordering so a rerun produces the same sequence and two results are comparable.

## Frequently asked questions

### How many cases is enough?

Enough that the confidence interval is narrower than the difference you would act on. That is usually several hundred for a few-percentage-point difference, and more if the metric is noisy.

### Can I reuse the same evaluation set forever?

For a while. It gradually gets fitted to as you make decisions from it, and it stops representing production as your traffic changes. Refresh the sample periodically and keep the critical set growing.