Docs
Evaluation and the gate
How does the Onrup evaluation gate decide whether a model can be deployed?
The candidate and the baseline answer the same held-out cases under randomised aliases, a judge scores them blinded, and the result is bootstrapped for a confidence interval. The verdict is pass, fail or inconclusive; only pass unlocks a deploy or publish.
Comparative, not absolute
The gate never asks whether a model is good. It asks whether it is better than the one it would replace, which is the only question a deployment decision actually turns on.
An absolute score is unanchored — knowing a model scores 0.82 on your rubric tells you nothing about whether to ship it. Knowing it beats the incumbent on 58 per cent of held-out cases, with an interval that excludes break-even, tells you everything.
Blinding, and why it is not optional
Both models answer the same cases under randomised aliases, and presentation order is randomised too. Judges — human and model alike — favour whichever output they believe is new, and both show position preferences that otherwise get attributed to quality.
Case order is seeded, so a rerun of the same evaluation produces the same ordering and the results are comparable.
Three verdicts
Inconclusive usually means the evaluation set is too small or too easy to separate the two models. That is information about your evaluation, not about your model.
- Pass — the candidate beat the baseline by enough, with an interval that excludes break-even, and no critical case regressed.
- Fail — the candidate lost, or a critical case regressed. A critical regression fails the gate regardless of the aggregate.
- Inconclusive — the difference sits inside the noise. This is genuinely different from a fail and is not rounded to one.
What is measured
- Deterministic task metrics, where the task has a checkable answer.
- Schema and tool-call validity, reported separately from content correctness.
- Critical cases — safety refusals, compliance requirements, past incidents — with veto power over the aggregate.
- A blinded model-graded rubric for qualities that resist automatic checking.
- Latency, output length and estimated serving cost, because a better model that is twice as slow is a trade rather than a win.
A human cannot quietly overturn it
A person can approve a new evaluation suite. What they cannot do is mark a recorded failure as passed, because the verdict and the run that produced it are written to an append-only record.
That constraint is the whole value. A gate somebody can wave through at the end of a long week is a dashboard.
Frequently asked questions
What if I do not have a baseline?
The base model you fine-tuned from is the natural first baseline. It answers the question that matters at that point: did the fine-tune achieve anything?
How many evaluation cases do I need?
Enough that the confidence interval is narrower than the difference you care about. A hundred cases cannot reliably detect a few percentage points; if your verdicts keep coming back inconclusive, that is the reason.
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.