Pillar
It ships only if it wins.
What is the Onrup evaluation gate?
A blocking check between training and deployment. The candidate and the model it would replace answer the same held-out cases under randomised aliases, a judge scores them blinded, and the result is bootstrapped for a confidence interval. The verdict is pass, fail or inconclusive, and only pass unlocks a deploy or a publish.
It never asks whether a model is good
It asks whether the model is better than the one it would replace, which is the only question a deployment decision actually turns on. An absolute score is unanchored: knowing a model scores 0.82 on your rubric tells you nothing about whether to ship it.
Knowing it beats the incumbent on 58 per cent of held-out cases, with an interval that excludes break-even and no critical case regressed, tells you everything. That is a decision rather than a data point.
Blinding is not optional
Both models answer the same cases under randomised aliases, and presentation order is randomised too. Judges favour whichever output they believe is the new one, and this applies to model judges as much as human ones. Both also show position preferences, which get recorded as quality if you do not control for them.
Case order is seeded, so rerunning an evaluation produces the same sequence and two results are comparable rather than merely similar.
Critical cases have a veto
Averages hide the failures that matter most. A model that improves on ninety-nine per cent of cases and breaks the one that caused an outage last year has an excellent aggregate score and must not ship.
So critical cases — safety refusals, compliance requirements, known past incidents — are evaluated separately with binary outcomes, and a regression on any of them fails the gate regardless of the aggregate. This set grows every time something goes wrong, which is the right growth mechanism for it.
What gets measured
- Deterministic task metrics, wherever the task has a checkable answer.
- Schema and tool-call validity, reported separately from content correctness — a valid envelope with wrong values is a different failure from a broken envelope.
- Critical cases, with veto power over the aggregate.
- A blinded model-graded rubric for qualities that resist automatic checking.
- Latency, output length and estimated serving cost, because a better model that is twice as slow is a trade rather than a win.
Running it
curl https://api.onrup.com/v1/evaluations \
-H "Authorization: Bearer $ONRUP_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"candidate_run_id": "run_...",
"baseline": {"kind": "deployment", "id": "dep_..."},
"evaluation_asset_id": "asset_...",
"max_cost_usd": 8.00
}'The cost ceiling is declared before the job starts and the job stops at it rather than exceeding it. The verdict is available on its own route, so a deployment pipeline can branch on it without parsing a report.
Designing the set it runs against
The gate is only as good as the cases it runs. Most evaluation sets are too small to support the decisions made from them, and most are drawn from cleaner data than production contains.
Frequently asked questions
What are the three verdicts?
Pass, fail and inconclusive. Pass means the candidate beat the baseline by enough, with a confidence interval that excludes break-even, and no critical case regressed. Fail means it lost or a critical case broke. Inconclusive means the difference sat inside the noise, which is genuinely different from a fail and is not rounded to one.
Can a person override a failed gate?
They can approve a new evaluation suite. They cannot mark a recorded failure as passed — the verdict and the run that produced it are written to an append-only record. A gate somebody can wave through at the end of a long week is a dashboard.
What if I do not have a baseline yet?
The base model you fine-tuned from is the natural first baseline. It answers the question that matters at that point: did the fine-tune achieve anything at all?
Why does it keep returning inconclusive?
Almost always because the evaluation set is too small, or too easy to separate the two models. That is information about your evaluation rather than about your model.
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.