Evaluation
Schema validity
What is schema validity?
Schema validity is the proportion of model outputs that parse and conform to the expected structure. It is measured separately from correctness of content, because a valid envelope with wrong values and a broken envelope are different failures with different fixes.
It is one of the metrics that improves most reliably with fine-tuning, often from the low nineties to above ninety-nine per cent, which is usually the difference between needing a repair layer and not.
Track it separately per field. Aggregate validity hides the one optional field that is malformed a fifth of the time.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.