Supervised fine-tuning
Semantic routing between models
How do I route requests between a small and a large model automatically?
Fine-tune a very small model to predict whether the large one is needed. A 0.6B router adds a few milliseconds and can move a large share of traffic to a cheaper model without a quality change anyone notices.
| Objective | Supervised fine-tuning |
|---|---|
| Dataset shape | Request in, a binary decision out, labelled by whether the small model’s answer was actually acceptable. |
| Metric that decides it | Cost saved per thousand requests, against the rate of requests wrongly kept from the large model. |
| Cheapest starting point | Qwen3 0.6B on RTX 3080$0.09 per GPU-hour |
Most production traffic is easy. A minority is hard. Sending everything to the model sized for the hard minority is the single most common source of avoidable inference spend, and it persists because nobody wants to hand-write the routing rules.
Label the training data by outcome rather than by intuition. Run both models over historical traffic, judge whether the small one’s answer was acceptable, and train the router on that judgement. The router then encodes the real boundary rather than a guessed one.
Keep the router asymmetric. Escalating an easy request to the large model costs money; failing to escalate a hard one costs quality. Bias the threshold toward escalation and tune it against measured harm.
The mistake to avoid
Routing on surface features like input length. It correlates weakly with difficulty and fails exactly on the short, hard requests where escalation matters most.
Models to start from
In order. Start at the top and move down the list only if the evaluation gate says you have to — the smallest model that clears your bar is the one you pay for on every request forever.
Before you deploy it
Whatever the metric above says in isolation, the deployment decision is comparative: is this better than what is running today? Gate the candidate against the incumbent, blinded, on held-out cases from your own traffic. A fail is a cheap answer to an expensive question.
Frequently asked questions
Does the router add noticeable latency?
A 0.6B model producing a single-token decision runs in single-digit milliseconds on an entry class. Against a large-model call it is not measurable.
Can the router itself be the small model?
It can, and it saves a hop, but it makes the routing decision much harder to measure and change independently. Keep them separate until the numbers are stable.
Start with Qwen3 0.6B
Upload your dataset, read the validation report, and forecast the run before anything is leased.