Supervised fine-tuning
Replacing a frontier API on one task
How do I cut my LLM bill by replacing a frontier model with a fine-tuned open one?
Capture real production traffic, fine-tune a small model on it, then gate the candidate against the model it would replace on your own cases. Most single-task workloads move to an 8B model with no measurable quality loss and a large reduction in per-request cost.
| Objective | Supervised fine-tuning |
|---|---|
| Dataset shape | Your production prompts and the frontier model’s responses, filtered to the ones that were actually good. |
| Metric that decides it | Blinded win rate against the incumbent on held-out production cases, plus cost per thousand requests. |
| Cheapest starting point | Llama 3.1 8B on RTX 4000 Ada$0.09 per GPU-hour |
This is the most common reason people arrive here, and it works more often than it does not — but only on narrow tasks. A model doing one job on one distribution can be shrunk dramatically. A model doing everything cannot, and no amount of fine-tuning changes that.
The step teams skip is filtering. Your frontier model was not right every time, and training on its full output teaches the small model to reproduce its mistakes along with its successes. Filter to responses that were accepted, unedited, or otherwise known good before training.
Then gate it. The evaluation gate exists exactly for this decision: the candidate is scored against the incumbent, blinded, on held-out cases from your own traffic. A pass is permission to deploy. A fail is a cheap answer to an expensive question.
The mistake to avoid
Evaluating on the training distribution and declaring victory. The frontier model was handling your long tail invisibly; a small model will not, and you will find out in production rather than in the gate.
Models to start from
In order. Start at the top and move down the list only if the evaluation gate says you have to — the smallest model that clears your bar is the one you pay for on every request forever.
Before you deploy it
Whatever the metric above says in isolation, the deployment decision is comparative: is this better than what is running today? Gate the candidate against the incumbent, blinded, on held-out cases from your own traffic. A fail is a cheap answer to an expensive question.
Frequently asked questions
How much traffic do I need to capture?
A few thousand well-filtered request and response pairs is enough for a single narrow task. Coverage of the input distribution matters far more than raw count.
What if the fine-tune loses to the incumbent?
Then you have spent a few dollars to learn that the task is not shrinkable yet, which is the cheapest possible way to learn it. Usually the answer is more or better data rather than a larger model.
Start with Llama 3.1 8B
Upload your dataset, read the validation report, and forecast the run before anything is leased.