Migrating from a closed model API to an open-weight fine-tune

This works on tasks doing one job on one distribution. It does not work on a model doing everything, and no amount of training data changes that.

The practical test: can you describe what the model does in one sentence without using the word "and"? If not, split it into separate models and migrate them one at a time.

Your existing model was not right every time. Training on its complete output teaches the small model to reproduce its mistakes alongside its successes, and those mistakes then become permanent rather than occasional.

Filter to responses that were accepted, unedited, acted on, or otherwise known good. Wherever a downstream outcome signal exists, use it — it is a better filter than reading the examples.

That is where most single-task replacements land. Below it, quality often drops enough to notice; above it, the cost advantage that motivated the migration starts eroding.

Move down afterwards, not up. Once the 8B passes the gate, try the same dataset at 4B and then smaller — the gate makes that a measurement rather than a guess, and the saving is on every request forever.

Set the closed model as the baseline and run the comparison blinded on held-out cases from your own traffic. This is precisely the decision the gate exists for, and it turns "we think this is as good" into a win rate with an interval.

A fail is a cheap answer. It usually points at data — more coverage, better filtering — rather than at the idea being wrong.

You gain weights you hold, an endpoint you control, and a cost structure that is a GPU-hour rate rather than a token rate — materially cheaper at volume and much easier to forecast. The long system prompt disappears into the weights, and at high volume that prompt was usually where the money went.

You give up a very strong general model and a two-call API where nobody thinks about VRAM. If you have no portability requirement and no cost pressure, staying is a perfectly reasonable decision.