Supervised fine-tuning
Summarisation in a house style
How do I fine-tune a model to summarise in our specific format?
Train on a few hundred summaries your team actually approved. Style, length discipline and section structure are the most learnable things in any dataset, and they are the things prompts convey worst.
| Objective | Supervised fine-tuning |
|---|---|
| Dataset shape | Source document in, the approved summary out. Five hundred to two thousand pairs. |
| Metric that decides it | Editor acceptance rate without changes, and adherence to length and section constraints. |
| Cheapest starting point | Mistral Nemo 12B on RTX 4000 Ada$0.09 per GPU-hour |
Everybody has written the prompt: three bullets, no more than forty words each, no adjectives, always lead with the decision. It works about eighty per cent of the time, and the other twenty per cent is a person fixing it by hand — which is the cost you were trying to remove.
A few hundred approved examples move that reliably above ninety-five per cent, because format compliance is the single easiest thing to learn from examples. The prompt then shrinks to the document itself.
Long-context models earn their keep here. If the source is a forty-page report, a 128K-context model handles it in one pass and you avoid building a chunk-and-merge pipeline whose failure modes are worse than the ones you started with.
The mistake to avoid
Training on published summaries rather than approved-as-drafted ones. Published text has been through an editor, so you are teaching the model to produce output that still needs the editing step you wanted to remove.
Models to start from
In order. Start at the top and move down the list only if the evaluation gate says you have to — the smallest model that clears your bar is the one you pay for on every request forever.
Before you deploy it
Whatever the metric above says in isolation, the deployment decision is comparative: is this better than what is running today? Gate the candidate against the incumbent, blinded, on held-out cases from your own traffic. A fail is a cheap answer to an expensive question.
Frequently asked questions
How few examples can I get away with?
Format compliance starts improving around two hundred consistent examples. Content judgement needs considerably more.
Will it work across document types?
Only if your examples span them. A model trained solely on earnings reports will apply that structure to a customer interview.
Start with Mistral Nemo 12B
Upload your dataset, read the validation report, and forecast the run before anything is leased.