Supervised fine-tuning
Domain question answering
Should I fine-tune or use retrieval for questions about my own documents?
Use retrieval for the facts and fine-tuning for the behaviour. Fine-tuning teaches a model your vocabulary, your answer format and when to decline; it is a poor and expensive way to teach it facts that change.
| Objective | Supervised fine-tuning |
|---|---|
| Dataset shape | Question, retrieved context, and the answer a good analyst would give — including the refusals. |
| Metric that decides it | Answer quality graded against a rubric, and refusal accuracy on questions the context cannot answer. |
| Cheapest starting point | Granite 4.1 8B on RTX 4000 Ada$0.09 per GPU-hour |
The question is put as a choice far more often than it should be. Retrieval and fine-tuning solve different problems: retrieval decides what the model sees, fine-tuning decides what it does with it. A model that has never been tuned on your domain will use retrieved context in a generic way, and will answer confidently when the context does not support an answer.
Train on triples of question, retrieved context and ideal answer. Crucially, include cases where the correct answer is that the context does not contain one. Refusal is a learnable behaviour and it is the behaviour that determines whether the system can be trusted.
Granite models are worth a look here specifically because they are already tuned toward retrieval-augmented answering and toward declining rather than improvising.
The mistake to avoid
Fine-tuning to memorise facts. They go stale, you cannot update one without retraining, and the model will state the old version with total confidence.
Models to start from
In order. Start at the top and move down the list only if the evaluation gate says you have to — the smallest model that clears your bar is the one you pay for on every request forever.
Before you deploy it
Whatever the metric above says in isolation, the deployment decision is comparative: is this better than what is running today? Gate the candidate against the incumbent, blinded, on held-out cases from your own traffic. A fail is a cheap answer to an expensive question.
Frequently asked questions
Can I skip retrieval entirely if I fine-tune?
Only if your knowledge base is small, static and small enough to fit in the training set — which is rare. Anything that changes should be retrieved, not trained.
How do I teach it to say "I do not know"?
Include those cases in the training data explicitly, with the context that fails to support an answer and the refusal as the target. A model only learns to refuse if it has seen refusals.
Start with Granite 4.1 8B
Upload your dataset, read the validation report, and forecast the run before anything is leased.