Preparing a fine-tuning dataset

The best training data usually exists before the project does. Resolved support tickets, approved summaries, merged review comments, analyst-written queries, translation memory, moderation decisions — all of these are records of the task done acceptably, labelled by outcome.

Writing examples specially for training is slower, more expensive, and produces a cleaner distribution than production will ever contain. Prefer harvesting to authoring wherever the option exists.

The single highest-leverage decision is what to exclude. Review comments that were ignored, first-assignment ticket labels that were later corrected, generated responses that a human edited — all of these look like valid training data and teach the model to reproduce mistakes.

Use an outcome signal wherever one exists: was it accepted, was it acted on, did it survive review. That signal is a better filter than any judgement made by reading the examples.

Two examples giving different answers to substantively the same question teach the model that the task is ambiguous, and it will hedge. This is why a small, coherent dataset routinely outperforms a large, mixed one.

It also means the model learns the distribution you show it including the parts you did not intend. If half your examples end with a follow-up question, expect the fine-tuned model to end half its answers with a follow-up question.

All four are cheap to run and all four are expensive to discover three hours into a training run.

Split the held-out set first, from the same distribution as production rather than from the cleaned pool. A held-out set drawn from curated data overstates performance because production contains the messy cases that got filtered.

Then do not touch it. Every decision made by looking at it fits the model to it a little more, and the number it eventually produces is the one you will quote to other people.