--- title: Preparing a fine-tuning dataset description: From raw material to a validated dataset, including the four checks worth running before any compute is leased. url: https://www.onrup.com/guides/preparing-a-fine-tuning-dataset category: Preparing data published: 2026-08-06 updated: 2026-08-06 source: Onrup --- # Preparing a fine-tuning dataset **How do I prepare a dataset for fine-tuning?** Collect real examples of the task done well, make them consistent, deduplicate before splitting, and check the token length distribution against your sequence length. Consistency matters more than volume: a thousand coherent examples beat ten thousand contradictory ones. ## Start from work already done The best training data usually exists before the project does. Resolved support tickets, approved summaries, merged review comments, analyst-written queries, translation memory, moderation decisions — all of these are records of the task done acceptably, labelled by outcome. Writing examples specially for training is slower, more expensive, and produces a cleaner distribution than production will ever contain. Prefer harvesting to authoring wherever the option exists. ## Filter by outcome, not by appearance The single highest-leverage decision is what to exclude. Review comments that were ignored, first-assignment ticket labels that were later corrected, generated responses that a human edited — all of these look like valid training data and teach the model to reproduce mistakes. Use an outcome signal wherever one exists: was it accepted, was it acted on, did it survive review. That signal is a better filter than any judgement made by reading the examples. ## Consistency beats volume Two examples giving different answers to substantively the same question teach the model that the task is ambiguous, and it will hedge. This is why a small, coherent dataset routinely outperforms a large, mixed one. It also means the model learns the distribution you show it including the parts you did not intend. If half your examples end with a follow-up question, expect the fine-tuned model to end half its answers with a follow-up question. ## The four checks before you spend anything All four are cheap to run and all four are expensive to discover three hours into a training run. - Token length percentiles against your intended sequence length. The 95th percentile is the number to look at, not the mean. - Truncation count. Anything above about one per cent of targets being cut off is worth fixing first — a truncated target teaches the model to stop mid-answer, silently. - Duplicate rate, exact and near. Deduplicate before splitting, or duplicates span the training and held-out sets and your evaluation measures memorisation. - Class or category balance. A category with thirty examples will be handled badly however good the aggregate number looks. ## Hold out before you do anything else Split the held-out set first, from the same distribution as production rather than from the cleaned pool. A held-out set drawn from curated data overstates performance because production contains the messy cases that got filtered. Then do not touch it. Every decision made by looking at it fits the model to it a little more, and the number it eventually produces is the one you will quote to other people. ## Frequently asked questions ### How many examples do I need? For format and tone, a few hundred consistent ones make a visible difference. For task competence, low thousands. For preference tuning, one to three thousand genuine pairs. Coverage of the input distribution matters more than the raw count. ### Should I include examples where the right answer is to refuse? Yes, and most people do not. A model that has never seen a refusal will answer everything, confidently, including questions your context cannot support.