Reinforcement learning (GRPO)
Reasoning with reinforcement learning
When should I use GRPO instead of supervised fine-tuning?
Use reinforcement learning when you can score an answer automatically but cannot write the ideal reasoning that produced it. Maths, code that must pass tests, and anything with a verifiable outcome are the cases where it beats supervised training.
| Objective | Reinforcement learning (GRPO) |
|---|---|
| Dataset shape | Prompts plus a programmatic reward function. No target completions required. |
| Metric that decides it | Pass rate on held-out problems, plus a watch on response length, which reinforcement learning inflates. |
| Cheapest starting point | Qwen3 8B on RTX 4000 Ada$0.09 per GPU-hour |
Supervised fine-tuning needs you to supply the right answer, including the reasoning. For a maths problem or a coding task, writing thousands of ideal derivations by hand is impractical — but checking whether an answer is correct is trivial. That asymmetry is exactly what reinforcement learning exploits.
The reward function is the entire design. A reward for correct final answers alone will produce a model that guesses more confidently. A reward that also values verifiable intermediate steps produces one that reasons. Whatever you reward, you will get precisely that and nothing else.
Budget for a longer run. Reinforcement learning generates multiple candidate responses per prompt and scores each, so a step costs several times what a supervised step costs. Start on a small model to get the reward function right before scaling.
The mistake to avoid
Not watching response length. Models reliably discover that longer answers score better under a sloppy reward, and you end up with a verbose model that is no more correct.
Models to start from
In order. Start at the top and move down the list only if the evaluation gate says you have to — the smallest model that clears your bar is the one you pay for on every request forever.
Before you deploy it
Whatever the metric above says in isolation, the deployment decision is comparative: is this better than what is running today? Gate the candidate against the incumbent, blinded, on held-out cases from your own traffic. A fail is a cheap answer to an expensive question.
Frequently asked questions
Do I need a separate reward model?
Not for verifiable tasks — a programmatic checker is better, cheaper and harder to game. Reward models are for subjective quality, where no checker exists.
Can I combine it with supervised fine-tuning?
Yes, and it is usually the right order: supervised first to establish format and basic competence, then reinforcement learning to improve correctness.
Start with Qwen3 8B
Upload your dataset, read the validation report, and forecast the run before anything is leased.