Training
Using reinforcement learning to improve reasoning
How does GRPO work and when should I use reinforcement learning to fine-tune?
Use it when an answer can be checked automatically but the ideal reasoning cannot be written at scale — maths, code that must pass tests, anything with a verifiable outcome. The reward function is the entire design, and the model will optimise exactly what it measures.
The asymmetry it exploits
Supervised fine-tuning needs the ideal output, including the reasoning. Writing thousands of ideal mathematical derivations by hand is a research project; checking whether an answer equals the expected value is three lines of code.
Reinforcement learning uses the checker. The model generates several candidate answers per prompt, each is scored, and the training signal comes from the comparison between them. No target text is required.
The reward function is the design
Everything that goes wrong here goes wrong in the reward. Reward the final answer alone and you get a model that guesses more confidently. Reward anything correlated with length and you get verbosity. Whatever you measure improves, precisely, and with no regard for what you meant.
Programmatic checks are the most reliable form — do the tests pass, does the value match — because they are objective and cannot be talked around.
Watch response length
Models reliably discover that longer answers score better under an imprecise reward. The symptom is a rising score alongside samples that are getting worse, and the only way to see it is to read outputs rather than watch metrics.
Track length as a first-class metric throughout the run, not as an afterthought.
It costs more, so prove the reward first
Each step generates and scores multiple completions rather than one, so a step costs several times a supervised step. That makes an ill-specified reward expensive as well as ineffective.
Run it on the smallest model that supports the objective first. A tiny model will find the holes in your reward function far more cheaply than a large one, and the holes are the same holes.
Sequence it after supervised training
Supervised fine-tuning first to establish format and basic competence, then reinforcement learning to improve correctness. Starting cold with reinforcement learning spends expensive steps teaching the model things a cheap objective teaches better.
Frequently asked questions
Do I need a reward model?
Not for verifiable tasks — a programmatic checker is better in every respect. Reward models are for subjective quality where no checker can exist, and they introduce a second model whose own errors join the training signal.
Which models support it?
A narrower set than supervised fine-tuning. Filter for it before building the dataset rather than discovering the constraint afterwards.
Terms used here
More on training
LoRA, QLoRA or full fine-tuning: how to choose
What is the difference between LoRA, QLoRA and full fine-tuning?
Why fine-tuned models fail, and how to tell which failure you have
Why did my fine-tuned model get worse instead of better?
When to use preference tuning instead of supervised fine-tuning
When should I use DPO instead of supervised fine-tuning?
Last verified 6 August 2026.
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.