Serving
Quantisation
What is quantisation?
Quantisation stores model weights at lower numerical precision — commonly eight or four bits instead of sixteen — to reduce memory and increase speed. Quality loss is small at eight bits and becomes noticeable, though often acceptable, at four.
It is what makes a model fit on hardware that could not otherwise hold it, both for training and for serving. The saving is close to proportional: four-bit weights occupy roughly a quarter of half-precision weights.
The degradation is not uniform across tasks. Reasoning and precise generation tend to suffer more than classification, so the right level is a decision to make against your own evaluation rather than a general rule.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.