Serving
Throughput
What is throughput?
Throughput is how much work a deployment completes per unit time — tokens per second, or requests per second. It is improved by batching and by adding replicas, and it trades against the latency of any individual request.
Batching is the main lever. Processing several requests together uses the hardware far more efficiently, which is why throughput per GPU rises steeply with concurrency and then flattens.
The trade against latency is direct: a request that waits to be batched with others completes later than it would have alone. Which to optimise depends entirely on whether a human is waiting.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.