Serving
Batching
What is batching?
Batching processes multiple inference requests together in one pass. It dramatically improves hardware utilisation because generation is limited by memory bandwidth rather than compute, and the same weight read serves every request in the batch.
Continuous batching improves on the naive approach by adding new requests to the running batch as earlier ones finish, rather than waiting for the whole batch to complete. This keeps utilisation high under variable request lengths.
The cost is a small latency penalty per request and increased memory pressure, since every request in flight needs its own attention cache.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.