Serving
Replica
What is replica?
A replica is one running copy of a model serving requests. Adding replicas increases throughput and cost roughly linearly; it does not reduce the latency of an individual request, which is set by the model and the hardware.
The distinction is worth being precise about because it is a common confusion. More replicas means more requests handled at once; each one still takes as long as it took.
Replica count is therefore a capacity decision driven by concurrency, and reducing single-request latency is a different problem addressed by model size, hardware class or output length.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.