Serving
Latency
What is latency?
Latency is the time between sending a request and receiving a response. For language models it decomposes into time to first token and the rate of subsequent tokens, which are affected by different things and should be measured separately.
Total latency is roughly time to first token plus output length divided by generation rate. Output length is usually the dominant term and the one most under your control — a model trained to be concise is faster in a way no hardware change matches.
Report percentiles, not means. The mean hides the tail, and the tail is what users complain about.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.