Serving
Time to first token
Also called TTFT.
What is time to first token?
Time to first token is the delay between a request arriving and the first output token being produced. It covers queueing and processing the input prompt, and it is what a user perceives as responsiveness when output is streamed.
It scales with prompt length because the whole input must be processed before generation begins. This is a concrete argument for fine-tuning: a shorter prompt is directly a lower time to first token on every request.
With streaming enabled, this is effectively the perceived latency of the system. Everything after it feels like the response arriving rather than the system thinking.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.