Serving
Cold start
What is cold start?
A cold start is the delay on the first request to an endpoint with no resident replica: capacity must be acquired, the model weights loaded and the runtime initialised. It is dominated by weight loading, so it scales with model size.
The user-visible effect is one very slow request followed by normal ones, which is a distinctive and easily misdiagnosed pattern in latency graphs.
The mitigations are keeping a minimum replica warm, choosing a smaller model, or accepting it where the caller is a batch process that does not care. Adapter-based serving helps too: loading a small adapter against an already-resident base is far quicker than loading a whole model.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.