Scale to zero or always warm: choosing a serving mode

An always-warm endpoint bills continuously for the time it is resident, regardless of traffic. That is the honest description of reserved capacity, and its virtue is that the bill does not change when usage does.

A scale-to-zero endpoint costs nothing while idle and pays a cold start on the first request after a quiet period. The saving is real and can be large for intermittent workloads.

Ask who waits. If a person is sitting in front of an interface, keep it warm — a cold start reads as the product being broken, however good the steady-state latency is. If the caller is a nightly job, an automation or an internal tool, let it scale to zero.

The saving is not the deciding factor. A cheap endpoint that makes your product feel broken is not a saving.

Cooldown is how long the endpoint stays resident after the last request, and it is the dial that decides how often cold starts happen at all.

Set it below the typical gap between requests and normal pauses trigger reloads — the worst outcome, because you pay for the reload and the user waits for it. Set it far above the quiet period and you have built an always-warm endpoint with extra steps.

Look at the actual distribution of gaps in your traffic. Set cooldown above the typical gap and below the quiet period.

A cold endpoint queues requests while capacity is acquired. Both bounds are yours: a depth beyond which requests are refused with a retry hint, and a wait beyond which they time out.

Setting these is not pessimism. An unbounded queue turns a capacity problem into a mysterious latency problem, and the client never learns which it is facing.

With streaming enabled, the perceived latency is the time to the first token. That does not remove a cold start, but it does mean that once capacity is up, the response begins immediately rather than after the full generation.

For borderline cases — an internal tool that people do sometimes wait on — streaming plus a generous cooldown is often enough to make scale-to-zero acceptable.