Serving
KV cache
What is kv cache?
The KV cache stores the key and value tensors computed for tokens already processed, so generating each new token does not require reprocessing the whole sequence. It makes generation tractable and consumes memory proportional to sequence length and concurrency.
Without it, generating the thousandth token would mean recomputing attention over the preceding nine hundred and ninety-nine. With it, each new token attends to cached state and the cost per token is roughly constant.
The memory cost is what limits concurrency at long context. Serving many long conversations at once can exhaust memory through cache alone, with the weights themselves comfortably resident.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.