Serving
Inference
What is inference?
Inference is running a trained model to produce output. It is where nearly all lifetime cost accumulates for a successful model, because training happens a handful of times and inference happens on every request forever.
The cost structure differs fundamentally from training. Training is a bounded job with a known end; inference is an open-ended commitment whose cost scales with product success.
This is the argument for optimising model size against your accuracy bar rather than against ambition. A model twice as large costs roughly twice as much on every request for the rest of its deployment.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.