Model architecture
Attention
What is attention?
Attention is the mechanism that lets a model weigh the relevance of every earlier token when processing the current one. Its cost grows quadratically with sequence length, which is why long context is expensive.
Each token computes a query, and every token produces a key and a value. Comparing queries against keys is what makes cost scale with the square of the sequence.
Grouped-query attention, which shares keys and values across several query heads, is the standard mitigation in modern models and is a large part of why long contexts became practical.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.