Model architecture
Dense model
What is dense model?
A dense model uses every parameter for every token. Compute and memory both scale with the full parameter count, which makes its behaviour simple to predict and its fine-tuning well understood.
Most models are dense, and most published fine-tuning practice assumes it. For a first fine-tune the predictability is worth something in itself.
The comparison against a sparse model of the same total size is not simple: the sparse one is faster per token, and the dense one usually holds more capability per gigabyte of memory.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.