Model architecture
Feed-forward network
Also called MLP.
What is feed-forward network?
The feed-forward network is the part of each transformer block that processes each position independently after attention has mixed information between positions. It holds the majority of a model’s parameters.
Attention decides what to look at; the feed-forward layers decide what to do with it. Much of a model’s factual and procedural knowledge is thought to reside here.
This is why extending an adapter to the feed-forward layers gives it more capacity than attaching to attention alone — it reaches where more of the model’s substance lives.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.