Training
Target modules
What is target modules?
Target modules are the specific layers of a model that an adapter is attached to. Attaching to the attention projections alone is the cheapest configuration; including the feed-forward layers adds capacity and memory.
The common minimal choice is the query and value projections in each attention block. It is cheap, well tested, and enough for most stylistic and format adaptation.
Adding the feed-forward layers gives the adapter access to where much of a model’s factual and procedural knowledge is thought to sit. It roughly doubles trainable parameters, and it is the change worth trying when a task needs more than surface adaptation.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.