From Token to Transformer - 08 / 10
A sentence contains several relationships at once. A token can depend on a subject, a nearby modifier, and a distant topic. If every relationship shares one attention map, the model has only one set of projections for deciding where to look and what to retrieve.
Multi-head attention learns several Q, K, and V projection sets in parallel. Each head runs attention in its own lower-dimensional subspace, and the results are combined afterward.
What one head computes
Head i owns its own parameters:
textQ_i = XW_Q_i K_i = XW_K_i V_i = XW_V_i
It produces an independent weighted result:
texthead_i = Attention(Q_i, K_i, V_i)
The model concatenates all head outputs along the feature dimension and applies an output projection:
textMultiHead(X) = Concat(head_1, ..., head_h) W_O
W_O is not cosmetic. It remixes information from the heads and returns a representation with the width required by the residual path.
Multiple heads are multiple subspaces, not a vote
Suppose a model has hidden width 768 and 12 heads. A common arrangement gives each head 64 dimensions. The total representation is not twelve times wider; the available width is divided across separate projections.
Those projections can create different matching geometries. One head may often connect local phrases, another may preserve distant entity information, and others may specialize in delimiters, position, or redundant support.
These are learned tendencies, not job descriptions. It is unsafe to claim that head 3 always performs syntax and head 7 always resolves references. Heads can overlap, cooperate, change behavior with context, and sometimes be pruned.
Why not use one wider head
A wider single head still produces one attention distribution per position. Its Values may carry more features, but every feature is mixed using the same set of source weights.
Multiple heads let one position form several routing distributions at once. A pronoun can connect strongly to candidate entities in one head while another preserves local predicate information. The output projection and later layers can combine both.
The advantage is not merely additional parameters. It is the ability for routing patterns to diverge.
More heads are not always better
With a fixed hidden width, increasing the number of heads reduces the width of each head. Very small head dimensions can limit the capacity of individual projections, and more heads add implementation and cache-management overhead.
Modern models also vary the number of Key and Value heads. Multi-query attention shares one K/V set across many Query heads. Grouped-query attention shares K/V within groups. Both reduce the KV cache used during autoregressive decoding.
These variants show that Query-head count, K/V-head count, width, and latency are engineering tradeoffs. There is no universal best ratio.
Heads cannot replace depth or the FFN
One multi-head attention layer performs one round of information exchange. Complex behavior usually requires repeated layers: establish a local relation, aggregate entities, then use the result in a higher-level prediction.
Attention also mainly mixes Values linearly across positions. The feed-forward network provides richer nonlinear transformation at each position. Residual connections and normalization keep the combined system trainable.
Reducing a Transformer to multi-head attention leaves out the structures that make deep stacking possible.
Takeaway
Multi-head attention gives the model several learned routing spaces and several connection distributions in parallel, then projects their combined output back into the model stream. Heads are not a panel of named experts; they are flexible, overlapping computational subspaces.
The next article steps back from attention and reconstructs a complete Transformer block, including its feed-forward network, residual paths, and normalization.