Beyond Attention: The FFN, Residual Paths, and Normalization

From Token to Transformer - 09 / 10

"Attention Is All You Need" is a memorable paper title. It also created a persistent misconception that a Transformer is just attention repeated many times.

A working Transformer block also depends on a feed-forward network, residual connections, and normalization. Attention moves information across positions. The other components transform that information, preserve a stable path through the network, and make dozens of layers possible to optimize.

A simplified block

A common pre-norm decoder block can be summarized as:

text
x = x + Attention(Norm(x))
x = x + FFN(Norm(x))

The original Transformer uses post-norm:

text
x = Norm(x + Attention(x))
x = Norm(x + FFN(x))

Both contain two sublayers and two residual paths. Pre-norm often improves optimization in very deep networks, while real architectures also vary the normalization type and residual layout. The responsibilities matter more than memorizing one canonical diagram.

Attention communicates across positions

Attention returns a weighted mixture of Values from other visible positions. It answers, in a learned way, "where should this position obtain information?"

Weighted mixing alone has limited nonlinear expressive power. The information retrieved at a position still needs to be transformed. That is the role of the FFN.

The FFN transforms each position

The classic feed-forward network applies the same parameters independently to every sequence position:

text
FFN(x) = activation(xW_1 + b_1)W_2 + b_2

The intermediate width usually expands before being projected back to the model width. The original Transformer uses ReLU; modern models often use GELU or gated variants such as SwiGLU.

The FFN does not directly exchange information between tokens. A useful division of labor is:

attention mixes information across the sequence dimension;
the FFN transforms information across the feature dimension.

Alternating the two lets the model communicate and then compute on the result.

Residual connections preserve a direct path

A residual connection adds the input back to a sublayer output:

text
output = x + sublayer(x)

The sublayer can learn an update instead of recreating the complete representation. More importantly, the residual stream provides a relatively direct route for information and gradients through a deep network.

It also explains why an isolated attention head is not a complete explanation. The final representation includes that head, other heads, the output projection, the previous residual state, feed-forward updates, and later layers.

Normalization controls numerical scale

LayerNorm normalizes features within each token representation and then applies learned scale and offset parameters. RMSNorm uses a simpler root-mean-square scale without centering by the mean.

Normalization does not erase information. It keeps activation scales manageable across samples and depth, reducing numerical drift during optimization. Where normalization sits relative to the residual path can materially change training stability.

Pre-norm is not universally superior to post-norm. Architecture design balances stability, final quality, and scaling behavior.

What depth adds

One layer might connect a pronoun to a candidate entity. Its FFN can transform the combined features into a representation useful for the next layer. A later attention layer can then use that result to establish a higher-level relation.

Deep stacking repeatedly performs:

text
read context -> transform features -> preserve prior state -> control scale

That is more concrete than saying each layer simply "understands more."

Other members of the architecture

Complete models also include dropout, vocabulary projections, positional mechanisms, and masks. Modern variants add mixture-of-experts layers, local attention, gated residuals, or parallel sublayers.

Transformer is an evolving architecture family, not a circuit frozen in 2017. Attention, the FFN, residual paths, and normalization remain the most stable skeleton for understanding that family.

Takeaway

Attention routes information across positions. The FFN performs nonlinear transformation at each position. Residual connections preserve information and gradient paths. Normalization keeps scales controlled.

The final article zooms out one more level: Transformer is an architecture, while BERT and GPT are model families built from different parts of that architecture and trained with different objectives.