Attention Is Context Routing, Not Human Focus

From Token to Transformer - 06 / 10

"Maya handed the report to Jordan because she was leaving early."

When reading "she," a person revisits the sentence and considers which earlier entity fits. A model needs a way to connect the current position to other positions, but it has no hard-coded grammatical lookup. Attention provides a trainable routing mechanism: issue a query, score possible sources, and retrieve a weighted mixture of their information.

The name can sound cognitive. A more precise description is simpler: attention dynamically computes how representations at different positions exchange information.

Three steps in one equation

Each position produces a Query, Key, and Value from its current hidden state. For now, think of them as a search request, a searchable index, and the content returned by a match.

Scaled dot-product attention is:

text
Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

The equation contains three operations:

Score: compare each Query with the available Keys using dot products.
Normalize: divide by sqrt(d_k) for numerical scale, then apply softmax.
Aggregate: use the resulting weights to form a weighted sum of Values.

The output is not necessarily one selected token. It is usually a mixture of Values from several positions. The distribution may be sharp for one input and broad for another.

What "self" means in self-attention

In self-attention, Q, K, and V are all projected from the same sequence. Every position both asks for information and offers information to other positions.

In cross-attention, Queries come from one representation set while Keys and Values come from another. The original Transformer decoder uses cross-attention to read encoder output. Multimodal models can use a similar pattern for text queries over image features.

The operation is the same; the source of the representations changes.

Masks define forbidden connections

Attention does not always expose every position. A mask modifies scores before softmax so that some sources cannot receive weight.

A padding mask hides empty positions introduced for batching.
A causal mask prevents a generative model from seeing future tokens.
Structured masks can enforce local windows or other connection patterns.

The causal mask is essential for GPT-style training. Even if the complete training sequence is present in a tensor, the representation at each position may only use tokens to its left. Otherwise the model could read the answer it is supposed to predict.

Why attention changed sequence modeling

An RNN passes information through a chain of recurrent steps. A distant dependency must travel through many updates. Self-attention can create a direct connection between any two visible positions in a single layer, and the entire training sequence can be processed in parallel.

This gives the model two useful properties:

connection patterns change with the input rather than staying in a fixed window;
distant positions can exchange information without a long recurrent path.

Direct access is not guaranteed retrieval. The model still has to learn which links matter, and very long contexts can dilute or misroute attention.

Why full attention is expensive

Standard attention constructs scores for pairs of positions. With sequence length N, the score matrix is roughly N x N. Its compute and memory costs are a major obstacle to longer contexts.

FlashAttention improves memory access and practical performance. Sliding-window and sparse attention reduce the set of computed connections. These techniques matter greatly, but none turns every long-document problem into a solved retrieval problem.

Are attention weights explanations?

An attention map shows routing in a particular layer and head during one forward pass. It is useful diagnostic evidence, but not a complete causal account of the final answer.

Information also travels through other heads, output projections, residual paths, feed-forward networks, and later layers. To show that a weight caused an output, an intervention is stronger evidence than a visualization alone.

Takeaway

Attention lets each position choose information sources dynamically. Query and Key determine how much to retrieve, Value determines what is retrieved, and masks determine which connections are allowed.

The next article separates Q, K, and V in more detail. The three projections exist to decouple matching from content transfer, not to make the notation look sophisticated.