From Token to Transformer - 05 / 10
"The dog chased the cat" and "The cat chased the dog" contain almost the same tokens. If a model knew only which tokens were present and not where they appeared, the two sentences would be difficult to distinguish.
An RNN reads one step after another, so order is built into state propagation. A Transformer processes a sequence in parallel. That choice improves training efficiency but gives up the free notion of "before" and "after." Positional mechanisms put structure back into the computation.
Why self-attention does not know order by itself
Ignore positional information and feed a set of token vectors into self-attention. If the inputs are permuted, the outputs are permuted in the same way. The operation can model relationships among vectors, but it has no independent clue that one vector originally occupied position 1 and another position 4.
This permutation equivariance is useful for sets and insufficient for language. Syntax, reference, and causality depend on direction and distance:
- subject and object roles change when word order changes;
- negation depends on where it appears;
- scope, parentheses, and statement order matter in code.
Attention can use order only after the representation or the attention calculation contains an order signal.
The original Transformer's solution
The 2017 Transformer adds vectors made from sine and cosine functions to token embeddings. A range of frequencies gives each position a distinct pattern and allows the model to infer relative offsets from combinations of those patterns.
Conceptually:
textfirst-layer input = token embedding + position encoding
The same token at positions 2 and 20 now begins with a different representation, and its Q, K, and V projections can carry that difference into attention.
The paper also evaluated learned positional embeddings. That comparison highlights the real requirement: position need not come from one specific formula, but sequence structure must enter in a form the network can use.
Absolute position is not the only option
As contexts grew, positional design shifted from only asking "which index is this?" to representing "how far apart are these tokens, and in which direction?"
- Learned absolute positions store parameters for each index. They are direct but often tied to a trained length range.
- Relative position methods alter attention based on distance or direction between tokens.
- RoPE rotates Q and K representations so their dot products carry relative position information.
RoPE and its variants are common in modern language models, but no positional method grants reliable infinite context. Extrapolation beyond training, attention quality, and memory cost remain separate problems. A model accepting a longer sequence does not guarantee that it uses every position equally well.
What position does and does not provide
A positional signal helps the model distinguish order, proximity, and distance. It does not encode grammar directly. It does not label a subject or force the model to attend to the nearest noun.
Those relationships are learned from data. Position supplies a coordinate system, attention decides how to use it, and deeper layers turn the result into representations useful for prediction.
| Question | Main source of information |
|---|---|
| Which token is this? | Token embedding |
| Where is it? | Positional mechanism |
| Which positions are related? | Attention |
| What does it mean here? | The full stack of contextual layers |
Why long context remains difficult
Even with a position assigned to every token, the model still has to find relevant information among many candidates. Full attention becomes expensive as sequence length grows, and training data may not provide enough signal for stable use of very distant dependencies.
Long-context capability therefore contains at least three questions: can the sequence fit, can it be computed efficiently, and can the model retrieve the right evidence? Positional encoding mainly addresses the representation of order and distance.
Takeaway
Parallel self-attention does not preserve sequence order automatically. Positional mechanisms make identical token sets distinguishable by providing direction, distance, and index information.
The model now has token content and position. The next article turns to the exchange mechanism itself: how attention decides which positions should contribute information to the current one.