From Token to Transformer - 03 / 10
After a tokenizer splits the text, the model has not understood anything yet. It has a sequence of vocabulary indices. Before Transformer computation can begin, those integers must be replaced with high-dimensional vectors.
This short pipeline contains three common sources of confusion: an ID carries no semantics, an input embedding is not contextual understanding, and the model does not perform arithmetic directly on the vocabulary numbers.
Follow one input through the boundary
Suppose a tokenizer produces four pieces:
text["Meet", " me", " in", " Shanghai"]
The vocabulary maps them to integer IDs:
text[9412, 502, 304, 18377]
Those numbers are addresses. The vocabulary could be reordered and "Meet" could become ID 42 without changing its meaning, provided every associated parameter uses the same mapping.
The runtime also adds a batch dimension and related fields:
textinput_ids shape: [batch, sequence] attention_mask shape: [batch, sequence]
An attention_mask is not a learned attention weight. It usually marks which positions contain real input and which are padding or otherwise unavailable.
Embedding is a lookup
The model owns a trainable embedding matrix. If the vocabulary has size V and the hidden width is D, the matrix has shape [V, D].
Each token ID selects one row:
textembedding = embedding_table[token_id]
This operation is closer to a database lookup than multiplication by the ID. A sequence of length N becomes a matrix of shape [N, D], or [B, N, D] after adding a batch dimension.
IDs 9412 and 9413 are numerically adjacent but need not represent related tokens. Any geometric similarity comes from the learned vectors, not the index values.
Position enters before or during attention
Token embeddings alone do not identify sequence order. The original Transformer adds sinusoidal position encodings to token embeddings. Other models use learned positions, relative position biases, or rotary position embeddings.
At a conceptual level:
textinitial representation = token information + positional information
The implementation does not always use literal addition, but the purpose is stable: the same token at positions 2 and 20 must be distinguishable.
The input embedding is not the current meaning
The embedding table normally returns the same initial vector for a token wherever it appears. "Apple" begins from the same row in a fruit sentence and a company sentence.
Context enters through the Transformer layers. Self-attention brings in information from neighboring and distant positions; the feed-forward network transforms the result. After several layers, the two occurrences of "Apple" have different hidden states.
Keep these concepts separate:
| Representation | Purpose | Current context included? |
|---|---|---|
| Token ID | Locate a vocabulary entry | No |
| Input embedding | Provide a trainable starting vector | Usually not yet |
| Hidden state | Store the representation after contextual updates | Yes |
Calling all three "word vectors" is convenient, but it hides where contextualization actually happens.
How the model returns to tokens
At the output end, the current hidden state is projected into a vector with one value per vocabulary entry. These values are logits. Softmax converts them into a probability distribution, and a decoding strategy selects the next token.
Many language models tie the input embedding matrix to the output projection, although this is not required. The full loop is:
texttoken ID -> initial vector -> layered hidden state -> vocabulary logits -> next token ID
Generation repeats that loop, appending one selected token at a time.
Takeaway
A token is a segmentation unit, an ID is its vocabulary address, an embedding is a trainable starting vector, and a hidden state is the context-dependent representation produced by the network.
Once these layers are distinct, attention stops looking like an operation performed on words. It is a learned exchange among vectors that already carry token and position information.