From Token to Transformer - 04 / 10
Embeddings are often described as coordinates on a semantic map: cats sit near dogs, and familiar vector directions encode familiar concepts. The metaphor is useful, but it can suggest that a model contains a clean, human-readable dictionary of meaning.
It does not. An embedding does place a discrete token in a continuous vector space, but that space is learned to reduce a training loss. It reflects linguistic statistics, the training objective, the data distribution, and the compromises of the architecture.
From discrete symbols to continuous values
Token IDs are indices. Consecutive IDs have no inherent relationship. Neural networks, however, can transform continuous vectors through dot products, projections, weighted sums, and nonlinear functions.
The embedding layer connects those two worlds. Every vocabulary entry owns a vector of width D. The values may begin near random initialization and are adjusted by backpropagation alongside attention, feed-forward, and output parameters.
The word "trainable" matters. An embedding is not an external dictionary definition inserted into the model. It is part of the model's parameter set.
Why structure appears in the vector space
Tokens that occur in similar contexts often need to support similar predictions. Training can therefore create useful local structure for numbers, months, locations, grammatical roles, or other recurring patterns.
But "nearby vectors" does not automatically mean "the same meaning to a human." Geometry depends on the representation and its objective:
- input embeddings are initial parameters associated with vocabulary entries;
- intermediate hidden states already include sentence context;
- sentence embeddings trained for retrieval optimize a different target again.
Comparing vectors from these categories can be meaningless. Before discussing similarity, specify which layer, which object, and which training objective produced the representation.
One token, multiple uses
The base embedding table normally provides one row per token. "Apple" in a fruit sentence and "Apple" in a product announcement begin from the same input vector.
Their representations diverge in later layers. Words such as "orchard" and "ripe" pull one occurrence toward a fruit context; "chip" and "launch" change the other. Self-attention gathers those signals, and feed-forward networks transform them into different hidden states.
An input embedding is therefore a context-free draft, not the final meaning of a token in a sentence.
Is a larger embedding dimension always better?
More dimensions provide more capacity, but they also increase parameters, memory, and computation. A vocabulary of size V and hidden width D requires roughly V x D embedding parameters.
Increasing D does not guarantee cleaner semantics. The model still needs enough data and an objective that can use the added capacity. Width is a design variable, not a direct measurement of understanding.
Vocabulary size creates a related tradeoff. A larger vocabulary can shorten some sequences, but it expands the embedding and output matrices and gives rare entries fewer examples. Tokenizer design and embedding design cannot be optimized independently.
Why embeddings matter to the rest of the model
All later Transformer operations begin with these vectors. Q, K, and V are linear projections of the current representation. Residual paths assume a stable hidden width. The output layer eventually maps a hidden state back to vocabulary logits.
If the initial representation does not offer distinguishable, learnable signals, the rest of the network has poor material to work with. The embedding layer does not complete understanding; it creates a representation that the network can repeatedly revise.
Be cautious when visualizing embeddings
Two-dimensional plots of embeddings can reveal patterns, but dimensionality reduction discards most of the original geometry. A visible cluster is a hypothesis, not proof that the model has formed a stable human concept.
Stronger evidence connects geometry to behavior: does similarity improve retrieval, can a probe reliably recover an attribute, and does intervening on a direction change model output? A compelling scatter plot is not a map of the model's thoughts.
Takeaway
Embeddings convert discrete tokens into continuous, trainable vectors. They create a space in which the network can compute relationships, but they are not dictionary definitions and they do not by themselves resolve context.
The next requirement is order. Without positional information, self-attention sees a set of token representations rather than a sentence with direction and distance.