From Token to Transformer - 01 / 10
When you send "Meet me in Shanghai" to a language model, it is tempting to imagine the model seeing the words first and understanding them second. The actual entry point is less intuitive: a tokenizer rewrites the string as symbols from a fixed vocabulary, and those symbols become integers.
That conversion is often dismissed as preprocessing. It is more consequential than that. Tokenization determines what units the model receives, how much text fits in a context window, and where the numerical computation begins.
Why a model cannot consume a string
Neural networks operate on numeric tensors with defined shapes. A Unicode string cannot participate directly in matrix multiplication, so the system needs a finite vocabulary and a mapping from text to entries in that vocabulary.
A simplified pipeline looks like this:
textraw text -> tokens -> token IDs -> embedding vectors -> Transformer layers
A token and a token ID are not the same thing. The token is the symbol produced by the tokenizer. Its ID is only an index into a vocabulary. ID 1532 is not more meaningful than ID 97; it simply selects a different row from the embedding table.
Why not use characters or whole words
Character-level tokenization is robust because a modest character set can represent nearly any word. Its cost is sequence length. A model must spend more positions reconstructing patterns that could have been represented as a single common unit.
Whole-word tokenization has the opposite problem. It preserves familiar lexical units, but real language is open-ended. Product names, identifiers, misspellings, compound words, and new terminology appear constantly. A fixed vocabulary cannot contain every complete word without becoming enormous.
Most modern language models therefore use subword tokenization. Frequent pieces can remain intact, while rare words are decomposed into smaller known pieces. The tokenizer balances three competing goals:
- keep the vocabulary finite;
- avoid unnecessarily long sequences;
- retain a way to represent unseen text.
The exact split depends on the tokenizer. A token is not a natural atom of language; it is a design choice learned or configured for a particular model.
A token is not a word
Whitespace makes English look word-based, but spaces, punctuation, capitalization, and prefixes can all affect token boundaries. Chinese does not imply one token per character either. Frequent phrases may be represented compactly, while rare characters or mixed-language text may fall back to smaller units or bytes.
That is why two strings with similar visible length can produce different token counts, and why the same string can have different counts across models. Rules such as "one Chinese character equals one token" are rough anecdotes, not reliable accounting. Use the tokenizer associated with the target model.
The tokenizer also defines an input protocol
Tokenizers do more than split the visible text. They often add special tokens that represent sequence boundaries, roles, padding, or task structure. The runtime may also produce fields such as attention_mask, which distinguishes real input positions from padding used to align a batch.
In a chat system, the model may receive system instructions, role markers, tool definitions, and formatting tokens that never appear in the text box. All of them consume context. The visible message is only one part of the final sequence.
The consequences continue downstream
Tokenization choices affect the rest of the stack:
- finer splits usually produce longer sequences and consume more context;
- domain terms split into many pieces require the model to compose them repeatedly;
- code, tables, and multilingual text can have very different token densities;
- chat templates and special tokens shape the interaction patterns seen during training.
This does not mean fewer tokens are always better. A tiny vocabulary lengthens sequences. A very large vocabulary increases the embedding and output matrices and gives rare entries fewer training examples. A good tokenizer is a system-level compromise among data, model size, and intended use.
Takeaway
A model does not understand text and then convert that understanding into numbers. The order is reversed. Text is first rewritten as vocabulary symbols, IDs, and vectors; contextual meaning only begins to emerge in the layers that follow.
The next article follows the same sequence into production: why token count affects cost, latency, and the amount of useful evidence a model can keep in context.