From Token to Transformer - 10 / 10
Transformer, BERT, and GPT are often listed as if they were three competing model types. The relationship is different: Transformer introduced a set of architectural building blocks, while BERT and GPT selected different parts and objectives to create two influential model families.
The useful questions are not which acronym wins. Ask what context each position can see, what prediction objective trained the model, and what interface the final system exposes.
The original Transformer is encoder-decoder
The 2017 Transformer was designed for machine translation and contains two stacks.
The encoder reads the complete source sequence. Every position can use self-attention over other visible source positions and gradually build contextual representations.
The decoder generates the target sequence autoregressively. Masked self-attention sees only target tokens already produced, while cross-attention reads the encoder output.
textsource text -> encoder representations generated target prefix -> decoder + cross-attention -> next token
Transformer names this architecture and its core components. It is not synonymous with any one chat model.
BERT takes the encoder path
BERT stands for Bidirectional Encoder Representations from Transformers. It stacks Transformer encoders and pretrains them with masked language modeling: hide some input tokens and ask the model to recover them using context on both sides.
"Bidirectional" refers to the ability of a representation to condition on both left and right context. It does not generate a sequence from left to right by default. This objective makes BERT-style models natural for classification, entity recognition, retrieval representations, and extractive question answering after fine-tuning.
The original BERT also uses next sentence prediction. Later work changed or removed that objective. Today, "BERT-like" usually refers more broadly to the encoder-only, bidirectional representation approach rather than an exact copy of the original recipe.
GPT takes the decoder-only path
GPT is a generatively pretrained Transformer. Modern GPT-style models usually use a decoder-only architecture with a causal mask: each position predicts the next token using only the prefix to its left.
textp(text) = p(t_1) p(t_2 | t_1) ... p(t_n | t_<n)
The objective looks like ordinary continuation, but it provides a general interface. Instructions, examples, source material, and desired answers can all be represented as token sequences. Classification becomes label generation, translation becomes target-text generation, and question answering becomes conditional continuation.
Instruction tuning, preference optimization, tool-use training, and system scaffolding later turn a base language model into a conversational or agentic product. Those capabilities do not follow automatically from the letters GPT.
"Understanding versus generation" is too simple
A common shorthand says BERT understands and GPT generates. It is useful for orientation and inaccurate as a boundary.
GPT must build contextual representations before it can generate. BERT representations can also feed a decoder. The more precise differences are attention visibility, training objective, and default output interface:
| Family | Typical structure | Pretraining visibility | Common interface |
|---|---|---|---|
| BERT-like | Encoder-only | Both left and right context | Represent, classify, extract |
| GPT-like | Decoder-only | Causal left context | Autoregressive generation |
| Original Transformer / T5-like | Encoder-decoder | Full input, causal output | Conditional generation |
These are architectural tendencies, not absolute capability limits. Data, scale, fine-tuning, and serving systems also shape the result.
Why decoder-only models became general-purpose systems
Autoregressive training has a practical advantage: any text supplies next-token prediction targets without manual labels. Many tasks can be expressed through the same "given this context, continue" interface.
At scale, that interface works well for few-shot examples, instruction following, and conversation. KV caching also provides a clear optimization path for incremental decoding.
The tradeoffs remain. Generation is serial, bidirectional encoding tasks may be less economical, and long output directly increases latency. Dominant does not mean optimal for every workload.
Choose by task shape
For high-throughput classification, retrieval, or entity extraction, a small encoder can be the simpler and cheaper system. For open-ended writing, code generation, and multi-turn agents, a decoder-only model is a natural fit. When both source and target are long and the task is an explicit transformation, encoder-decoder models retain structural advantages.
Parameter count alone is not a model-selection strategy. Evaluate the attention pattern, training objective, context limits, deployment cost, and performance on your own data.
Closing the series
The complete path from a string to generated text is now visible:
texttext -> tokens -> IDs -> embeddings + position -> attention + FFN + residual + normalization -> hidden state -> vocabulary logits -> next token
Transformer supplies the skeleton for repeated representation updates. BERT and GPT show how different visibility rules and objectives can turn that skeleton into different model interfaces.
The system is still complex, but it is no longer a single black box. It is a sequence of engineering choices that can be inspected, measured, and challenged.