Machine Learning
Decoder-only
GPT is a transformer with only decoder blocks.
The decoder-only architecture
GPT's block is the original transformer decoder without cross-attention: only masked self-attention and the FFN. There is no encoder; the context is the input sequence itself:
x = gomme[ids] + konum[:n]
for blok in bloklar:
x = x + maskeli_dikkat(layer_norm(x)) # pre-LN
x = x + ffn(layer_norm(x))
logit = layer_norm(x) @ gomme.T # weight tying
From GPT-2 on, LayerNorm is placed before the sub-layer (pre-LN); this makes training stable in deep networks. The output layer reuses the embedding matrix (weight tying).
GPT-2 small: 124 million
GPT-2 small: vocabulary 50257, context 1024, d = 768, 12 layers. Thanks to weight tying, the output layer adds no parameters:
V, N, d, L = 50257, 1024, 768, 12
gomme = V * d + N * d
katman = 2 * (2 * d) + (d * 3 * d + 3 * d) + (d * d + d) + (d * 4 * d + 4 * d) + (4 * d * d + d)
print(gomme + L * katman + 2 * d) # including the final LayerNorm
Without weight tying, a separate 38.6-million-parameter output matrix would be needed.
Exercise GPT-2 parameter count
Compute the parameter counts of the embeddings and a single layer with the given settings and print them on one line. Print the total (12 layers + the final LayerNorm). Finally print the number of parameters that would be added without weight tying (V * d).
Show example solution
V, N, d, L = 50257, 1024, 768, 12
gomme = V * d + N * d
katman = 2 * (2 * d) + (d * 3 * d + 3 * d) + (d * d + d) + (d * 4 * d + 4 * d) + (4 * d * d + d)
print(gomme, katman)
print(gomme + L * katman + 2 * d)
print(V * d)
Test yourself
-
Which sub-layer is missing in a GPT block compared with the original transformer decoder?
- Cross-attention
- Masked self-attention
- FFN
- LayerNorm
Show answer
Cross-attention — Since there is no encoder, there is no source sequence to look at.
-
Reuse the embedding matrix at the output.
logit = layer_norm(x) @ gomme.____
Show answer
logit = layer_norm(x) @ gomme.T
-
Match.
- pre-LN
- Norm before the sub-layer
- weight tying
- Output = embeddingᵀ
- context
- 1024 tokens
- GPT-2 small
- ~124M parameters
Next lesson: Causal mask
