CodeBlanks Logo Icon CodeBlanks
Login Register

Decoder-only

GPT is a transformer with only decoder blocks.

The decoder-only architecture

GPT's block is the original transformer decoder without cross-attention: only masked self-attention and the FFN. There is no encoder; the context is the input sequence itself:

x = gomme[ids] + konum[:n]
for blok in bloklar:
    x = x + maskeli_dikkat(layer_norm(x))     # pre-LN
    x = x + ffn(layer_norm(x))
logit = layer_norm(x) @ gomme.T                # weight tying

From GPT-2 on, LayerNorm is placed before the sub-layer (pre-LN); this makes training stable in deep networks. The output layer reuses the embedding matrix (weight tying).

GPT-2 small: 124 million

GPT-2 small: vocabulary 50257, context 1024, d = 768, 12 layers. Thanks to weight tying, the output layer adds no parameters:

V, N, d, L = 50257, 1024, 768, 12
gomme = V * d + N * d
katman = 2 * (2 * d) + (d * 3 * d + 3 * d) + (d * d + d) + (d * 4 * d + 4 * d) + (4 * d * d + d)
print(gomme + L * katman + 2 * d)       # including the final LayerNorm

Without weight tying, a separate 38.6-million-parameter output matrix would be needed.

Practice GPT-2 parameter count

Compute the parameter counts of the embeddings and a single layer with the given settings and print them on one line. Print the total (12 layers + the final LayerNorm). Finally print the number of parameters that would be added without weight tying (V * d).

Starter code

model.py
V, N, d, L = 50257, 1024, 768, 12

# buraya yaz

Sample solution

model.py
V, N, d, L = 50257, 1024, 768, 12

gomme = V * d + N * d
katman = 2 * (2 * d) + (d * 3 * d + 3 * d) + (d * d + d) + (d * 4 * d + 4 * d) + (4 * d * d + d)
print(gomme, katman)
print(gomme + L * katman + 2 * d)
print(V * d)

Expected output

39383808 7087872
124439808
38597376

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Which sub-layer is missing in a GPT block compared with the original transformer decoder?

  • Cross-attention Correct answer
  • Masked self-attention
  • FFN
  • LayerNorm

Explanation: Since there is no encoder, there is no source sequence to look at.

2 Reuse the embedding matrix at the output.

Question 
logit = layer_norm(x) @ gomme.____
Answer 
logit = layer_norm(x) @ gomme.T

Explanation: The embedding is (V, d); its transpose (d, V) gives the logits.

3 Match.

Term Meaning
pre-LN Norm before the sub-layer
weight tying Output = embeddingᵀ
context 1024 tokens
GPT-2 small ~124M parameters

All lessons in Expert

1 Attention

  1. What is attention?
  2. Self-attention
  3. Query, Key, Value
  4. Scaled dot-product
  5. Multi-head
  6. Summary - Attention

2 Transformer

  1. What is a transformer?
  2. Encoder
  3. Decoder
  4. Positional encoding
  5. TransformerEncoderLayer
  6. Summary - Transformer

3 Project: Transformer

  1. Preparing data
  2. Attention layer
  3. Encoder block
  4. Results

4 BERT

  1. What is BERT?
  2. Masked LM
  3. Tokenization
  4. Fine-tuning
  5. CLS token
  6. Summary - BERT

5 GPT

  1. What is GPT?
  2. Autoregressive
  3. Decoder-only
  4. Causal mask
  5. Text generation
  6. Summary - GPT

6 GAN and VAE

  1. What is GAN?
  2. Generator
  3. Discriminator
  4. What is VAE?
  5. Encoder-Decoder
  6. Summary - GAN and VAE

7 Reinforcement learning

  1. What is RL?
  2. Agent
  3. Policy
  4. Reward
  5. Q-learning
  6. Summary - RL

8 Expert final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4