CodeBlanks Logo Icon CodeBlanks
Login Register

Challenge 2

Define a transformer encoder.

Challenge: a masked encoder

In this challenge you'll write a batched, padding-masked, pre-LN encoder block:

def encoder_blogu(H, pad):
    a = layer_norm(H)
    H = H + dikkat(a, pad)                 # pre-LN
    return H + ffn(layer_norm(H))

The property a good encoder must have: padding independence. However much padding is added to the end of a sentence, the output of the real positions must not change.

Practice A padding-independent encoder

With the given helpers write a batched dikkat(a, pad) (masked on the key axis) and a pre-LN encoder_blogu(H, pad). Give the same 3-word sentence once at length 3 and once at length 6 (3 padding). Print the output shapes on one line, then with np.allclose that the first three positions are the same in both cases.

Starter code

model.py
import numpy as np

def softmax(z, eksen=-1):
    e = np.exp(z - z.max(axis=eksen, keepdims=True))
    return e / e.sum(axis=eksen, keepdims=True)

def layer_norm(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)

rng = np.random.default_rng(0)
d = 8
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.4
W1, W2 = rng.normal(size=(d, 4 * d)) * 0.3, rng.normal(size=(4 * d, d)) * 0.3
ffn = lambda a: np.maximum(0, a @ W1) @ W2

gercek = rng.normal(size=(1, 3, d))
H3, pad3 = gercek, np.zeros((1, 3), dtype=bool)
H6 = np.concatenate([gercek, rng.normal(size=(1, 3, d))], axis=1)
pad6 = np.array([[False] * 3 + [True] * 3])

# buraya yaz

Sample solution

model.py
import numpy as np

def softmax(z, eksen=-1):
    e = np.exp(z - z.max(axis=eksen, keepdims=True))
    return e / e.sum(axis=eksen, keepdims=True)

def layer_norm(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)

rng = np.random.default_rng(0)
d = 8
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.4
W1, W2 = rng.normal(size=(d, 4 * d)) * 0.3, rng.normal(size=(4 * d, d)) * 0.3
ffn = lambda a: np.maximum(0, a @ W1) @ W2

gercek = rng.normal(size=(1, 3, d))
H3, pad3 = gercek, np.zeros((1, 3), dtype=bool)
H6 = np.concatenate([gercek, rng.normal(size=(1, 3, d))], axis=1)
pad6 = np.array([[False] * 3 + [True] * 3])

def dikkat(a, pad):
    Q, K, V = a @ Wq, a @ Wk, a @ Wv
    skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
    skor = np.where(pad[:, None, :], -1e9, skor)
    return softmax(skor) @ V

def encoder_blogu(H, pad):
    H = H + dikkat(layer_norm(H), pad)
    return H + ffn(layer_norm(H))

Y3, Y6 = encoder_blogu(H3, pad3), encoder_blogu(H6, pad6)
print(Y3.shape, Y6.shape)
print(np.allclose(Y3[0], Y6[0, :3]))

Expected output

(1, 3, 8) (1, 6, 8)
True

Practice A model-size calculator

Write a boyut(V, n_max, d, L, f) function giving the parameter count of an encoder-only model: word + position embeddings, L layers (attention 4·(d²+d), FFN d·f+f+f·d+d, 2 LayerNorms 4d) and a final LayerNorm (2d). Print the counts for a near-BERT-base (30522, 512, 768, 12, 3072) and a small model (8000, 128, 256, 4, 1024) on one line. Then print with round(..., 3) what fraction of the parameters remains in the small model when the FFN size is 2d instead of 4d.

Starter code

model.py
# buraya yaz

Sample solution

model.py
def boyut(V, n_max, d, L, f):
    gomme = (V + n_max) * d
    katman = 4 * (d * d + d) + (d * f + f + f * d + d) + 4 * d
    return gomme + L * katman + 2 * d

print(boyut(30522, 512, 768, 12, 3072), boyut(8000, 128, 256, 4, 1024))
print(round(boyut(8000, 128, 256, 4, 512) / boyut(8000, 128, 256, 4, 1024), 3))

Expected output

108890112 5240320
0.8

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the difference between pre-LN and post-LN?

  • Pre-LN doesn't normalize at all
  • Post-LN uses no mask
  • There is no difference
  • Pre-LN normalizes before the sub-layer, outside the residual path Correct answer

Explanation: In pre-LN the residual path (H + ...) flows unnormalized; it is more stable in deep networks.

2 Write the pre-LN residual connection.

Question 
H = H + dikkat(____(H), pad)
Answer 
H = H + dikkat(layer_norm(H), pad)

Explanation: In pre-LN the sub-layer's input is normalized.

3 Match.

Term Meaning
post-LN LN(H + f(H))
pre-LN H + f(LN(H))
padding mask Closes key columns
padding independence Real output doesn't change

All lessons in Expert

1 Attention

  1. What is attention?
  2. Self-attention
  3. Query, Key, Value
  4. Scaled dot-product
  5. Multi-head
  6. Summary - Attention

2 Transformer

  1. What is a transformer?
  2. Encoder
  3. Decoder
  4. Positional encoding
  5. TransformerEncoderLayer
  6. Summary - Transformer

3 Project: Transformer

  1. Preparing data
  2. Attention layer
  3. Encoder block
  4. Results

4 BERT

  1. What is BERT?
  2. Masked LM
  3. Tokenization
  4. Fine-tuning
  5. CLS token
  6. Summary - BERT

5 GPT

  1. What is GPT?
  2. Autoregressive
  3. Decoder-only
  4. Causal mask
  5. Text generation
  6. Summary - GPT

6 GAN and VAE

  1. What is GAN?
  2. Generator
  3. Discriminator
  4. What is VAE?
  5. Encoder-Decoder
  6. Summary - GAN and VAE

7 Reinforcement learning

  1. What is RL?
  2. Agent
  3. Policy
  4. Reward
  5. Q-learning
  6. Summary - RL

8 Expert final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4