Machine Learning
Challenge 2
Define a transformer encoder.
Challenge: a masked encoder
In this challenge you'll write a batched, padding-masked, pre-LN encoder block:
def encoder_blogu(H, pad):
a = layer_norm(H)
H = H + dikkat(a, pad) # pre-LN
return H + ffn(layer_norm(H))
The property a good encoder must have: padding independence. However much padding is added to the end of a sentence, the output of the real positions must not change.
Exercise A padding-independent encoder
With the given helpers write a batched dikkat(a, pad) (masked on the key axis) and a pre-LN encoder_blogu(H, pad). Give the same 3-word sentence once at length 3 and once at length 6 (3 padding). Print the output shapes on one line, then with np.allclose that the first three positions are the same in both cases.
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def layer_norm(x, eps=1e-5):
return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)
rng = np.random.default_rng(0)
d = 8
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.4
W1, W2 = rng.normal(size=(d, 4 * d)) * 0.3, rng.normal(size=(4 * d, d)) * 0.3
ffn = lambda a: np.maximum(0, a @ W1) @ W2
gercek = rng.normal(size=(1, 3, d))
H3, pad3 = gercek, np.zeros((1, 3), dtype=bool)
H6 = np.concatenate([gercek, rng.normal(size=(1, 3, d))], axis=1)
pad6 = np.array([[False] * 3 + [True] * 3])
def dikkat(a, pad):
Q, K, V = a @ Wq, a @ Wk, a @ Wv
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
skor = np.where(pad[:, None, :], -1e9, skor)
return softmax(skor) @ V
def encoder_blogu(H, pad):
H = H + dikkat(layer_norm(H), pad)
return H + ffn(layer_norm(H))
Y3, Y6 = encoder_blogu(H3, pad3), encoder_blogu(H6, pad6)
print(Y3.shape, Y6.shape)
print(np.allclose(Y3[0], Y6[0, :3]))
Exercise A model-size calculator
Write a boyut(V, n_max, d, L, f) function giving the parameter count of an encoder-only model: word + position embeddings, L layers (attention 4·(d²+d), FFN d·f+f+f·d+d, 2 LayerNorms 4d) and a final LayerNorm (2d). Print the counts for a near-BERT-base (30522, 512, 768, 12, 3072) and a small model (8000, 128, 256, 4, 1024) on one line. Then print with round(..., 3) what fraction of the parameters remains in the small model when the FFN size is 2d instead of 4d.
Show example solution
def boyut(V, n_max, d, L, f):
gomme = (V + n_max) * d
katman = 4 * (d * d + d) + (d * f + f + f * d + d) + 4 * d
return gomme + L * katman + 2 * d
print(boyut(30522, 512, 768, 12, 3072), boyut(8000, 128, 256, 4, 1024))
print(round(boyut(8000, 128, 256, 4, 512) / boyut(8000, 128, 256, 4, 1024), 3))
Test yourself
-
What is the difference between pre-LN and post-LN?
- Pre-LN doesn't normalize at all
- Post-LN uses no mask
- There is no difference
- Pre-LN normalizes before the sub-layer, outside the residual path
Show answer
Pre-LN normalizes before the sub-layer, outside the residual path — In pre-LN the residual path (H + ...) flows unnormalized; it is more stable in deep networks.
-
Write the pre-LN residual connection.
H = H + dikkat(____(H), pad)
Show answer
H = H + dikkat(layer_norm(H), pad)
-
Match.
- post-LN
- LN(H + f(H))
- pre-LN
- H + f(LN(H))
- padding mask
- Closes key columns
- padding independence
- Real output doesn't change
Next lesson: Challenge 3
