Machine Learning
Encoder
The encoder processes input with attention and feed-forward layers.
The encoder
The encoder sees the whole input sequence and produces a context-aware representation for each position. Its block has two steps: self-attention (positions exchange information) and FFN (each position is transformed separately):
def ffn(x, W1, W2):
return np.maximum(0, x @ W1) @ W2 # d -> 4d -> d
def encoder_blogu(x, Wq, Wk, Wv, W1, W2):
d = x.shape[-1]
A = softmax((x @ Wq) @ (x @ Wk).T / np.sqrt(d))
x = layer_norm(x + A @ (x @ Wv))
return layer_norm(x + ffn(x, W1, W2))
The FFN's hidden size is usually 4 times d_model; most of the parameters live here.
The FFN is per position
Applying the FFN row by row and applying it to the whole matrix at once give the same result, because the same weights are applied to each position separately. So changing one position changes only that row of the FFN output:
y1 = ffn(x, W1, W2)
x2 = x.copy(); x2[0] += 5
y2 = ffn(x2, W1, W2)
print(np.allclose(y1[1:], y2[1:])) # True: the other rows are the same
In the attention layer, however, changing a single position affects all outputs.
Exercise An encoder block
Complete the encoder_blogu function with the given helpers. Print the output's shape. Then print on one line with np.allclose that when only the first row of x changes, the other rows stay the same in the FFN output but change in the encoder block's output.
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
def ffn(x, W1, W2):
return np.maximum(0, x @ W1) @ W2
rng = np.random.default_rng(0)
n, d = 4, 8
x = rng.normal(size=(n, d))
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.3
W1 = rng.normal(size=(d, 4 * d)) * 0.3
W2 = rng.normal(size=(4 * d, d)) * 0.3
x2 = x.copy()
x2[0] += 5
def encoder_blogu(x):
A = softmax((x @ Wq) @ (x @ Wk).T / np.sqrt(d))
x = layer_norm(x + A @ (x @ Wv))
return layer_norm(x + ffn(x, W1, W2))
print(encoder_blogu(x).shape)
print(np.allclose(ffn(x, W1, W2)[1:], ffn(x2, W1, W2)[1:]), np.allclose(encoder_blogu(x)[1:], encoder_blogu(x2)[1:]))
Test yourself
-
Does the FFN in the encoder carry information across positions?
- No, it is applied to each position separately
- Yes, it mixes all positions
- Only neighbouring positions
- Only the first position
Show answer
No, it is applied to each position separately — Mixing across positions happens only in attention.
-
Apply ReLU in the FFN's hidden layer.
return np.____(0, x @ W1) @ W2
Show answer
return np.maximum(0, x @ W1) @ W2
-
Match the effects.
- x[0] changes → FFN
- Only row 0 changes
- x[0] changes → attention
- All rows change
- W1
- d → 4d
- W2
- 4d → d
Next lesson: Decoder
