Encoder
- Machine Learning course
- Expert
- Lesson 8/44
- 5 min
The encoder processes input with attention and feed-forward layers.
The encoder
The encoder sees the whole input sequence and produces a context-aware representation for each position. Its block has two steps: self-attention (positions exchange information) and FFN (each position is transformed separately):
def ffn(x, W1, W2):
return np.maximum(0, x @ W1) @ W2 # d -> 4d -> d
def encoder_blogu(x, Wq, Wk, Wv, W1, W2):
d = x.shape[-1]
A = softmax((x @ Wq) @ (x @ Wk).T / np.sqrt(d))
x = layer_norm(x + A @ (x @ Wv))
return layer_norm(x + ffn(x, W1, W2))The FFN's hidden size is usually 4 times d_model; most of the parameters live here.
The FFN is per position
Applying the FFN row by row and applying it to the whole matrix at once give the same result, because the same weights are applied to each position separately. So changing one position changes only that row of the FFN output:
y1 = ffn(x, W1, W2)
x2 = x.copy(); x2[0] += 5
y2 = ffn(x2, W1, W2)
print(np.allclose(y1[1:], y2[1:])) # True: the other rows are the sameIn the attention layer, however, changing a single position affects all outputs.
Practice An encoder block
Complete the encoder_blogu function with the given helpers. Print the output's shape. Then print on one line with np.allclose that when only the first row of x changes, the other rows stay the same in the FFN output but change in the encoder block's output.
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
def ffn(x, W1, W2):
return np.maximum(0, x @ W1) @ W2
rng = np.random.default_rng(0)
n, d = 4, 8
x = rng.normal(size=(n, d))
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.3
W1 = rng.normal(size=(d, 4 * d)) * 0.3
W2 = rng.normal(size=(4 * d, d)) * 0.3
x2 = x.copy()
x2[0] += 5
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
def layer_norm(x, eps=1e-5):
m = x.mean(axis=-1, keepdims=True)
v = x.var(axis=-1, keepdims=True)
return (x - m) / np.sqrt(v + eps)
def ffn(x, W1, W2):
return np.maximum(0, x @ W1) @ W2
rng = np.random.default_rng(0)
n, d = 4, 8
x = rng.normal(size=(n, d))
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.3
W1 = rng.normal(size=(d, 4 * d)) * 0.3
W2 = rng.normal(size=(4 * d, d)) * 0.3
x2 = x.copy()
x2[0] += 5
def encoder_blogu(x):
A = softmax((x @ Wq) @ (x @ Wk).T / np.sqrt(d))
x = layer_norm(x + A @ (x @ Wv))
return layer_norm(x + ffn(x, W1, W2))
print(encoder_blogu(x).shape)
print(np.allclose(ffn(x, W1, W2)[1:], ffn(x2, W1, W2)[1:]), np.allclose(encoder_blogu(x)[1:], encoder_blogu(x2)[1:]))Expected output
(4, 8) True False
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Does the FFN in the encoder carry information across positions?
- No, it is applied to each position separately Correct answer
- Yes, it mixes all positions
- Only neighbouring positions
- Only the first position
Explanation: Mixing across positions happens only in attention.
2 Apply ReLU in the FFN's hidden layer.
return np.____(0, x @ W1) @ W2return np.maximum(0, x @ W1) @ W2Explanation: np.maximum compares element-wise; np.max returns a single value.
3 Match the effects.
| Term | Meaning |
|---|---|
x[0] changes → FFN |
Only row 0 changes |
x[0] changes → attention |
All rows change |
W1 |
d → 4d |
W2 |
4d → d |