Encoder block
- Machine Learning course
- Expert
- Lesson 15/44
- 7 min
Define a transformer encoder block.
Building the encoder block
We wrap the attention layer with a residual connection, LayerNorm and FFN to make a full encoder block. A positional code is also added to the input:
def encoder(X, pad):
H = E[X] + konum_kodu(X.shape[1], d)
Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
H = layer_norm(H + Y)
H = layer_norm(H + np.maximum(0, H @ W1) @ W2)
return H, AThe output is a contextual vector for each word; the representation of "iyi" now also depends on the rest of the sentence.
The sentence vector: a masked mean
Classification needs a single vector per sentence. A simple way is averaging only the real words (BERT uses the [CLS] token instead):
m = (~pad)[:, :, None] # (5, 5, 1)
cumle = (H * m).sum(axis=1) / m.sum(axis=1) # (5, d)Including padding in the mean would pull short sentences' vectors toward the padding's representation.
Practice The encoder and the sentence vector
Complete the encoder(X, pad) function and compute the sentence vectors with a masked mean. Print the shapes of H and the sentence vectors on one line. Then print on one line with np.allclose that the representations of "iyi" in sentence 1 (position 2) and sentence 3 (position 0) differ, and with np.allclose that padding doesn't affect the sentence vector (by changing the padding embedding and recomputing).
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
def dikkat(Q, K, V, pad):
d = Q.shape[-1]
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, A
def layer_norm(x, eps=1e-5):
return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)
def konum_kodu(n, d):
aci = np.arange(n)[:, None] / 10000 ** (np.arange(0, d, 2) / d)
PE = np.zeros((n, d))
PE[:, 0::2], PE[:, 1::2] = np.sin(aci), np.cos(aci)
return PE
rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
W1 = rng.normal(size=(d, 4 * d)) * 0.3
W2 = rng.normal(size=(4 * d, d)) * 0.3
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
def dikkat(Q, K, V, pad):
d = Q.shape[-1]
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, A
def layer_norm(x, eps=1e-5):
return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)
def konum_kodu(n, d):
aci = np.arange(n)[:, None] / 10000 ** (np.arange(0, d, 2) / d)
PE = np.zeros((n, d))
PE[:, 0::2], PE[:, 1::2] = np.sin(aci), np.cos(aci)
return PE
rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
W1 = rng.normal(size=(d, 4 * d)) * 0.3
W2 = rng.normal(size=(4 * d, d)) * 0.3
def encoder(X, pad):
H = E[X] + konum_kodu(X.shape[1], d)
Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
H = layer_norm(H + Y)
return layer_norm(H + np.maximum(0, H @ W1) @ W2)
def cumle_vektoru(H, pad):
m = (~pad)[:, :, None]
return (H * m).sum(axis=1) / m.sum(axis=1)
H = encoder(X, pad)
C = cumle_vektoru(H, pad)
print(H.shape, C.shape)
E[0] += 10
C2 = cumle_vektoru(encoder(X, pad), pad)
print(np.allclose(H[0, 2], H[2, 0]), np.allclose(C, C2))Expected output
(5, 5, 8) (5, 8) False True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Is the encoder output for "iyi" the same in the sentences "film çok iyi" and "iyi"?
- The encoder doesn't look at the word
- No; it changes because the context and position differ Correct answer
- Yes, the same word is the same vector
- Only the padding differs
Explanation: The embedding is the same, but attention and the positional code change the representation according to context.
2 Sum only the real words.
cumle = (H * m).sum(axis=1) / m.____(axis=1)cumle = (H * m).sum(axis=1) / m.sum(axis=1)Explanation: The divisor is the number of real words in the sentence.
3 Match.
| Term | Meaning |
|---|---|
H + Y |
Residual connection |
konum_kodu |
Order information |
maskeli ortalama |
Sentence vector |
[CLS] |
BERT's sentence token |