Attention layer
- Machine Learning course
- Expert
- Lesson 14/44
- 7 min
Add a multi-head attention layer.
Embedding and the attention layer
Ids are turned into vectors by selecting rows from an embedding table (vocabulary size × d). Then scaled attention is applied in a batch; the shapes become (batch, n, d):
E = rng.normal(size=(len(sozluk), d)) * 0.5
H = E[X] # (5, 5, d)
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d) # (5, 5, 5)In NumPy, E[X] takes E's row for each id of X: this is exactly what nn.Embedding does.
Masking the padding
The mask is applied to the key (column) axis: pad[:, None, :] has shape (batch, 1, n) and broadcasts over all query rows. Setting the scores of padding columns to −1e9 zeroes their weights:
def dikkat(Q, K, V, pad):
d = Q.shape[-1]
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, AUsing −1e9 instead of −∞ prevents NaNs in a row that is entirely padding.
Practice A masked attention layer
Complete the given dikkat template (apply the mask). Compute Q, K, V with the embedding H = E[X] and apply attention. Print the shapes of the output and the weights on one line. Then print the first weight row of the third sentence ("iyi") with np.round(..., 3) and the total weight going to padding columns with round(..., 6).
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
def dikkat(Q, K, V, pad):
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(Q.shape[-1])
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, A
H = E[X]
Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
print(Y.shape, A.shape)
print(np.round(A[2, 0], 3))
print(round(float(A[pad[:, None, :].repeat(5, axis=1)].sum()), 6))Expected output
(5, 5, 8) (5, 5, 5) [1. 0. 0. 0. 0.] 0.0
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If E is (10, 8) and X is (5, 5), what is the shape of E[X]?
- (5, 8)
- (5, 5)
- (5, 5, 8) Correct answer
- (10, 8)
Explanation: Each id becomes an 8-dimensional vector.
2 Broadcast the mask over the key axis.
skor = np.where(pad[:, ____, :], -1e9, skor)skor = np.where(pad[:, None, :], -1e9, skor)Explanation: None adds a new axis: (batch, 1, n).
3 Match the shapes (batch 5, n 5, d 8).
| Term | Meaning |
|---|---|
E[X] |
(5, 5, 8) |
score |
(5, 5, 5) |
pad[:, None, :] |
(5, 1, 5) |
E |
(9, 8) |