Machine Learning
Attention layer
Add a multi-head attention layer.
Embedding and the attention layer
Ids are turned into vectors by selecting rows from an embedding table (vocabulary size × d). Then scaled attention is applied in a batch; the shapes become (batch, n, d):
E = rng.normal(size=(len(sozluk), d)) * 0.5
H = E[X] # (5, 5, d)
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d) # (5, 5, 5)
In NumPy, E[X] takes E's row for each id of X: this is exactly what nn.Embedding does.
Masking the padding
The mask is applied to the key (column) axis: pad[:, None, :] has shape (batch, 1, n) and broadcasts over all query rows. Setting the scores of padding columns to −1e9 zeroes their weights:
def dikkat(Q, K, V, pad):
d = Q.shape[-1]
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, A
Using −1e9 instead of −∞ prevents NaNs in a row that is entirely padding.
Exercise A masked attention layer
Complete the given dikkat template (apply the mask). Compute Q, K, V with the embedding H = E[X] and apply attention. Print the shapes of the output and the weights on one line. Then print the first weight row of the third sentence ("iyi") with np.round(..., 3) and the total weight going to padding columns with round(..., 6).
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
def dikkat(Q, K, V, pad):
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(Q.shape[-1])
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, A
H = E[X]
Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
print(Y.shape, A.shape)
print(np.round(A[2, 0], 3))
print(round(float(A[pad[:, None, :].repeat(5, axis=1)].sum()), 6))
Test yourself
-
If E is (10, 8) and X is (5, 5), what is the shape of E[X]?
- (5, 8)
- (5, 5)
- (5, 5, 8)
- (10, 8)
Show answer
(5, 5, 8) — Each id becomes an 8-dimensional vector.
-
Broadcast the mask over the key axis.
skor = np.where(pad[:, ____, :], -1e9, skor)
Show answer
skor = np.where(pad[:, None, :], -1e9, skor)
-
Match the shapes (batch 5, n 5, d 8).
- E[X]
- (5, 5, 8)
- score
- (5, 5, 5)
- pad[:, None, :]
- (5, 1, 5)
- E
- (9, 8)
Next lesson: Encoder block
