Summary - Attention
- Machine Learning course
- Expert
- Lesson 6/44
- 6 min
In this group you learned self-attention, QKV, and multi-head attention.
Summary: attention
In this group:
- Attention takes the weighted average of values with the softmax weights of scores.
- In self-attention every element looks at the whole sequence; the cost grows with n².
- Q, K, V are separately learned transformations: the score is q·k, the carried information v.
- Scaled dot product:
softmax(QKᵀ/√d_k) V; scaling prevents softmax saturation. - Multi-head attention splits the representation into h heads and learns different relationships in parallel.
Practice Compare with PyTorch
Write the attention function in NumPy. For the given small Q, K, V, print the result with np.round(..., 4). Then check that the first row you computed by hand is softmax([1/√2, 1/√2]) @ V[:, 0] and @ V[:, 1], i.e. that it comes out as [5, 5]: print the result of np.allclose(Y[0], [5, 5]).
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
Q = np.array([[1., 1.], [2., 0.]])
K = np.array([[1., 0.], [0., 1.]])
V = np.array([[10., 0.], [0., 10.]])
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
Q = np.array([[1., 1.], [2., 0.]])
K = np.array([[1., 0.], [0., 1.]])
V = np.array([[10., 0.], [0., 10.]])
def attention(Q, K, V):
A = softmax(Q @ K.T / np.sqrt(Q.shape[-1]), eksen=-1)
return A @ V
Y = attention(Q, K, V)
print(np.round(Y, 4))
print(np.allclose(Y[0], [5, 5]))Expected output
[[5. 5. ] [8.0443 1.9557]] True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If the sequence length grows 4 times, roughly how many times does self-attention's cost grow?
- 16 Correct answer
- 4
- 2
- 64
Explanation: The cost is n²: 4² = 16.
2 Complete the transformer attention formula.
Question
Y = softmax(Q @ K.T / np.sqrt(d_k)) ____ V
Answer
Y = softmax(Q @ K.T / np.sqrt(d_k)) @ VExplanation: The weights enter a matrix product with the values.
3 Match the concepts.
| Term | Meaning |
|---|---|
QKᵀ |
Similarity scores |
√d_k |
Scale |
softmax |
Row probabilities |
multi-head |
Parallel relationships |