Summary - Attention

In this group you learned self-attention, QKV, and multi-head attention.

Summary: attention

In this group:

Attention takes the weighted average of values with the softmax weights of scores.
In self-attention every element looks at the whole sequence; the cost grows with n².
Q, K, V are separately learned transformations: the score is q·k, the carried information v.
Scaled dot product: softmax(QKᵀ/√d_k) V; scaling prevents softmax saturation.
Multi-head attention splits the representation into h heads and learns different relationships in parallel.

Exercise Compare with PyTorch

Write the attention function in NumPy. For the given small Q, K, V, print the result with np.round(..., 4). Then check that the first row you computed by hand is softmax([1/√2, 1/√2]) @ V[:, 0] and @ V[:, 1], i.e. that it comes out as [5, 5]: print the result of np.allclose(Y[0], [5, 5]).

Show example solution

import numpy as np

def softmax(z, eksen=-1):
    e = np.exp(z - z.max(axis=eksen, keepdims=True))
    return e / e.sum(axis=eksen, keepdims=True)

Q = np.array([[1., 1.], [2., 0.]])
K = np.array([[1., 0.], [0., 1.]])
V = np.array([[10., 0.], [0., 10.]])

def attention(Q, K, V):
    A = softmax(Q @ K.T / np.sqrt(Q.shape[-1]), eksen=-1)
    return A @ V

Y = attention(Q, K, V)
print(np.round(Y, 4))
print(np.allclose(Y[0], [5, 5]))

Test yourself

If the sequence length grows 4 times, roughly how many times does self-attention's cost grow?
- 16
- 4
- 2
- 64
Show answer

16 — The cost is n²: 4² = 16.

Complete the transformer attention formula.

Y = softmax(Q @ K.T / np.sqrt(d_k)) ____ V

Show answer

Y = softmax(Q @ K.T / np.sqrt(d_k)) @ V

Match the concepts.

QKᵀ

Similarity scores
√d_k

Scale
softmax

Row probabilities
multi-head

Parallel relationships

Next lesson: What is a transformer?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies