Machine Learning
Query, Key, Value
Attention computes weights from query-key products and sums values.
Query, Key, Value
Each element takes on three different roles; each role is produced with a learned matrix:
- Query (Q = X W_q): "what am I looking for?"
- Key (K = X W_k): "what do I have?"
- Value (V = X W_v): "what do I give if I'm found?"
Q = X @ W_q # (n, d_k)
K = X @ W_k # (n, d_k)
V = X @ W_v # (n, d_v)
A = softmax(Q @ K.T, eksen=1)
Y = A @ V
Thanks to separate transformations, what an element "looks for" can differ from what it "offers": a verb can look for its subject while itself offering something else.
The search-engine analogy
Attention is a soft dictionary lookup: the query is compared with all the keys and the values are mixed in proportion to the match. If one key matches perfectly, the result is almost that key's value:
K = np.array([[10., 0.], [0., 10.]])
V = np.array([[1., 2.], [3., 4.]])
q = np.array([1., 0.]) # resembles the first key
print(np.round(softmax(q @ K.T) @ V, 3)) # [1. 2.]: the first value
In a classic dictionary a lookup is "exact match or nothing"; attention is its differentiable version.
Exercise Q, K, V
Compute Q, K and V with X (3 elements, 4 dimensions) and the given, as-if-learned W_q, W_k, W_v. Find the attention matrix with a per-row softmax; print the shapes (Q.shape, A.shape, Y.shape) on one line, then A's first row with np.round(..., 3).
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
X = np.array([[1., 0., 1., 0.], [0., 1., 0., 1.], [1., 1., 0., 0.]])
W_q = np.eye(4)[:, :2]
W_k = np.eye(4)[:, 1:3]
W_v = np.eye(4)
Q, K, V = X @ W_q, X @ W_k, X @ W_v
A = softmax(Q @ K.T, eksen=1)
Y = A @ V
print(Q.shape, A.shape, Y.shape)
print(np.round(A[0], 3))
Test yourself
-
From which two parts is how much one element attends to another computed?
- Both Values
- Query and Value
- Only the Key
- One's Query and the other's Key
Show answer
One's Query and the other's Key — Score = q_i · k_j; the Value is only the carried information.
-
Compare the queries with the keys.
skor = Q @ ____
Show answer
skor = Q @ K.T
-
Match the roles.
- Query
- What am I looking for?
- Key
- What do I have?
- Value
- What do I give?
- W_q, W_k, W_v
- Learned transformations
Next lesson: Scaled dot-product
