Query, Key, Value
- Machine Learning course
- Expert
- Lesson 3/44
- 5 min
Attention computes weights from query-key products and sums values.
Query, Key, Value
Each element takes on three different roles; each role is produced with a learned matrix:
- Query (Q = X W_q): "what am I looking for?"
- Key (K = X W_k): "what do I have?"
- Value (V = X W_v): "what do I give if I'm found?"
Q = X @ W_q # (n, d_k)
K = X @ W_k # (n, d_k)
V = X @ W_v # (n, d_v)
A = softmax(Q @ K.T, eksen=1)
Y = A @ VThanks to separate transformations, what an element "looks for" can differ from what it "offers": a verb can look for its subject while itself offering something else.
The search-engine analogy
Attention is a soft dictionary lookup: the query is compared with all the keys and the values are mixed in proportion to the match. If one key matches perfectly, the result is almost that key's value:
K = np.array([[10., 0.], [0., 10.]])
V = np.array([[1., 2.], [3., 4.]])
q = np.array([1., 0.]) # resembles the first key
print(np.round(softmax(q @ K.T) @ V, 3)) # [1. 2.]: the first valueIn a classic dictionary a lookup is "exact match or nothing"; attention is its differentiable version.
Practice Q, K, V
Compute Q, K and V with X (3 elements, 4 dimensions) and the given, as-if-learned W_q, W_k, W_v. Find the attention matrix with a per-row softmax; print the shapes (Q.shape, A.shape, Y.shape) on one line, then A's first row with np.round(..., 3).
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
X = np.array([[1., 0., 1., 0.], [0., 1., 0., 1.], [1., 1., 0., 0.]])
W_q = np.eye(4)[:, :2]
W_k = np.eye(4)[:, 1:3]
W_v = np.eye(4)
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
X = np.array([[1., 0., 1., 0.], [0., 1., 0., 1.], [1., 1., 0., 0.]])
W_q = np.eye(4)[:, :2]
W_k = np.eye(4)[:, 1:3]
W_v = np.eye(4)
Q, K, V = X @ W_q, X @ W_k, X @ W_v
A = softmax(Q @ K.T, eksen=1)
Y = A @ V
print(Q.shape, A.shape, Y.shape)
print(np.round(A[0], 3))Expected output
(3, 2) (3, 3) (3, 4) [0.155 0.422 0.422]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 From which two parts is how much one element attends to another computed?
- Both Values
- Query and Value
- Only the Key
- One's Query and the other's Key Correct answer
Explanation: Score = q_i · k_j; the Value is only the carried information.
2 Compare the queries with the keys.
skor = Q @ ____skor = Q @ K.TExplanation: Q Kᵀ is the dot product of every query-key pair.
3 Match the roles.
| Term | Meaning |
|---|---|
Query |
What am I looking for? |
Key |
What do I have? |
Value |
What do I give? |
W_q, W_k, W_v |
Learned transformations |