Self-attention
- Machine Learning course
- Expert
- Lesson 2/44
- 5 min
Self-attention computes the relationship of each element with all others.
Self-attention
In self-attention every element of a sequence attends to all elements of the same sequence (including itself). For a sequence of n elements an n × n weight matrix forms; each row is one element's distribution:
X = np.random.default_rng(0).normal(size=(4, 8)) # 4 elements, 8 dimensions
skor = X @ X.T # (4, 4) similarities
A = softmax(skor, eksen=1) # probability per row
Y = A @ X # (4, 8) new representationsThe computation grows with n²: a 1000-word text means a matrix of a million. This is the core cost of long-context models.
Properties of the attention matrix
Because softmax is applied per row, every row sums to 1 and all elements are positive. Since similarity is highest with itself (a vector's dot product with itself), the diagonal is usually dominant:
print(A.sum(axis=1)) # [1. 1. 1. 1.]
print((A.argmax(axis=1) == np.arange(4)).all()) # each element looks at itself mostIn real models Q, K and V are separately learned transformations, so this diagonal dominance is broken; you'll see this in the next lesson.
Practice Self-attention
For the given X, compute the attention matrix from the X @ X.T scores with a per-row softmax. Print its shape and the row sums (np.round(..., 3)) on separate lines, then that every element attends most to itself (argmax on the diagonal).
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
X = np.random.default_rng(0).normal(size=(4, 8))
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
X = np.random.default_rng(0).normal(size=(4, 8))
A = softmax(X @ X.T, eksen=1)
print(A.shape)
print(np.round(A.sum(axis=1), 3))
print((A.argmax(axis=1) == np.arange(4)).all())Expected output
(4, 4) [1. 1. 1. 1.] True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How many elements does the attention matrix have for a 512-token sequence?
- 262144 Correct answer
- 512
- 1024
- 5120
Explanation: 512 × 512 = 262144; doubling the length quadruples the cost.
2 Compute the similarity of all pairs.
skor = X @ X.____skor = X @ X.TExplanation: X Xᵀ is the dot product of every pair of rows.
3 Match the shapes (n=4, d=8).
| Term | Meaning |
|---|---|
X |
(4, 8) |
X @ X.T |
(4, 4) |
A.sum(axis=1) |
All ones |
A[0] |
(4,) |