What is attention?
- Machine Learning course
- Expert
- Lesson 1/44
- 5 min
Attention lets the model focus on important parts of the input.
What is attention?
Attention is the mechanism that learns how much information each element of a sequence takes from the others. The meaning of "bank" depends on whether "river" or "money" appears in the sentence; attention builds this relationship with weights.
# for each word: attention weights over the other words (summing to 1)
# new representation = weighted mean(values of the other words)
agirlik = softmax(skorlar) # (n,)
cikis = agirlik @ degerler # (d,)Unlike an RNN, attention looks at all positions at once: the link between distant words is made in a single step and the computation can be done in parallel.
A weighted average
The core of attention is simple: turn the scores into probabilities with softmax and take the weighted average of the values. The scores measure similarity; the most similar elements contribute most:
degerler = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
skor = np.array([2.0, 0.0, 0.0])
a = softmax(skor) # [0.787 0.107 0.107]
print(a @ degerler) # [0.893 0.213]: mostly the first valueIf the scores were equal, the result would be the plain mean of the values.
Practice First attention
Compute the softmax weights for the given skor and print them with np.round(..., 3). Then print the weighted average of degerler with np.round(..., 3). Finally print with np.allclose that with equal scores (np.zeros(3)) the result is the plain mean.
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
degerler = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
skor = np.array([2.0, 0.0, 0.0])
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
degerler = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
skor = np.array([2.0, 0.0, 0.0])
a = softmax(skor)
print(np.round(a, 3))
print(np.round(a @ degerler, 3))
print(np.allclose(softmax(np.zeros(3)) @ degerler, degerler.mean(axis=0)))Expected output
[0.787 0.107 0.107] [0.893 0.213] True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is attention's main advantage over an RNN?
- It needs no labels
- It relates distant positions in one step and is computed in parallel Correct answer
- It uses less memory
- It has no parameters
Explanation: In an RNN information is carried step by step; attention makes direct connections.
2 Take the weighted average of the values.
cikis = agirlik ____ degerlercikis = agirlik @ degerlerExplanation: The product of the weight vector and the value matrix is the weighted sum.
3 Match the concepts.
| Term | Meaning |
|---|---|
score |
Similarity measure |
softmax |
Turns into weights |
value |
Carried information |
equal scores |
Plain mean |