Machine Learning
What is attention?
Attention lets the model focus on important parts of the input.
What is attention?
Attention is the mechanism that learns how much information each element of a sequence takes from the others. The meaning of "bank" depends on whether "river" or "money" appears in the sentence; attention builds this relationship with weights.
# for each word: attention weights over the other words (summing to 1)
# new representation = weighted mean(values of the other words)
agirlik = softmax(skorlar) # (n,)
cikis = agirlik @ degerler # (d,)
Unlike an RNN, attention looks at all positions at once: the link between distant words is made in a single step and the computation can be done in parallel.
A weighted average
The core of attention is simple: turn the scores into probabilities with softmax and take the weighted average of the values. The scores measure similarity; the most similar elements contribute most:
degerler = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
skor = np.array([2.0, 0.0, 0.0])
a = softmax(skor) # [0.787 0.107 0.107]
print(a @ degerler) # [0.893 0.213]: mostly the first value
If the scores were equal, the result would be the plain mean of the values.
Exercise First attention
Compute the softmax weights for the given skor and print them with np.round(..., 3). Then print the weighted average of degerler with np.round(..., 3). Finally print with np.allclose that with equal scores (np.zeros(3)) the result is the plain mean.
Show example solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
degerler = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]])
skor = np.array([2.0, 0.0, 0.0])
a = softmax(skor)
print(np.round(a, 3))
print(np.round(a @ degerler, 3))
print(np.allclose(softmax(np.zeros(3)) @ degerler, degerler.mean(axis=0)))
Test yourself
-
What is attention's main advantage over an RNN?
- It needs no labels
- It relates distant positions in one step and is computed in parallel
- It uses less memory
- It has no parameters
Show answer
It relates distant positions in one step and is computed in parallel — In an RNN information is carried step by step; attention makes direct connections.
-
Take the weighted average of the values.
cikis = agirlik ____ degerler
Show answer
cikis = agirlik @ degerler
-
Match the concepts.
- score
- Similarity measure
- softmax
- Turns into weights
- value
- Carried information
- equal scores
- Plain mean
Next lesson: Self-attention
