Machine Learning
Vanishing gradient
RNNs suffer from vanishing gradients on long sequences.
Vanishing gradient
As backpropagation goes back through time, at each step the gradient is multiplied by W_hh and the tanh derivative. If these factors are below 1, the gradient shrinks exponentially and information in early steps can't be learned:
g = 1.0
for adim in range(50):
g *= 0.9 # a factor of 0.9 at every step
print(round(g, 4)) # 0.0052: almost no signal reaches the input 50 steps back
If the factors are above 1 the gradient explodes; gradient clipping is used against that. Against vanishing, gated architectures like LSTM and GRU were developed.
Observing it
Vanishing can be seen directly by comparing the gradient of an RNN's last output with respect to the first and last inputs:
x = torch.randn(1, 50, 1, requires_grad=True)
output, _ = rnn(x)
output[0, -1].sum().backward()
print(x.grad[0, 0].abs().item()) # first step: very small
print(x.grad[0, -1].abs().item()) # last step: large
If the gradient at the first step is many times smaller than at the last, the model can't learn long-range dependencies.
Exercise Measure the vanishing
Build nn.RNN(1, 8, batch_first=True) with seed 0. Track the gradient of a random (1, 40, 1) input; backpropagate the sum of the last step's output. Print that the gradient size at the first step is less than one hundredth of that at the last step.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
rnn = nn.RNN(1, 8, batch_first=True)
x = torch.randn(1, 40, 1, requires_grad=True)
output, _ = rnn(x)
output[0, -1].sum().backward()
ilk = x.grad[0, 0].abs().item()
son = x.grad[0, -1].abs().item()
print(ilk < son / 100)
Test yourself
-
What happens after 100 steps if the gradient is multiplied by 1.1 at every step?
- It stays the same
- It becomes zero
- It explodes (grows ~13780 times)
- It vanishes
Show answer
It explodes (grows ~13780 times) — 1.1¹⁰⁰ ≈ 13780; that's why gradient clipping is needed.
-
Multiply the gradient across 50 steps.
g = 0.9 ____ 50
Show answer
g = 0.9 ** 50
-
Match the situations.
- factor < 1
- Vanishing gradient
- factor > 1
- Exploding gradient
- gradient clipping
- Fix for explosion
- LSTM/GRU
- Fix for vanishing
Next lesson: Embedding
