Vanishing gradient
- Machine Learning course
- Advanced
- Lesson 50/78
- 5 min
RNNs suffer from vanishing gradients on long sequences.
Vanishing gradient
As backpropagation goes back through time, at each step the gradient is multiplied by W_hh and the tanh derivative. If these factors are below 1, the gradient shrinks exponentially and information in early steps can't be learned:
g = 1.0
for adim in range(50):
g *= 0.9 # a factor of 0.9 at every step
print(round(g, 4)) # 0.0052: almost no signal reaches the input 50 steps backIf the factors are above 1 the gradient explodes; gradient clipping is used against that. Against vanishing, gated architectures like LSTM and GRU were developed.
Observing it
Vanishing can be seen directly by comparing the gradient of an RNN's last output with respect to the first and last inputs:
x = torch.randn(1, 50, 1, requires_grad=True)
output, _ = rnn(x)
output[0, -1].sum().backward()
print(x.grad[0, 0].abs().item()) # first step: very small
print(x.grad[0, -1].abs().item()) # last step: largeIf the gradient at the first step is many times smaller than at the last, the model can't learn long-range dependencies.
Practice Measure the vanishing
Build nn.RNN(1, 8, batch_first=True) with seed 0. Track the gradient of a random (1, 40, 1) input; backpropagate the sum of the last step's output. Print that the gradient size at the first step is less than one hundredth of that at the last step.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
rnn = nn.RNN(1, 8, batch_first=True)
x = torch.randn(1, 40, 1, requires_grad=True)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
rnn = nn.RNN(1, 8, batch_first=True)
x = torch.randn(1, 40, 1, requires_grad=True)
output, _ = rnn(x)
output[0, -1].sum().backward()
ilk = x.grad[0, 0].abs().item()
son = x.grad[0, -1].abs().item()
print(ilk < son / 100)Expected output
True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens after 100 steps if the gradient is multiplied by 1.1 at every step?
- It stays the same
- It becomes zero
- It explodes (grows ~13780 times) Correct answer
- It vanishes
Explanation: 1.1¹⁰⁰ ≈ 13780; that's why gradient clipping is needed.
2 Multiply the gradient across 50 steps.
g = 0.9 ____ 50g = 0.9 ** 50Explanation: 0.9⁵⁰ shows exponential decay.
3 Match the situations.
| Term | Meaning |
|---|---|
factor < 1 |
Vanishing gradient |
factor > 1 |
Exploding gradient |
gradient clipping |
Fix for explosion |
LSTM/GRU |
Fix for vanishing |