What is LSTM?
- Machine Learning course
- Advanced
- Lesson 53/78
- 5 min
LSTM is an RNN type that preserves memory over long sequences.
What is an LSTM?
Long short-term memory (LSTM) is a kind of RNN developed against the vanishing gradient problem. In addition to the hidden state it carries a cell state, and controls with learned gates how much information is written to this cell, how much is kept and how much is read.
lstm = nn.LSTM(input_size=8, hidden_size=16, batch_first=True)
output, (h_n, c_n) = lstm(torch.zeros(2, 5, 8))
print(output.shape, h_n.shape, c_n.shape)
# torch.Size([2, 5, 16]) torch.Size([1, 2, 16]) torch.Size([1, 2, 16])Before the transformer, LSTM was the most common architecture for text, speech and time series; today it is still a strong option for small sequence problems.
Why does it work?
The cell state is updated from step to step mostly by addition: c_t = f_t ⊙ c_(t-1) + i_t ⊙ g_t. When the forget gate f ≈ 1, information and gradient flow almost unchanged for a long time; the repeated multiplication chain of the RNN is broken.
This is the same idea as the residual connection in ResNet: opening an additive rather than multiplicative path for information.
Practice LSTM shapes
Apply an nn.LSTM(8, 16, batch_first=True) layer to a (2, 5, 8) input. Print the output, h_n and c_n shapes on separate lines, then with torch.allclose that output[:, -1] equals h_n[0].
Starter code
import torch
from torch import nn
x = torch.randn(2, 5, 8)
# buraya yazSample solution
import torch
from torch import nn
x = torch.randn(2, 5, 8)
lstm = nn.LSTM(8, 16, batch_first=True)
output, (h_n, c_n) = lstm(x)
print(output.shape)
print(h_n.shape)
print(c_n.shape)
print(torch.allclose(output[:, -1], h_n[0]))Expected output
torch.Size([2, 5, 16]) torch.Size([1, 2, 16]) torch.Size([1, 2, 16]) True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does an <code>nn.LSTM</code> call return differently from <code>nn.RNN</code>?
- Only output
- Three separate outputs
- Nothing
- output together with an (h_n, c_n) pair Correct answer
Explanation: An LSTM carries the hidden state h and the cell state c.
2 Unpack the LSTM output.
output, (h_n, ____) = lstm(x)output, (h_n, c_n) = lstm(x)Explanation: The second element is the (h_n, c_n) pair.
3 Match the concepts.
| Term | Meaning |
|---|---|
h |
Hidden state (output) |
c |
Cell state (memory) |
gate |
Controls information flow |
additive update |
Reduces vanishing |