Machine Learning
What is LSTM?
LSTM is an RNN type that preserves memory over long sequences.
What is an LSTM?
Long short-term memory (LSTM) is a kind of RNN developed against the vanishing gradient problem. In addition to the hidden state it carries a cell state, and controls with learned gates how much information is written to this cell, how much is kept and how much is read.
lstm = nn.LSTM(input_size=8, hidden_size=16, batch_first=True)
output, (h_n, c_n) = lstm(torch.zeros(2, 5, 8))
print(output.shape, h_n.shape, c_n.shape)
# torch.Size([2, 5, 16]) torch.Size([1, 2, 16]) torch.Size([1, 2, 16])
Before the transformer, LSTM was the most common architecture for text, speech and time series; today it is still a strong option for small sequence problems.
Why does it work?
The cell state is updated from step to step mostly by addition: c_t = f_t ⊙ c_(t-1) + i_t ⊙ g_t. When the forget gate f ≈ 1, information and gradient flow almost unchanged for a long time; the repeated multiplication chain of the RNN is broken.
This is the same idea as the residual connection in ResNet: opening an additive rather than multiplicative path for information.
Exercise LSTM shapes
Apply an nn.LSTM(8, 16, batch_first=True) layer to a (2, 5, 8) input. Print the output, h_n and c_n shapes on separate lines, then with torch.allclose that output[:, -1] equals h_n[0].
Show example solution
import torch
from torch import nn
x = torch.randn(2, 5, 8)
lstm = nn.LSTM(8, 16, batch_first=True)
output, (h_n, c_n) = lstm(x)
print(output.shape)
print(h_n.shape)
print(c_n.shape)
print(torch.allclose(output[:, -1], h_n[0]))
Test yourself
-
What does an <code>nn.LSTM</code> call return differently from <code>nn.RNN</code>?
- Only output
- Three separate outputs
- Nothing
- output together with an (h_n, c_n) pair
Show answer
output together with an (h_n, c_n) pair — An LSTM carries the hidden state h and the cell state c.
-
Unpack the LSTM output.
output, (h_n, ____) = lstm(x)
Show answer
output, (h_n, c_n) = lstm(x)
-
Match the concepts.
- h
- Hidden state (output)
- c
- Cell state (memory)
- gate
- Controls information flow
- additive update
- Reduces vanishing
Next lesson: Gates
