Cell state
- Machine Learning course
- Advanced
- Lesson 55/78
- 5 min
Cell state is the LSTM's long-term memory carrier.
Cell state
The cell state c is the LSTM's long-term memory; the hidden state h is the current output. c is unbounded, while h = o · tanh(c) lies in (-1, 1):
lstm = nn.LSTM(4, 8, batch_first=True)
h0 = torch.zeros(1, 2, 8)
c0 = torch.zeros(1, 2, 8)
output, (h_n, c_n) = lstm(torch.randn(2, 6, 4), (h0, c0)) # an initial state can be given
print(h_n.abs().max().item() < 1) # TrueIf no initial state is given, zero is assumed. When processing a long text piece by piece, one piece's final state can be given as the next one's initial state.
Carrying the state
Processing a sequence in one go or in two pieces gives the same result as long as the state is carried. This is very useful with streaming data (a live sensor, a long document):
x = torch.randn(1, 10, 4)
_, (h_tam, _) = lstm(x)
_, durum = lstm(x[:, :6])
_, (h_parca, _) = lstm(x[:, 6:], durum)
print(torch.allclose(h_tam, h_parca, atol=1e-6)) # TrueIn training, splitting long sequences into pieces and carrying the state is called truncated backpropagation through time (truncated BPTT); the state is detach()ed between pieces.
Practice Process piece by piece
Build nn.LSTM(4, 8, batch_first=True) with seed 0. Process a (1, 10, 4) input in one go and in two pieces of 6 + 4 steps (carrying the state). Print with torch.allclose(..., atol=1e-6) that the final hidden states are the same, then that processing the second piece without carrying the state gives a different result.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
lstm = nn.LSTM(4, 8, batch_first=True)
x = torch.randn(1, 10, 4)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
lstm = nn.LSTM(4, 8, batch_first=True)
x = torch.randn(1, 10, 4)
_, (h_tam, _) = lstm(x)
_, durum = lstm(x[:, :6])
_, (h_parca, _) = lstm(x[:, 6:], durum)
print(torch.allclose(h_tam, h_parca, atol=1e-6))
_, (h_bos, _) = lstm(x[:, 6:])
print(not torch.allclose(h_tam, h_bos, atol=1e-6))Expected output
True True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is <code>h</code> always within (-1, 1)?
- It isn't
- h = o · tanh(c); tanh and sigmoid are bounded Correct answer
- Because c is bounded
- Because of normalisation
Explanation: o ∈ (0, 1) and tanh ∈ (-1, 1); their product is also within (-1, 1).
2 Give the previous piece's state as the initial state.
_, (h, c) = lstm(x[:, 6:], ____)_, (h, c) = lstm(x[:, 6:], durum)Explanation: The second argument is the (h0, c0) pair.
3 Match the concepts.
| Term | Meaning |
|---|---|
c |
Long-term memory |
h |
Current output |
(h0, c0) |
Initial state |
truncated BPTT |
Training in pieces |