Machine Learning
Cell state
Cell state is the LSTM's long-term memory carrier.
Cell state
The cell state c is the LSTM's long-term memory; the hidden state h is the current output. c is unbounded, while h = o · tanh(c) lies in (-1, 1):
lstm = nn.LSTM(4, 8, batch_first=True)
h0 = torch.zeros(1, 2, 8)
c0 = torch.zeros(1, 2, 8)
output, (h_n, c_n) = lstm(torch.randn(2, 6, 4), (h0, c0)) # an initial state can be given
print(h_n.abs().max().item() < 1) # True
If no initial state is given, zero is assumed. When processing a long text piece by piece, one piece's final state can be given as the next one's initial state.
Carrying the state
Processing a sequence in one go or in two pieces gives the same result as long as the state is carried. This is very useful with streaming data (a live sensor, a long document):
x = torch.randn(1, 10, 4)
_, (h_tam, _) = lstm(x)
_, durum = lstm(x[:, :6])
_, (h_parca, _) = lstm(x[:, 6:], durum)
print(torch.allclose(h_tam, h_parca, atol=1e-6)) # True
In training, splitting long sequences into pieces and carrying the state is called truncated backpropagation through time (truncated BPTT); the state is detach()ed between pieces.
Exercise Process piece by piece
Build nn.LSTM(4, 8, batch_first=True) with seed 0. Process a (1, 10, 4) input in one go and in two pieces of 6 + 4 steps (carrying the state). Print with torch.allclose(..., atol=1e-6) that the final hidden states are the same, then that processing the second piece without carrying the state gives a different result.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
lstm = nn.LSTM(4, 8, batch_first=True)
x = torch.randn(1, 10, 4)
_, (h_tam, _) = lstm(x)
_, durum = lstm(x[:, :6])
_, (h_parca, _) = lstm(x[:, 6:], durum)
print(torch.allclose(h_tam, h_parca, atol=1e-6))
_, (h_bos, _) = lstm(x[:, 6:])
print(not torch.allclose(h_tam, h_bos, atol=1e-6))
Test yourself
-
Why is <code>h</code> always within (-1, 1)?
- It isn't
- h = o · tanh(c); tanh and sigmoid are bounded
- Because c is bounded
- Because of normalisation
Show answer
h = o · tanh(c); tanh and sigmoid are bounded — o ∈ (0, 1) and tanh ∈ (-1, 1); their product is also within (-1, 1).
-
Give the previous piece's state as the initial state.
_, (h, c) = lstm(x[:, 6:], ____)
Show answer
_, (h, c) = lstm(x[:, 6:], durum)
-
Match the concepts.
- c
- Long-term memory
- h
- Current output
- (h0, c0)
- Initial state
- truncated BPTT
- Training in pieces
Next lesson: input_size
