Hidden state
- Machine Learning course
- Advanced
- Lesson 48/78
- 5 min
Hidden state is the RNN's memory from the previous step.
Hidden state
nn.RNN returns two outputs: output, containing the hidden states of every step, and h_n, the hidden state of the last step. With batch_first=True the input is given as (batch, time, feature):
rnn = nn.RNN(input_size=4, hidden_size=8, batch_first=True)
x = torch.zeros(3, 5, 4) # 3 sequences, 5 steps, 4 features
output, h_n = rnn(x)
print(output.shape) # torch.Size([3, 5, 8])
print(h_n.shape) # torch.Size([1, 3, 8]) (layer, batch, hidden)In a single-layer, unidirectional RNN, output[:, -1] and h_n[0] are the same.
Parameters
An RNN layer's parameters don't depend on the sequence length: W_ih (hidden × input), W_hh (hidden × hidden) and two biases:
rnn = nn.RNN(4, 8)
for ad, p in rnn.named_parameters():
print(ad, tuple(p.shape))
# weight_ih_l0 (8, 4) weight_hh_l0 (8, 8) bias_ih_l0 (8,) bias_hh_l0 (8,)
print(sum(p.numel() for p in rnn.parameters())) # 32 + 64 + 16 = 112With num_layers=2 RNNs are stacked: the second layer takes the first one's output sequence as input.
Practice RNN outputs
Apply an nn.RNN(4, 8, batch_first=True) layer to a random (3, 5, 4) input. Print the output and h_n shapes on one line, then with torch.allclose that output[:, -1] equals h_n[0], and finally the parameter count.
Starter code
import torch
from torch import nn
x = torch.randn(3, 5, 4)
# buraya yazSample solution
import torch
from torch import nn
x = torch.randn(3, 5, 4)
rnn = nn.RNN(4, 8, batch_first=True)
output, h_n = rnn(x)
print(output.shape, h_n.shape)
print(torch.allclose(output[:, -1], h_n[0]))
print(sum(p.numel() for p in rnn.parameters()))Expected output
torch.Size([3, 5, 8]) torch.Size([1, 3, 8]) True 112
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Without <code>batch_first=True</code>, in which order does <code>nn.RNN</code> expect the input?
- (time, batch, feature) Correct answer
- (batch, time, feature)
- (feature, time)
- The order doesn't matter
Explanation: PyTorch's RNN default puts the time axis first.
2 Put the batch dimension first.
rnn = nn.RNN(4, 8, ____=True)rnn = nn.RNN(4, 8, batch_first=True)Explanation: batch_first=True makes the input and output (N, T, ·).
3 Match the shapes (N=3, T=5, I=4, H=8).
| Term | Meaning |
|---|---|
girdi |
(3, 5, 4) |
output |
(3, 5, 8) |
h_n |
(1, 3, 8) |
weight_hh_l0 |
(8, 8) |