Machine Learning
Hidden state
Hidden state is the RNN's memory from the previous step.
Hidden state
nn.RNN returns two outputs: output, containing the hidden states of every step, and h_n, the hidden state of the last step. With batch_first=True the input is given as (batch, time, feature):
rnn = nn.RNN(input_size=4, hidden_size=8, batch_first=True)
x = torch.zeros(3, 5, 4) # 3 sequences, 5 steps, 4 features
output, h_n = rnn(x)
print(output.shape) # torch.Size([3, 5, 8])
print(h_n.shape) # torch.Size([1, 3, 8]) (layer, batch, hidden)
In a single-layer, unidirectional RNN, output[:, -1] and h_n[0] are the same.
Parameters
An RNN layer's parameters don't depend on the sequence length: W_ih (hidden × input), W_hh (hidden × hidden) and two biases:
rnn = nn.RNN(4, 8)
for ad, p in rnn.named_parameters():
print(ad, tuple(p.shape))
# weight_ih_l0 (8, 4) weight_hh_l0 (8, 8) bias_ih_l0 (8,) bias_hh_l0 (8,)
print(sum(p.numel() for p in rnn.parameters())) # 32 + 64 + 16 = 112
With num_layers=2 RNNs are stacked: the second layer takes the first one's output sequence as input.
Exercise RNN outputs
Apply an nn.RNN(4, 8, batch_first=True) layer to a random (3, 5, 4) input. Print the output and h_n shapes on one line, then with torch.allclose that output[:, -1] equals h_n[0], and finally the parameter count.
Show example solution
import torch
from torch import nn
x = torch.randn(3, 5, 4)
rnn = nn.RNN(4, 8, batch_first=True)
output, h_n = rnn(x)
print(output.shape, h_n.shape)
print(torch.allclose(output[:, -1], h_n[0]))
print(sum(p.numel() for p in rnn.parameters()))
Test yourself
-
Without <code>batch_first=True</code>, in which order does <code>nn.RNN</code> expect the input?
- (time, batch, feature)
- (batch, time, feature)
- (feature, time)
- The order doesn't matter
Show answer
(time, batch, feature) — PyTorch's RNN default puts the time axis first.
-
Put the batch dimension first.
rnn = nn.RNN(4, 8, ____=True)
Show answer
rnn = nn.RNN(4, 8, batch_first=True)
-
Match the shapes (N=3, T=5, I=4, H=8).
- girdi
- (3, 5, 4)
- output
- (3, 5, 8)
- h_n
- (1, 3, 8)
- weight_hh_l0
- (8, 8)
Next lesson: Sequence
