Machine Learning
hidden_size
hidden_size is the dimension of the LSTM's hidden state vector.
hidden_size
hidden_size is the size of the hidden and cell states; it sets the model's "memory capacity". A large value holds more complex patterns, but the parameter count grows roughly quadratically and the overfitting risk grows:
for H in [16, 64, 256]:
print(H, sum(p.numel() for p in nn.LSTM(8, H).parameters()))
# 16 1664 64 18944 256 272384
Typical values are between 64 and 512; the choice is made by validation according to the data size.
Bidirectional and multi-layer
bidirectional=True processes the sequence both start to end and end to start and concatenates the two directions' outputs; useful when a word's meaning also depends on the following words. num_layers stacks layers, and dropout is applied between layers:
lstm = nn.LSTM(16, 32, num_layers=2, bidirectional=True, dropout=0.2, batch_first=True)
output, (h_n, c_n) = lstm(torch.zeros(4, 10, 16))
print(output.shape) # torch.Size([4, 10, 64]): 2 directions × 32
print(h_n.shape) # torch.Size([4, 4, 32]): 2 layers × 2 directions
Because a bidirectional model sees the future, it can't be used for live prediction (forecasting the next step).
Exercise Bidirectional LSTM
Apply an nn.LSTM(16, 32, num_layers=2, bidirectional=True, batch_first=True) layer to a (4, 10, 16) input. Print the output and h_n shapes on one line. Then print the parameter counts of the unidirectional LSTM with the same structure and of the bidirectional one on one line.
Show example solution
import torch
from torch import nn
x = torch.zeros(4, 10, 16)
cift = nn.LSTM(16, 32, num_layers=2, bidirectional=True, batch_first=True)
output, (h_n, c_n) = cift(x)
print(output.shape, h_n.shape)
tek = nn.LSTM(16, 32, num_layers=2, batch_first=True)
say = lambda m: sum(p.numel() for p in m.parameters())
print(say(tek), say(cift))
Test yourself
-
If hidden_size is doubled, roughly how many times do the <code>weight_hh</code> parameters grow?
Show answer
4 — weight_hh has shape 4H × H; doubling H grows it 4 times.
-
Process the sequence in both directions.
lstm = nn.LSTM(16, 32, ____=True, batch_first=True)
Show answer
lstm = nn.LSTM(16, 32, bidirectional=True, batch_first=True)
-
Match the shapes (H=32, 2 layers, bidirectional).
- output son boyut
- 64
- h_n ilk boyut
- 4
- h_n son boyut
- 32
- dropout
- Between layers
Next lesson: Summary - LSTM
