hidden_size
- Machine Learning course
- Advanced
- Lesson 57/78
- 5 min
hidden_size is the dimension of the LSTM's hidden state vector.
hidden_size
hidden_size is the size of the hidden and cell states; it sets the model's "memory capacity". A large value holds more complex patterns, but the parameter count grows roughly quadratically and the overfitting risk grows:
for H in [16, 64, 256]:
print(H, sum(p.numel() for p in nn.LSTM(8, H).parameters()))
# 16 1664 64 18944 256 272384Typical values are between 64 and 512; the choice is made by validation according to the data size.
Bidirectional and multi-layer
bidirectional=True processes the sequence both start to end and end to start and concatenates the two directions' outputs; useful when a word's meaning also depends on the following words. num_layers stacks layers, and dropout is applied between layers:
lstm = nn.LSTM(16, 32, num_layers=2, bidirectional=True, dropout=0.2, batch_first=True)
output, (h_n, c_n) = lstm(torch.zeros(4, 10, 16))
print(output.shape) # torch.Size([4, 10, 64]): 2 directions × 32
print(h_n.shape) # torch.Size([4, 4, 32]): 2 layers × 2 directionsBecause a bidirectional model sees the future, it can't be used for live prediction (forecasting the next step).
Practice Bidirectional LSTM
Apply an nn.LSTM(16, 32, num_layers=2, bidirectional=True, batch_first=True) layer to a (4, 10, 16) input. Print the output and h_n shapes on one line. Then print the parameter counts of the unidirectional LSTM with the same structure and of the bidirectional one on one line.
Starter code
import torch
from torch import nn
x = torch.zeros(4, 10, 16)
# buraya yazSample solution
import torch
from torch import nn
x = torch.zeros(4, 10, 16)
cift = nn.LSTM(16, 32, num_layers=2, bidirectional=True, batch_first=True)
output, (h_n, c_n) = cift(x)
print(output.shape, h_n.shape)
tek = nn.LSTM(16, 32, num_layers=2, batch_first=True)
say = lambda m: sum(p.numel() for p in m.parameters())
print(say(tek), say(cift))Expected output
torch.Size([4, 10, 64]) torch.Size([4, 4, 32]) 14848 37888
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If hidden_size is doubled, roughly how many times do the <code>weight_hh</code> parameters grow?
- 2
- 8
- They don't change
- 4 Correct answer
Explanation: weight_hh has shape 4H × H; doubling H grows it 4 times.
2 Process the sequence in both directions.
lstm = nn.LSTM(16, 32, ____=True, batch_first=True)lstm = nn.LSTM(16, 32, bidirectional=True, batch_first=True)Explanation: bidirectional=True doubles the output size.
3 Match the shapes (H=32, 2 layers, bidirectional).
| Term | Meaning |
|---|---|
output son boyut |
64 |
h_n ilk boyut |
4 |
h_n son boyut |
32 |
dropout |
Between layers |