input_size
- Machine Learning course
- Advanced
- Lesson 56/78
- 5 min
input_size is the dimension of the feature vector entering the LSTM.
input_size
input_size is the size of the input vector at each time step: the number of measured variables in a time series, the embedding size in text. The sequence length isn't a parameter; it can differ on every call:
lstm = nn.LSTM(input_size=3, hidden_size=16, batch_first=True)
print(lstm(torch.zeros(2, 10, 3))[0].shape) # torch.Size([2, 10, 16])
print(lstm(torch.zeros(2, 50, 3))[0].shape) # torch.Size([2, 50, 16])
# lstm(torch.zeros(2, 10, 4)) error: input_size 3 expectedIn text models input_size must equal the embedding layer's embedding_dim.
Parameter count
Because an LSTM computes four components (f, i, g, o) together, its parameters are four times an RNN's: 4·(H·I + H·H + 2H):
lstm = nn.LSTM(3, 16)
for ad, p in lstm.named_parameters():
print(ad, tuple(p.shape))
# weight_ih_l0 (64, 3) weight_hh_l0 (64, 16) bias_ih_l0 (64,) bias_hh_l0 (64,)
print(sum(p.numel() for p in lstm.parameters())) # 4·(48 + 256 + 32) = 1344The 64 rows are the four gates' 16 rows each, stacked.
Practice LSTM parameters
Print the parameter counts of the nn.LSTM(5, 32) and nn.RNN(5, 32) layers on one line. Then print the LSTM's weight_ih_l0 shape.
Starter code
import torch
from torch import nn
# buraya yazSample solution
import torch
from torch import nn
lstm = nn.LSTM(5, 32)
rnn = nn.RNN(5, 32)
say = lambda m: sum(p.numel() for p in m.parameters())
print(say(lstm), say(rnn))
print(lstm.weight_ih_l0.shape)Expected output
4992 1248 torch.Size([128, 5])
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In a weather series measuring temperature, humidity and wind speed, what is <code>input_size</code>?
- 3 Correct answer
- The sequence length
- 1
- The hidden size
Explanation: Three variables are measured at each time step.
2 Build an LSTM taking 5 features per step.
lstm = nn.LSTM(____=5, hidden_size=32, batch_first=True)lstm = nn.LSTM(input_size=5, hidden_size=32, batch_first=True)Explanation: input_size is the number of features per step.
3 Match the values (I=3, H=16).
| Term | Meaning |
|---|---|
weight_ih_l0 |
(64, 3) |
weight_hh_l0 |
(64, 16) |
bias_ih_l0 |
(64,) |
toplam |
1344 |