input_size

input_size is the dimension of the feature vector entering the LSTM.

input_size

input_size is the size of the input vector at each time step: the number of measured variables in a time series, the embedding size in text. The sequence length isn't a parameter; it can differ on every call:

lstm = nn.LSTM(input_size=3, hidden_size=16, batch_first=True)
print(lstm(torch.zeros(2, 10, 3))[0].shape)    # torch.Size([2, 10, 16])
print(lstm(torch.zeros(2, 50, 3))[0].shape)    # torch.Size([2, 50, 16])
# lstm(torch.zeros(2, 10, 4))   error: input_size 3 expected

In text models input_size must equal the embedding layer's embedding_dim.

Parameter count

Because an LSTM computes four components (f, i, g, o) together, its parameters are four times an RNN's: 4·(H·I + H·H + 2H):

lstm = nn.LSTM(3, 16)
for ad, p in lstm.named_parameters():
    print(ad, tuple(p.shape))
# weight_ih_l0 (64, 3)  weight_hh_l0 (64, 16)  bias_ih_l0 (64,)  bias_hh_l0 (64,)
print(sum(p.numel() for p in lstm.parameters()))   # 4·(48 + 256 + 32) = 1344

The 64 rows are the four gates' 16 rows each, stacked.

Exercise LSTM parameters

Print the parameter counts of the nn.LSTM(5, 32) and nn.RNN(5, 32) layers on one line. Then print the LSTM's weight_ih_l0 shape.

Show example solution

import torch
from torch import nn

lstm = nn.LSTM(5, 32)
rnn = nn.RNN(5, 32)
say = lambda m: sum(p.numel() for p in m.parameters())
print(say(lstm), say(rnn))
print(lstm.weight_ih_l0.shape)

Test yourself

In a weather series measuring temperature, humidity and wind speed, what is <code>input_size</code>?
- 3
- The sequence length
- 1
- The hidden size
Show answer

3 — Three variables are measured at each time step.

Build an LSTM taking 5 features per step.

lstm = nn.LSTM(____=5, hidden_size=32, batch_first=True)

Show answer

lstm = nn.LSTM(input_size=5, hidden_size=32, batch_first=True)

Match the values (I=3, H=16).

weight_ih_l0

(64, 3)
weight_hh_l0

(64, 16)
bias_ih_l0

(64,)
toplam

1344

Next lesson: hidden_size

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies