Machine Learning
Sequence
RNN steps through the sequence, updating hidden state at each step.
Sequence
The sequences in a batch usually have different lengths. They are all brought to the same length with padding; the real lengths are also kept so the model can ignore the padding steps:
from torch.nn.utils.rnn import pad_sequence
diziler = [torch.tensor([3, 1, 4]), torch.tensor([1, 5]), torch.tensor([9, 2, 6, 5])]
X = pad_sequence(diziler, batch_first=True, padding_value=0)
print(X.tolist()) # [[3, 1, 4, 0], [1, 5, 0, 0], [9, 2, 6, 5]]
uzunluk = torch.tensor([len(d) for d in diziler])
For text the numbers are word indices; 0 is usually reserved for padding.
Selecting the last real step
The output of each sequence's real last step is taken with fancy indexing. Alternatively, pack_padded_sequence makes the RNN not process the padding steps at all:
output, _ = rnn(gomulu) # (N, T, H)
son = output[torch.arange(len(uzunluk)), uzunluk - 1] # (N, H)
from torch.nn.utils.rnn import pack_padded_sequence
paket = pack_padded_sequence(gomulu, uzunluk, batch_first=True, enforce_sorted=False)
_, h_n = rnn(paket) # h_n holds the real last steps
Packing gains both correctness and speed; the padding steps aren't computed.
Exercise Padding and length
Pad the three given sequences with pad_sequence(..., batch_first=True) and print them with tolist(). Collect the lengths in a tensor; pick each sequence's last real element with X[torch.arange(3), uzunluk - 1] and print it with tolist().
Show example solution
import torch
from torch import nn
from torch.nn.utils.rnn import pad_sequence
diziler = [torch.tensor([3, 1, 4]), torch.tensor([1, 5]), torch.tensor([9, 2, 6, 5])]
X = pad_sequence(diziler, batch_first=True)
print(X.tolist())
uzunluk = torch.tensor([len(d) for d in diziler])
print(X[torch.arange(3), uzunluk - 1].tolist())
Test yourself
-
In a padded sequence, why isn't the last real step's output <code>output[:, -1]</code>?
- It always is
- The RNN starts from the end
- There is no output
- In short sequences the last position is padding
Show answer
In short sequences the last position is padding — A short sequence's real last step is at position length - 1.
-
Bring the sequences to the same length.
X = ____(diziler, batch_first=True)
Show answer
X = pad_sequence(diziler, batch_first=True)
-
Match the tools.
- pad_sequence
- Adds padding
- uzunluk
- Real step count
- pack_padded_sequence
- Skips padding
- padding_value=0
- Padding index
Next lesson: Vanishing gradient
