Sequence
- Machine Learning course
- Advanced
- Lesson 49/78
- 5 min
RNN steps through the sequence, updating hidden state at each step.
Sequence
The sequences in a batch usually have different lengths. They are all brought to the same length with padding; the real lengths are also kept so the model can ignore the padding steps:
from torch.nn.utils.rnn import pad_sequence
diziler = [torch.tensor([3, 1, 4]), torch.tensor([1, 5]), torch.tensor([9, 2, 6, 5])]
X = pad_sequence(diziler, batch_first=True, padding_value=0)
print(X.tolist()) # [[3, 1, 4, 0], [1, 5, 0, 0], [9, 2, 6, 5]]
uzunluk = torch.tensor([len(d) for d in diziler])For text the numbers are word indices; 0 is usually reserved for padding.
Selecting the last real step
The output of each sequence's real last step is taken with fancy indexing. Alternatively, pack_padded_sequence makes the RNN not process the padding steps at all:
output, _ = rnn(gomulu) # (N, T, H)
son = output[torch.arange(len(uzunluk)), uzunluk - 1] # (N, H)
from torch.nn.utils.rnn import pack_padded_sequence
paket = pack_padded_sequence(gomulu, uzunluk, batch_first=True, enforce_sorted=False)
_, h_n = rnn(paket) # h_n holds the real last stepsPacking gains both correctness and speed; the padding steps aren't computed.
Practice Padding and length
Pad the three given sequences with pad_sequence(..., batch_first=True) and print them with tolist(). Collect the lengths in a tensor; pick each sequence's last real element with X[torch.arange(3), uzunluk - 1] and print it with tolist().
Starter code
import torch
from torch import nn
from torch.nn.utils.rnn import pad_sequence
diziler = [torch.tensor([3, 1, 4]), torch.tensor([1, 5]), torch.tensor([9, 2, 6, 5])]
# buraya yazSample solution
import torch
from torch import nn
from torch.nn.utils.rnn import pad_sequence
diziler = [torch.tensor([3, 1, 4]), torch.tensor([1, 5]), torch.tensor([9, 2, 6, 5])]
X = pad_sequence(diziler, batch_first=True)
print(X.tolist())
uzunluk = torch.tensor([len(d) for d in diziler])
print(X[torch.arange(3), uzunluk - 1].tolist())Expected output
[[3, 1, 4, 0], [1, 5, 0, 0], [9, 2, 6, 5]] [4, 5, 5]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 In a padded sequence, why isn't the last real step's output <code>output[:, -1]</code>?
- It always is
- The RNN starts from the end
- There is no output
- In short sequences the last position is padding Correct answer
Explanation: A short sequence's real last step is at position length - 1.
2 Bring the sequences to the same length.
X = ____(diziler, batch_first=True)X = pad_sequence(diziler, batch_first=True)Explanation: pad_sequence pads with zeros up to the longest sequence.
3 Match the tools.
| Term | Meaning |
|---|---|
pad_sequence |
Adds padding |
uzunluk |
Real step count |
pack_padded_sequence |
Skips padding |
padding_value=0 |
Padding index |