Challenge 2
- Machine Learning course
- Advanced
- Lesson 76/78
- 8 min
Define a model with nn.Module.
Challenge 2: sequence sum
You'll teach an LSTM to predict the sum of 12-step number sequences. The model sees one number per step; to produce the result it has to learn to keep an "accumulator" in its hidden state:
X = torch.randn(512, 12, 1) # 512 sequences, 12 steps, 1 feature
y = X.sum(dim=1) # (512, 1) target
lstm = nn.LSTM(1, 32, batch_first=True)
bas = nn.Linear(32, 1)The target's standard deviation is √12 ≈ 3.46; a model always predicting 0 gets MSE ≈ 12. Success is measured against this baseline.
Practice LSTM adder
Write the Toplayici class: nn.LSTM(1, 32, batch_first=True) and an nn.Linear(32, 1) on the last hidden state. Train for 300 steps on all the data with Adam(lr=0.01) and MSELoss. Print on one line that the MSE before training is above 5 and after training below 0.5.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
X = torch.randn(512, 12, 1)
y = X.sum(dim=1)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
X = torch.randn(512, 12, 1)
y = X.sum(dim=1)
class Toplayici(nn.Module):
def __init__(self):
super().__init__()
self.lstm = nn.LSTM(1, 32, batch_first=True)
self.bas = nn.Linear(32, 1)
def forward(self, x):
_, (h_n, _) = self.lstm(x)
return self.bas(h_n[-1])
model = Toplayici()
mse = nn.MSELoss()
with torch.no_grad():
once = mse(model(X), y).item()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(300):
opt.zero_grad()
kayip = mse(model(X), y)
kayip.backward()
opt.step()
print(once > 5, kayip.item() < 0.5)Expected output
True True
Practice Generalising to longer sequences
Train the given model for 300 steps to imitate a trained adder (the code is given). Then measure the MSE on 24-step sequences never seen in training (torch.randn(256, 24, 1)); print that the MSE is below the zero-prediction baseline (24).
Starter code
import torch
from torch import nn
torch.manual_seed(0)
X = torch.randn(512, 12, 1)
y = X.sum(dim=1)
lstm = nn.LSTM(1, 32, batch_first=True)
bas = nn.Linear(32, 1)
model = lambda x: bas(lstm(x)[1][0][-1])
opt = torch.optim.Adam(list(lstm.parameters()) + list(bas.parameters()), lr=0.01)
for _ in range(300):
opt.zero_grad()
nn.functional.mse_loss(model(X), y).backward()
opt.step()
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
X = torch.randn(512, 12, 1)
y = X.sum(dim=1)
lstm = nn.LSTM(1, 32, batch_first=True)
bas = nn.Linear(32, 1)
model = lambda x: bas(lstm(x)[1][0][-1])
opt = torch.optim.Adam(list(lstm.parameters()) + list(bas.parameters()), lr=0.01)
for _ in range(300):
opt.zero_grad()
nn.functional.mse_loss(model(X), y).backward()
opt.step()
X_uzun = torch.randn(256, 24, 1)
y_uzun = X_uzun.sum(dim=1)
with torch.no_grad():
hata = nn.functional.mse_loss(model(X_uzun), y_uzun).item()
print(hata < 24)Expected output
True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is the MSE of a model always predicting 0 about 12?
- From the hidden size
- It isn't
- The variance of a sum of 12 independent standard normals is 12 Correct answer
- The sequence has 12 steps, a coincidence
Explanation: Variances of independent variables add up; since the mean is 0, MSE = variance.
2 Sum along the sequence axis.
y = X.sum(____=1)y = X.sum(dim=1)Explanation: dim=1 is the time axis (batch_first=True).
3 Match the pieces.
| Term | Meaning |
|---|---|
(512, 12, 1) |
Input sequences |
(512, 1) |
Target sum |
MSE ≈ 12 |
Zero-prediction baseline |
h_n[-1] |
Summary of the accumulation |