Machine Learning
Challenge 2
Define a model with nn.Module.
Challenge 2: sequence sum
You'll teach an LSTM to predict the sum of 12-step number sequences. The model sees one number per step; to produce the result it has to learn to keep an "accumulator" in its hidden state:
X = torch.randn(512, 12, 1) # 512 sequences, 12 steps, 1 feature
y = X.sum(dim=1) # (512, 1) target
lstm = nn.LSTM(1, 32, batch_first=True)
bas = nn.Linear(32, 1)
The target's standard deviation is √12 ≈ 3.46; a model always predicting 0 gets MSE ≈ 12. Success is measured against this baseline.
Exercise LSTM adder
Write the Toplayici class: nn.LSTM(1, 32, batch_first=True) and an nn.Linear(32, 1) on the last hidden state. Train for 300 steps on all the data with Adam(lr=0.01) and MSELoss. Print on one line that the MSE before training is above 5 and after training below 0.5.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
X = torch.randn(512, 12, 1)
y = X.sum(dim=1)
class Toplayici(nn.Module):
def __init__(self):
super().__init__()
self.lstm = nn.LSTM(1, 32, batch_first=True)
self.bas = nn.Linear(32, 1)
def forward(self, x):
_, (h_n, _) = self.lstm(x)
return self.bas(h_n[-1])
model = Toplayici()
mse = nn.MSELoss()
with torch.no_grad():
once = mse(model(X), y).item()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
for _ in range(300):
opt.zero_grad()
kayip = mse(model(X), y)
kayip.backward()
opt.step()
print(once > 5, kayip.item() < 0.5)
Exercise Generalising to longer sequences
Train the given model for 300 steps to imitate a trained adder (the code is given). Then measure the MSE on 24-step sequences never seen in training (torch.randn(256, 24, 1)); print that the MSE is below the zero-prediction baseline (24).
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
X = torch.randn(512, 12, 1)
y = X.sum(dim=1)
lstm = nn.LSTM(1, 32, batch_first=True)
bas = nn.Linear(32, 1)
model = lambda x: bas(lstm(x)[1][0][-1])
opt = torch.optim.Adam(list(lstm.parameters()) + list(bas.parameters()), lr=0.01)
for _ in range(300):
opt.zero_grad()
nn.functional.mse_loss(model(X), y).backward()
opt.step()
X_uzun = torch.randn(256, 24, 1)
y_uzun = X_uzun.sum(dim=1)
with torch.no_grad():
hata = nn.functional.mse_loss(model(X_uzun), y_uzun).item()
print(hata < 24)
Test yourself
-
Why is the MSE of a model always predicting 0 about 12?
- From the hidden size
- It isn't
- The variance of a sum of 12 independent standard normals is 12
- The sequence has 12 steps, a coincidence
Show answer
The variance of a sum of 12 independent standard normals is 12 — Variances of independent variables add up; since the mean is 0, MSE = variance.
-
Sum along the sequence axis.
y = X.sum(____=1)
Show answer
y = X.sum(dim=1)
-
Match the pieces.
- (512, 12, 1)
- Input sequences
- (512, 1)
- Target sum
- MSE ≈ 12
- Zero-prediction baseline
- h_n[-1]
- Summary of the accumulation
Next lesson: Challenge 3
