Training loop
- Machine Learning course
- Advanced
- Lesson 35/78
- 5 min
The training loop feeds data to the model and reduces loss.
The training loop
In PyTorch you write the training loop yourself. The same five steps repeat for every batch:
for xb, yb in yukleyici:
opt.zero_grad() # 1. clear the old gradients
cikti = model(xb) # 2. forward pass
kayip = kayip_fn(cikti, yb) # 3. loss
kayip.backward() # 4. backward pass
opt.step() # 5. updateHaving the loop explicit gives flexibility: changes like gradient clipping, mixed precision or multiple losses are inserted between these five steps.
DataLoader
TensorDataset turns tensors into a dataset accessible example by example, and DataLoader turns that into shuffled batches:
from torch.utils.data import TensorDataset, DataLoader
ds = TensorDataset(X, y)
yukleyici = DataLoader(ds, batch_size=32, shuffle=True)
xb, yb = next(iter(yukleyici))
print(xb.shape, yb.shape) # torch.Size([32, 2]) torch.Size([32])Training with batches both makes things fit in memory and helps generalisation by adding useful noise to the gradient.
Practice Batches
Build a TensorDataset and a DataLoader(..., batch_size=64, shuffle=False) from the 200-example X, y. Print the number of batches (len(yukleyici)) and the size of each batch as a list.
Starter code
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(0)
X = torch.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).long()
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)
# buraya yazSample solution
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(0)
X = torch.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).long()
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)
yukleyici = DataLoader(TensorDataset(X, y), batch_size=64, shuffle=False)
print(len(yukleyici))
print([len(xb) for xb, _ in yukleyici])Expected output
4 [64, 64, 64, 8]
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens if <code>opt.zero_grad()</code> is forgotten?
- Nothing
- Gradients accumulate across steps and updates go wrong Correct answer
- It errors
- It learns faster
Explanation: backward adds the gradient on every call; without zeroing the total grows.
2 Produce the batches with shuffling.
yukleyici = DataLoader(ds, batch_size=32, ____=True)yukleyici = DataLoader(ds, batch_size=32, shuffle=True)Explanation: shuffle=True reshuffles the order at the start of every epoch.
3 Match the steps.
| Term | Meaning |
|---|---|
zero_grad |
Clear gradients |
model(xb) |
Forward pass |
backward |
Gradient computation |
step |
Parameter update |