Machine Learning
Training loop
The training loop feeds data to the model and reduces loss.
The training loop
In PyTorch you write the training loop yourself. The same five steps repeat for every batch:
for xb, yb in yukleyici:
opt.zero_grad() # 1. clear the old gradients
cikti = model(xb) # 2. forward pass
kayip = kayip_fn(cikti, yb) # 3. loss
kayip.backward() # 4. backward pass
opt.step() # 5. update
Having the loop explicit gives flexibility: changes like gradient clipping, mixed precision or multiple losses are inserted between these five steps.
DataLoader
TensorDataset turns tensors into a dataset accessible example by example, and DataLoader turns that into shuffled batches:
from torch.utils.data import TensorDataset, DataLoader
ds = TensorDataset(X, y)
yukleyici = DataLoader(ds, batch_size=32, shuffle=True)
xb, yb = next(iter(yukleyici))
print(xb.shape, yb.shape) # torch.Size([32, 2]) torch.Size([32])
Training with batches both makes things fit in memory and helps generalisation by adding useful noise to the gradient.
Exercise Batches
Build a TensorDataset and a DataLoader(..., batch_size=64, shuffle=False) from the 200-example X, y. Print the number of batches (len(yukleyici)) and the size of each batch as a list.
Show example solution
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(0)
X = torch.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).long()
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)
yukleyici = DataLoader(TensorDataset(X, y), batch_size=64, shuffle=False)
print(len(yukleyici))
print([len(xb) for xb, _ in yukleyici])
Test yourself
-
What happens if <code>opt.zero_grad()</code> is forgotten?
- Nothing
- Gradients accumulate across steps and updates go wrong
- It errors
- It learns faster
Show answer
Gradients accumulate across steps and updates go wrong — backward adds the gradient on every call; without zeroing the total grows.
-
Produce the batches with shuffling.
yukleyici = DataLoader(ds, batch_size=32, ____=True)
Show answer
yukleyici = DataLoader(ds, batch_size=32, shuffle=True)
-
Match the steps.
- zero_grad
- Clear gradients
- model(xb)
- Forward pass
- backward
- Gradient computation
- step
- Parameter update
Next lesson: Epoch
