Machine Learning
Epoch
An epoch is one complete pass through all training data.
Epoch
An epoch is one pass of the whole training data through the model. With N examples and batch size B, an epoch takes ⌈N/B⌉ steps. Models are usually trained for tens or hundreds of epochs:
for epoch in range(20):
toplam = 0.0
for xb, yb in yukleyici:
opt.zero_grad()
kayip = kayip_fn(model(xb), yb)
kayip.backward()
opt.step()
toplam += kayip.item() * len(xb)
print(epoch, toplam / len(ds)) # epoch mean loss
Weighting the epoch loss by batch size stops the last small batch from distorting the mean.
Training and evaluation mode
Layers like dropout and BatchNorm behave differently in training and evaluation. model.train() and model.eval() switch this mode; during evaluation torch.no_grad() is used as well:
model.eval()
with torch.no_grad():
tahmin = model(X_val).argmax(dim=1)
dogruluk = (tahmin == y_val).float().mean().item()
model.train() # before going back to training
eval() doesn't turn off gradient computation; it only changes layer behaviour. They are two separate jobs.
Exercise Epoch loop
With the given model, loss and optimiser, build a batch_size=32, shuffle=True loader and train for 30 epochs. Collect each epoch's example-weighted mean loss in a list. Print son < ilk for the first and last epoch losses, then that the training accuracy is above 0.9.
Show example solution
import torch
from torch import nn
from torch.utils.data import TensorDataset, DataLoader
torch.manual_seed(0)
X = torch.randn(200, 2)
y = (X[:, 0] + X[:, 1] > 0).long()
model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.SGD(model.parameters(), lr=0.1)
yukleyici = DataLoader(TensorDataset(X, y), batch_size=32, shuffle=True)
kayiplar = []
for epoch in range(30):
toplam = 0.0
for xb, yb in yukleyici:
opt.zero_grad()
kayip = kayip_fn(model(xb), yb)
kayip.backward()
opt.step()
toplam += kayip.item() * len(xb)
kayiplar.append(toplam / len(X))
print(kayiplar[-1] < kayiplar[0])
with torch.no_grad():
print((model(X).argmax(dim=1) == y).float().mean().item() > 0.9)
Test yourself
-
With 1000 examples and batch_size=32, how many steps is one epoch?
Show answer
32 — ⌈1000/32⌉ = 32: 31 full batches and a final batch of 8 examples.
-
Put the model in evaluation mode.
model.____()
Show answer
model.eval()
-
Match the concepts.
- epoch
- A full pass over the data
- step
- One batch update
- model.eval()
- Evaluation behaviour
- torch.no_grad()
- Gradient-free computation
Next lesson: Loss
