Machine Learning
Loss and optimizer
Use CrossEntropy loss and SGD optimizer.
Loss and optimiser
For two-class classification we use CrossEntropyLoss (2 logits). As the optimiser, Adam is fast and not very sensitive to tuning on small CNNs:
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
with torch.no_grad():
print(kayip_fn(model(X_train), y_train).item()) # before training ≈ log(2) = 0.69
A loss around log(number of classes) before training is a sanity check showing the model still predicts at random and the setup is right.
The overfit-one-batch test
The fastest way to verify that a model can learn is making it memorise a single small batch. If the loss doesn't approach zero, there is a bug in the code (wrong labels, a missing zero_grad, a wrong loss):
xb, yb = X_train[:16], y_train[:16]
for _ in range(100):
opt.zero_grad()
kayip = kayip_fn(model(xb), yb)
kayip.backward()
opt.step()
print(kayip.item() < 0.05) # True expected
This test takes a few seconds and prevents hours of wasted training.
Exercise Sanity checks
Print that the loss before training is close to log(2) (abs(kayip - 0.693) < 0.1). Then train 100 steps on the first 16 examples and print that the loss is below 0.05.
Show example solution
import torch
from torch import nn
torch.manual_seed(0)
def veri(n):
X = torch.rand(n, 1, 16, 16) * 0.3
y = torch.randint(0, 2, (n,))
for k in range(n):
if y[k] == 0:
X[k, 0, 7:9, :] += 1.0
else:
X[k, 0, :, 7:9] += 1.0
return X, y
X_train, y_train = veri(256)
X_test, y_test = veri(64)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(16 * 4 * 4, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
with torch.no_grad():
ilk = kayip_fn(model(X_train), y_train).item()
print(abs(ilk - 0.693) < 0.1)
xb, yb = X_train[:16], y_train[:16]
for _ in range(100):
opt.zero_grad()
kayip = kayip_fn(model(xb), yb)
kayip.backward()
opt.step()
print(kayip.item() < 0.05)
Test yourself
-
Roughly what should the loss be before training in a 10-class problem?
Show answer
2.3 (log 10) — Random prediction gives each class probability 1/10: -log(0.1) ≈ 2.30.
-
Set up the Adam optimiser.
opt = torch.optim.____(model.parameters(), lr=0.01)
Show answer
opt = torch.optim.Adam(model.parameters(), lr=0.01)
-
Match the checks.
- initial loss ≈ log(K)
- Setup is right
- overfitting a single batch
- The model can learn
- loss not decreasing
- A bug in the code
- Adam
- Adaptive optimiser
Next lesson: Training
