Loss and optimizer
- Machine Learning course
- Advanced
- Lesson 61/78
- 7 min
Use CrossEntropy loss and SGD optimizer.
Loss and optimiser
For two-class classification we use CrossEntropyLoss (2 logits). As the optimiser, Adam is fast and not very sensitive to tuning on small CNNs:
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
with torch.no_grad():
print(kayip_fn(model(X_train), y_train).item()) # before training ≈ log(2) = 0.69A loss around log(number of classes) before training is a sanity check showing the model still predicts at random and the setup is right.
The overfit-one-batch test
The fastest way to verify that a model can learn is making it memorise a single small batch. If the loss doesn't approach zero, there is a bug in the code (wrong labels, a missing zero_grad, a wrong loss):
xb, yb = X_train[:16], y_train[:16]
for _ in range(100):
opt.zero_grad()
kayip = kayip_fn(model(xb), yb)
kayip.backward()
opt.step()
print(kayip.item() < 0.05) # True expectedThis test takes a few seconds and prevents hours of wasted training.
Practice Sanity checks
Print that the loss before training is close to log(2) (abs(kayip - 0.693) < 0.1). Then train 100 steps on the first 16 examples and print that the loss is below 0.05.
Starter code
import torch
from torch import nn
torch.manual_seed(0)
def veri(n):
X = torch.rand(n, 1, 16, 16) * 0.3
y = torch.randint(0, 2, (n,))
for k in range(n):
if y[k] == 0:
X[k, 0, 7:9, :] += 1.0
else:
X[k, 0, :, 7:9] += 1.0
return X, y
X_train, y_train = veri(256)
X_test, y_test = veri(64)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(16 * 4 * 4, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
# buraya yazSample solution
import torch
from torch import nn
torch.manual_seed(0)
def veri(n):
X = torch.rand(n, 1, 16, 16) * 0.3
y = torch.randint(0, 2, (n,))
for k in range(n):
if y[k] == 0:
X[k, 0, 7:9, :] += 1.0
else:
X[k, 0, :, 7:9] += 1.0
return X, y
X_train, y_train = veri(256)
X_test, y_test = veri(64)
model = nn.Sequential(
nn.Conv2d(1, 8, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(8, 16, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten(), nn.Linear(16 * 4 * 4, 2))
kayip_fn = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
with torch.no_grad():
ilk = kayip_fn(model(X_train), y_train).item()
print(abs(ilk - 0.693) < 0.1)
xb, yb = X_train[:16], y_train[:16]
for _ in range(100):
opt.zero_grad()
kayip = kayip_fn(model(xb), yb)
kayip.backward()
opt.step()
print(kayip.item() < 0.05)Expected output
True True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Roughly what should the loss be before training in a 10-class problem?
- 10
- 2.3 (log 10) Correct answer
- 0
- 0.69
Explanation: Random prediction gives each class probability 1/10: -log(0.1) ≈ 2.30.
2 Set up the Adam optimiser.
opt = torch.optim.____(model.parameters(), lr=0.01)opt = torch.optim.Adam(model.parameters(), lr=0.01)Explanation: Adam uses adaptive learning rates.
3 Match the checks.
| Term | Meaning |
|---|---|
initial loss ≈ log(K) |
Setup is right |
overfitting a single batch |
The model can learn |
loss not decreasing |
A bug in the code |
Adam |
Adaptive optimiser |