Machine Learning
Loss
Loss measures how wrong the model's predictions are.
Loss functions
The loss function is chosen by the type of problem:
nn.MSELoss: regression.nn.CrossEntropyLoss: multi-class classification; takes raw logits and integer labels and does the softmax internally.nn.BCEWithLogitsLoss: binary classification; takes a raw logit and a 0/1 float label.
ce = nn.CrossEntropyLoss()
logit = torch.zeros(2, 3) # an equal score for every class
print(round(ce(logit, torch.tensor([0, 2])).item(), 4)) # 1.0986 = log(3)
Giving CrossEntropyLoss probabilities that already went through softmax is a common mistake; it runs but learns wrongly.
MSE and the mean
Losses average over the batch by default (reduction="mean"). "sum" gives the total, "none" the per-example losses; useful for weighting examples:
mse = nn.MSELoss()
t = torch.tensor([1., 2., 3.])
g = torch.tensor([1., 4., 6.])
print(mse(t, g).item()) # (0 + 4 + 9) / 3 = 4.333...
print(nn.MSELoss(reduction="none")(t, g).tolist()) # [0.0, 4.0, 9.0]
In MSE the prediction and the target must have the same shape; otherwise broadcasting silently produces a wrong result.
Exercise Compute the losses
Print the nn.CrossEntropyLoss() value while the logits are zero (round(..., 4)). Then print the loss with round(..., 4) for a tensor giving the correct class a logit of 10 and the others 0. Finally print the reduction="sum" MSE between [1., 2., 3.] and [1., 4., 6.].
Show example solution
import torch
from torch import nn
ce = nn.CrossEntropyLoss()
y = torch.tensor([0, 2])
print(round(ce(torch.zeros(2, 3), y).item(), 4))
iyi = torch.zeros(2, 3)
iyi[torch.arange(2), y] = 10.0
print(round(ce(iyi, y).item(), 4))
print(nn.MSELoss(reduction="sum")(torch.tensor([1., 2., 3.]), torch.tensor([1., 4., 6.])).item())
Test yourself
-
Which input should be given to CrossEntropyLoss?
- Softmax probabilities and one-hot labels
- Sigmoid outputs
- Predicted classes
- Raw logits and int64 labels
Show answer
Raw logits and int64 labels — Log-softmax is applied internally; doing softmax outside applies it twice.
-
Get the per-example losses.
kayip_fn = nn.MSELoss(reduction=____)
Show answer
kayip_fn = nn.MSELoss(reduction="none")
-
Match the losses.
- MSELoss
- Regression
- CrossEntropyLoss
- Multi-class, logits
- BCEWithLogitsLoss
- Binary, logit
- reduction="sum"
- Total loss
Next lesson: backward()
