Loss
- Machine Learning course
- Advanced
- Lesson 37/78
- 5 min
Loss measures how wrong the model's predictions are.
Loss functions
The loss function is chosen by the type of problem:
nn.MSELoss: regression.nn.CrossEntropyLoss: multi-class classification; takes raw logits and integer labels and does the softmax internally.nn.BCEWithLogitsLoss: binary classification; takes a raw logit and a 0/1 float label.
ce = nn.CrossEntropyLoss()
logit = torch.zeros(2, 3) # an equal score for every class
print(round(ce(logit, torch.tensor([0, 2])).item(), 4)) # 1.0986 = log(3)Giving CrossEntropyLoss probabilities that already went through softmax is a common mistake; it runs but learns wrongly.
MSE and the mean
Losses average over the batch by default (reduction="mean"). "sum" gives the total, "none" the per-example losses; useful for weighting examples:
mse = nn.MSELoss()
t = torch.tensor([1., 2., 3.])
g = torch.tensor([1., 4., 6.])
print(mse(t, g).item()) # (0 + 4 + 9) / 3 = 4.333...
print(nn.MSELoss(reduction="none")(t, g).tolist()) # [0.0, 4.0, 9.0]In MSE the prediction and the target must have the same shape; otherwise broadcasting silently produces a wrong result.
Practice Compute the losses
Print the nn.CrossEntropyLoss() value while the logits are zero (round(..., 4)). Then print the loss with round(..., 4) for a tensor giving the correct class a logit of 10 and the others 0. Finally print the reduction="sum" MSE between [1., 2., 3.] and [1., 4., 6.].
Starter code
import torch
from torch import nn
ce = nn.CrossEntropyLoss()
y = torch.tensor([0, 2])
# buraya yazSample solution
import torch
from torch import nn
ce = nn.CrossEntropyLoss()
y = torch.tensor([0, 2])
print(round(ce(torch.zeros(2, 3), y).item(), 4))
iyi = torch.zeros(2, 3)
iyi[torch.arange(2), y] = 10.0
print(round(ce(iyi, y).item(), 4))
print(nn.MSELoss(reduction="sum")(torch.tensor([1., 2., 3.]), torch.tensor([1., 4., 6.])).item())Expected output
1.0986 0.0001 13.0
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which input should be given to CrossEntropyLoss?
- Softmax probabilities and one-hot labels
- Sigmoid outputs
- Predicted classes
- Raw logits and int64 labels Correct answer
Explanation: Log-softmax is applied internally; doing softmax outside applies it twice.
2 Get the per-example losses.
kayip_fn = nn.MSELoss(reduction=____)kayip_fn = nn.MSELoss(reduction="none")Explanation: "none" returns each example's loss without reduction.
3 Match the losses.
| Term | Meaning |
|---|---|
MSELoss |
Regression |
CrossEntropyLoss |
Multi-class, logits |
BCEWithLogitsLoss |
Binary, logit |
reduction="sum" |
Total loss |