Loss computation

Compute MSE loss.

Computing the loss

In regression the loss is the mean squared error (MSE). Since the loss is a scalar, backward() can be called directly:

def mse(tahmin, hedef):
    return ((tahmin - hedef) ** 2).mean()

kayip = mse(model(x), y)
kayip.backward()
print(w.grad, b.grad)

While the parameters are zero the gradients are negative: they say w and b must increase to lower the loss.

Verifying the gradient by hand

MSE's derivatives are known in closed form: ∂L/∂w = mean(2(ŷ - y)·x), ∂L/∂b = mean(2(ŷ - y)). Comparing the autograd result with these formulas shows the code is correct:

fark = model(x) - y
print(torch.allclose(w.grad, (2 * fark * x).mean()))   # True
print(torch.allclose(b.grad, (2 * fark).mean()))       # True

In large models derivatives aren't written by hand, but such small tests help grasp the logic of autograd.

Exercise First loss and gradient

With the parameters at zero, compute the MSE loss and call backward(). Print the loss, w.grad and b.grad with item() on one line. (The y values are 2, 5, ..., 29; mean square = 314.5.)

Show example solution

import torch

x = torch.arange(10, dtype=torch.float32).unsqueeze(1)
y = 3 * x + 2
w = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)

def model(x):
    return x * w + b

kayip = ((model(x) - y) ** 2).mean()
kayip.backward()
print(kayip.item(), w.grad.item(), b.grad.item())

Test yourself

If the gradient is negative, how does gradient descent change the parameter?
- Decreases it
- Zeroes it
- Doesn't change it
- Increases it
Show answer

Increases it — w ← w - lr·grad; subtracting a negative grad increases w.

Compute the mean squared error.

kayip = ((tahmin - hedef) ** 2).____()

Show answer

kayip = ((tahmin - hedef) ** 2).mean()

Match the derivatives.

∂L/∂w

mean(2(ŷ - y)·x)
∂L/∂b

mean(2(ŷ - y))
negative gradient

The parameter must grow
MSE

Regression loss

Next lesson: Optimizer

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies