Optimizer

Create an SGD optimizer.

The optimiser step

The training loop is the repetition of four steps: forward pass, loss, backward pass, update (and zeroing the gradients). Written by hand:

lr = 0.01
for adim in range(2000):
    kayip = ((model(x) - y) ** 2).mean()
    kayip.backward()
    with torch.no_grad():
        for p in (w, b):
            p -= lr * p.grad
            p.grad.zero_()

If the learning rate is too large the loss explodes; if too small, training progresses very slowly.

The same job with optim.SGD

PyTorch's optimiser takes over the update and zeroing; the loop gets shorter and less error-prone:

opt = torch.optim.SGD([w, b], lr=0.01)
for adim in range(2000):
    opt.zero_grad()
    kayip = ((model(x) - y) ** 2).mean()
    kayip.backward()
    opt.step()

The zero_grad, backward, step trio is the skeleton of every PyTorch training loop.

Exercise Train the model

Train for 3000 steps with torch.optim.SGD([w, b], lr=0.01) (each step: zero_grad, loss, backward, step). Print the learned w and b with round(..., 2) on one line, then that the final loss is below 0.001.

Show example solution

import torch

x = torch.arange(10, dtype=torch.float32).unsqueeze(1)
y = 3 * x + 2
w = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)

def model(x):
    return x * w + b

opt = torch.optim.SGD([w, b], lr=0.01)
for _ in range(3000):
    opt.zero_grad()
    kayip = ((model(x) - y) ** 2).mean()
    kayip.backward()
    opt.step()
print(round(w.item(), 2), round(b.item(), 2))
print(kayip.item() < 0.001)

Test yourself

What happens on this data with <code>lr = 0.1</code>?
- Nothing changes
- The same result
- The loss explodes (diverges)
- It converges faster
Show answer

The loss explodes (diverges) — Since x goes up to 9 the gradient is large; a big step overshoots the optimum and the error grows each step.
Update the parameters.
```
opt.____()
```
Show answer
```
opt.step()
```
Match the loop steps.

opt.zero_grad()

Zero the gradients
model(x)

Forward pass
kayip.backward()

Backward pass
opt.step()

Update

Next lesson: What is nn.Module?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies