Optimizer
- Machine Learning course
- Advanced
- Lesson 28/78
- 7 min
Create an SGD optimizer.
The optimiser step
The training loop is the repetition of four steps: forward pass, loss, backward pass, update (and zeroing the gradients). Written by hand:
lr = 0.01
for adim in range(2000):
kayip = ((model(x) - y) ** 2).mean()
kayip.backward()
with torch.no_grad():
for p in (w, b):
p -= lr * p.grad
p.grad.zero_()If the learning rate is too large the loss explodes; if too small, training progresses very slowly.
The same job with optim.SGD
PyTorch's optimiser takes over the update and zeroing; the loop gets shorter and less error-prone:
opt = torch.optim.SGD([w, b], lr=0.01)
for adim in range(2000):
opt.zero_grad()
kayip = ((model(x) - y) ** 2).mean()
kayip.backward()
opt.step()The zero_grad, backward, step trio is the skeleton of every PyTorch training loop.
Practice Train the model
Train for 3000 steps with torch.optim.SGD([w, b], lr=0.01) (each step: zero_grad, loss, backward, step). Print the learned w and b with round(..., 2) on one line, then that the final loss is below 0.001.
Starter code
import torch
x = torch.arange(10, dtype=torch.float32).unsqueeze(1)
y = 3 * x + 2
w = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
def model(x):
return x * w + b
# buraya yazSample solution
import torch
x = torch.arange(10, dtype=torch.float32).unsqueeze(1)
y = 3 * x + 2
w = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)
def model(x):
return x * w + b
opt = torch.optim.SGD([w, b], lr=0.01)
for _ in range(3000):
opt.zero_grad()
kayip = ((model(x) - y) ** 2).mean()
kayip.backward()
opt.step()
print(round(w.item(), 2), round(b.item(), 2))
print(kayip.item() < 0.001)Expected output
3.0 2.0 True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens on this data with <code>lr = 0.1</code>?
- Nothing changes
- The same result
- The loss explodes (diverges) Correct answer
- It converges faster
Explanation: Since x goes up to 9 the gradient is large; a big step overshoots the optimum and the error grows each step.
2 Update the parameters.
opt.____()opt.step()Explanation: step() updates the parameters with the gradients.
3 Match the loop steps.
| Term | Meaning |
|---|---|
opt.zero_grad() |
Zero the gradients |
model(x) |
Forward pass |
kayip.backward() |
Backward pass |
opt.step() |
Update |