SGD
- Machine Learning course
- Advanced
- Lesson 70/78
- 5 min
SGD updates weights on each example.
SGD and momentum
The name "stochastic" comes from computing the gradient from a random batch rather than all the data; it is noisy but cheap. Momentum keeps a running sum of past gradients: it speeds up in consistent directions and slows down in zigzags:
# PyTorch momentum rule:
# buf = momentum · buf + g
# p -= lr · buf
opt = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)With momentum 0.9, under a constant gradient the step reaches about 10 times a single gradient (1 / (1 - 0.9)).
Momentum step by step
The first two steps from w = 0 for f(w) = (w - 3)² with lr = 0.1 and momentum = 0.9:
# step 1: g = -6, buf = -6, w = 0 + 0.6 = 0.6
# step 2: g = -4.8, buf = 0.9·(-6) - 4.8 = -10.2, w = 0.6 + 1.02 = 1.62
# with plain SGD w would be 1.08 after step 2Momentum can overshoot the optimum and come back; large momentum values produce oscillation.
Practice See momentum
For f(w) = (w - 3)², take 2 steps with SGD([w], lr=0.1, momentum=0.9); after every step print w with round(..., 3) on separate lines.
Starter code
import torch
w = torch.tensor(0.0, requires_grad=True)
# buraya yazSample solution
import torch
w = torch.tensor(0.0, requires_grad=True)
opt = torch.optim.SGD([w], lr=0.1, momentum=0.9)
for _ in range(2):
opt.zero_grad()
((w - 3) ** 2).backward()
opt.step()
print(round(w.item(), 3))Expected output
0.6 1.62
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why does momentum help in a narrow, long valley?
- It ignores the gradient
- It doesn't help
- It accumulates the consistent direction along the valley and damps side oscillations Correct answer
- It zeroes the learning rate
Explanation: Zigzagging components cancel each other while the advancing component accumulates.
2 Set up SGD with momentum.
opt = torch.optim.SGD(params, lr=0.01, ____=0.9)opt = torch.optim.SGD(params, lr=0.01, momentum=0.9)Explanation: momentum is the weight of past gradients.
3 Match the values (lr = 0.1, momentum = 0.9).
| Term | Meaning |
|---|---|
w after step 1 |
0.6 |
w after step 2 |
1.62 |
plain SGD step 2 |
1.08 |
1 / (1 - 0.9) |
10× speed-up |