SGD

SGD updates weights on each example.

SGD and momentum

The name "stochastic" comes from computing the gradient from a random batch rather than all the data; it is noisy but cheap. Momentum keeps a running sum of past gradients: it speeds up in consistent directions and slows down in zigzags:

# PyTorch momentum rule:
# buf = momentum · buf + g
# p  -= lr · buf
opt = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

With momentum 0.9, under a constant gradient the step reaches about 10 times a single gradient (1 / (1 - 0.9)).

Momentum step by step

The first two steps from w = 0 for f(w) = (w - 3)² with lr = 0.1 and momentum = 0.9:

# step 1: g = -6,   buf = -6,                 w = 0 + 0.6 = 0.6
# step 2: g = -4.8, buf = 0.9·(-6) - 4.8 = -10.2, w = 0.6 + 1.02 = 1.62
# with plain SGD w would be 1.08 after step 2

Momentum can overshoot the optimum and come back; large momentum values produce oscillation.

Exercise See momentum

For f(w) = (w - 3)², take 2 steps with SGD([w], lr=0.1, momentum=0.9); after every step print w with round(..., 3) on separate lines.

Show example solution

import torch

w = torch.tensor(0.0, requires_grad=True)

opt = torch.optim.SGD([w], lr=0.1, momentum=0.9)
for _ in range(2):
    opt.zero_grad()
    ((w - 3) ** 2).backward()
    opt.step()
    print(round(w.item(), 3))

Test yourself

Why does momentum help in a narrow, long valley?
- It ignores the gradient
- It doesn't help
- It accumulates the consistent direction along the valley and damps side oscillations
- It zeroes the learning rate
Show answer

It accumulates the consistent direction along the valley and damps side oscillations — Zigzagging components cancel each other while the advancing component accumulates.

Set up SGD with momentum.

opt = torch.optim.SGD(params, lr=0.01, ____=0.9)

Show answer

opt = torch.optim.SGD(params, lr=0.01, momentum=0.9)

Match the values (lr = 0.1, momentum = 0.9).

w after step 1

0.6
w after step 2

1.62
plain SGD step 2

1.08
1 / (1 - 0.9)

10× speed-up

Next lesson: Adam

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies