grad
- Machine Learning course
- Advanced
- Lesson 22/78
- 5 min
The grad property contains the computed gradients.
The grad attribute
.grad is filled on leaf tensors after backward(). An important detail: gradients accumulate. Each backward() call adds the new gradient on top of the existing one:
w = torch.tensor(1.0, requires_grad=True)
for _ in range(3):
(2 * w).backward()
print(w.grad.item()) # 6.0: 2 + 2 + 2
w.grad.zero_()
print(w.grad.item()) # 0.0That's why the gradients are zeroed at every step of the training loop: optimizer.zero_grad().
Leaves and intermediate tensors
Only the .grad of leaf tensors (is_leaf) created by the user is kept. Gradients of intermediate results are dropped to save memory; they can be requested with retain_grad() if needed:
x = torch.tensor(2.0, requires_grad=True)
h = x * 3 # intermediate tensor
h.retain_grad()
y = h ** 2
y.backward()
print(x.is_leaf, h.is_leaf) # True False
print(h.grad.item(), x.grad.item()) # 12.0 36.0The chain rule: dy/dx = dy/dh · dh/dx = 2h · 3 = 36.
Practice See accumulation
For w = torch.tensor(3.0, requires_grad=True), call (w ** 2).backward() three times and print w.grad.item(). Then zero the gradient with zero_(), backpropagate once more and print it again.
Starter code
import torch
# buraya yazSample solution
import torch
w = torch.tensor(3.0, requires_grad=True)
for _ in range(3):
(w ** 2).backward()
print(w.grad.item())
w.grad.zero_()
(w ** 2).backward()
print(w.grad.item())Expected output
18.0 6.0
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What happens if training runs 5 steps without zeroing the gradients?
- Each step updates with the sum of the previous steps' gradients Correct answer
- Nothing happens
- It errors
- The gradients stay zero
Explanation: Accumulation leads to wrong and ever-growing updates.
2 Zero the gradient.
w.grad.____()w.grad.zero_()Explanation: zero_() zeroes the gradient tensor in place.
3 Match the concepts.
| Term | Meaning |
|---|---|
birikim |
backward adds up |
zero_grad |
Zeroing every step |
is_leaf |
User tensor |
retain_grad |
Keep intermediate gradient |