grad

The grad property contains the computed gradients.

The grad attribute

.grad is filled on leaf tensors after backward(). An important detail: gradients accumulate. Each backward() call adds the new gradient on top of the existing one:

w = torch.tensor(1.0, requires_grad=True)
for _ in range(3):
    (2 * w).backward()
print(w.grad.item())     # 6.0: 2 + 2 + 2
w.grad.zero_()
print(w.grad.item())     # 0.0

That's why the gradients are zeroed at every step of the training loop: optimizer.zero_grad().

Leaves and intermediate tensors

Only the .grad of leaf tensors (is_leaf) created by the user is kept. Gradients of intermediate results are dropped to save memory; they can be requested with retain_grad() if needed:

x = torch.tensor(2.0, requires_grad=True)
h = x * 3            # intermediate tensor
h.retain_grad()
y = h ** 2
y.backward()
print(x.is_leaf, h.is_leaf)        # True False
print(h.grad.item(), x.grad.item())  # 12.0 36.0

The chain rule: dy/dx = dy/dh · dh/dx = 2h · 3 = 36.

Exercise See accumulation

For w = torch.tensor(3.0, requires_grad=True), call (w ** 2).backward() three times and print w.grad.item(). Then zero the gradient with zero_(), backpropagate once more and print it again.

Show example solution

import torch

w = torch.tensor(3.0, requires_grad=True)
for _ in range(3):
    (w ** 2).backward()
print(w.grad.item())
w.grad.zero_()
(w ** 2).backward()
print(w.grad.item())

Test yourself

What happens if training runs 5 steps without zeroing the gradients?
- Each step updates with the sum of the previous steps' gradients
- Nothing happens
- It errors
- The gradients stay zero
Show answer

Each step updates with the sum of the previous steps' gradients — Accumulation leads to wrong and ever-growing updates.
Zero the gradient.
```
w.grad.____()
```
Show answer
```
w.grad.zero_()
```
Match the concepts.

birikim

backward adds up
zero_grad

Zeroing every step
is_leaf

User tensor
retain_grad

Keep intermediate gradient

Next lesson: Computational graph

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies