backward()

The backward() method computes gradients from the loss.

backward()

backward() starts from a scalar result (usually the loss), walks the graph backwards and fills the .grad attribute of every leaf tensor. A vector result must first be reduced to a scalar:

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = (x ** 2).sum()      # a scalar
y.backward()
print(x.grad.tolist())  # [2.0, 4.0, 6.0]
# (x ** 2).backward()   error: non-scalar output

In training the loss is already the mean over the batch, i.e. a scalar.

The graph is used once

By default backward() deletes the graph to free memory; calling it a second time on the same graph raises an error. Since the forward pass is redone in every training step, this isn't a problem:

y = (x ** 2).sum()
y.backward()
# y.backward()    RuntimeError: the graph has been freed
y = (x ** 2).sum()   # a new forward pass, a new graph
y.backward()

If the graph must be kept, backward(retain_graph=True) is used; this is rarely needed.

Exercise Gradient of the mean

For x = torch.tensor([2.0, 4.0, 6.0, 8.0], requires_grad=True), compute y = (x ** 2).mean() and backpropagate. Print y.item(), then x.grad.tolist() (derivative: 2x/4).

Show example solution

import torch

x = torch.tensor([2.0, 4.0, 6.0, 8.0], requires_grad=True)
y = (x ** 2).mean()
y.backward()
print(y.item())
print(x.grad.tolist())

Test yourself

For <code>y = x.mean()</code> (x has 4 elements), what is each element's gradient?
- 0
- 0.25
- 1
- 4
Show answer

0.25 — Mean = Σx/4; each element enters the result with a coefficient of 1/4.

Reduce the vector result to a scalar.

y = (x ** 2).____()
y.backward()

Show answer

y = (x ** 2).sum()
y.backward()

Match the situations.

scalar loss

backward works
vector output

sum/mean first
second backward

Graph freed
retain_graph=True

Keeps the graph

Next lesson: grad

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies