Computational graph

Autograd records operations as a computational graph.

The computational graph

Autograd records each operation as a node and builds a directed computational graph. Each intermediate tensor's grad_fn attribute shows the operation that produced it; backward() applies the chain rule by walking this graph in reverse:

x = torch.tensor(2.0, requires_grad=True)
y = x * 3
z = y.exp()
print(type(y.grad_fn).__name__)   # MulBackward0
print(type(z.grad_fn).__name__)   # ExpBackward0
print(x.grad_fn)                  # None: a leaf

The graph is rebuilt on every forward pass (a dynamic graph); that's why Python's if and loops can be used freely inside the model.

The chain rule step by step

While the graph is walked backwards, each node multiplies its local derivative by the incoming gradient. For z = exp(3x):

# forward:  x=2 -> y=3x=6 -> z=exp(y)
# backward: dz/dy = exp(y) = exp(6)
#           dy/dx = 3
#           dz/dx = 3·exp(6)
z.backward()
print(round(x.grad.item(), 1))   # 1210.3

In branching graphs, if a tensor is used in several places the gradients are summed: for y = x * x, dy/dx = x + x = 2x.

Exercise Follow the graph

For x = torch.tensor(1.0, requires_grad=True), compute a = x * x and b = a + x. Print the grad_fn class names of a and b on one line. Then call b.backward() and print x.grad.item() (db/dx = 2x + 1).

Show example solution

import torch

x = torch.tensor(1.0, requires_grad=True)
a = x * x
b = a + x
print(type(a.grad_fn).__name__, type(b.grad_fn).__name__)
b.backward()
print(x.grad.item())

Test yourself

What is the <code>grad_fn</code> of a leaf tensor?
- AddBackward0
- Itself
- 0
- None
Show answer

None — Leaves weren't produced by an operation; they are the starting points of the graph.

Show the operation that produced the tensor.

print(type(z.____).__name__)

Show answer

print(type(z.grad_fn).__name__)

Match the nodes.

x * 3

MulBackward0
y.exp()

ExpBackward0
a + b

AddBackward0
leaf

grad_fn = None

Next lesson: Summary - Autograd

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies