CodeBlanks Logo Icon CodeBlanks
Login Register

Computational graph

Autograd records operations as a computational graph.

The computational graph

Autograd records each operation as a node and builds a directed computational graph. Each intermediate tensor's grad_fn attribute shows the operation that produced it; backward() applies the chain rule by walking this graph in reverse:

x = torch.tensor(2.0, requires_grad=True)
y = x * 3
z = y.exp()
print(type(y.grad_fn).__name__)   # MulBackward0
print(type(z.grad_fn).__name__)   # ExpBackward0
print(x.grad_fn)                  # None: a leaf

The graph is rebuilt on every forward pass (a dynamic graph); that's why Python's if and loops can be used freely inside the model.

The chain rule step by step

While the graph is walked backwards, each node multiplies its local derivative by the incoming gradient. For z = exp(3x):

# forward:  x=2 -> y=3x=6 -> z=exp(y)
# backward: dz/dy = exp(y) = exp(6)
#           dy/dx = 3
#           dz/dx = 3·exp(6)
z.backward()
print(round(x.grad.item(), 1))   # 1210.3

In branching graphs, if a tensor is used in several places the gradients are summed: for y = x * x, dy/dx = x + x = 2x.

Practice Follow the graph

For x = torch.tensor(1.0, requires_grad=True), compute a = x * x and b = a + x. Print the grad_fn class names of a and b on one line. Then call b.backward() and print x.grad.item() (db/dx = 2x + 1).

Starter code

model.py
import torch

# buraya yaz

Sample solution

model.py
import torch

x = torch.tensor(1.0, requires_grad=True)
a = x * x
b = a + x
print(type(a.grad_fn).__name__, type(b.grad_fn).__name__)
b.backward()
print(x.grad.item())

Expected output

MulBackward0 AddBackward0
3.0

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the <code>grad_fn</code> of a leaf tensor?

  • AddBackward0
  • Itself
  • 0
  • None Correct answer

Explanation: Leaves weren't produced by an operation; they are the starting points of the graph.

2 Show the operation that produced the tensor.

Question 
print(type(z.____).__name__)
Answer 
print(type(z.grad_fn).__name__)

Explanation: grad_fn is the node in the graph.

3 Match the nodes.

Term Meaning
x * 3 MulBackward0
y.exp() ExpBackward0
a + b AddBackward0
leaf grad_fn = None

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4