CodeBlanks Logo Icon CodeBlanks
Login Register

backward()

The backward() method computes gradients from the loss.

backward()

backward() starts from a scalar result (usually the loss), walks the graph backwards and fills the .grad attribute of every leaf tensor. A vector result must first be reduced to a scalar:

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = (x ** 2).sum()      # a scalar
y.backward()
print(x.grad.tolist())  # [2.0, 4.0, 6.0]
# (x ** 2).backward()   error: non-scalar output

In training the loss is already the mean over the batch, i.e. a scalar.

The graph is used once

By default backward() deletes the graph to free memory; calling it a second time on the same graph raises an error. Since the forward pass is redone in every training step, this isn't a problem:

y = (x ** 2).sum()
y.backward()
# y.backward()    RuntimeError: the graph has been freed
y = (x ** 2).sum()   # a new forward pass, a new graph
y.backward()

If the graph must be kept, backward(retain_graph=True) is used; this is rarely needed.

Practice Gradient of the mean

For x = torch.tensor([2.0, 4.0, 6.0, 8.0], requires_grad=True), compute y = (x ** 2).mean() and backpropagate. Print y.item(), then x.grad.tolist() (derivative: 2x/4).

Starter code

model.py
import torch

# buraya yaz

Sample solution

model.py
import torch

x = torch.tensor([2.0, 4.0, 6.0, 8.0], requires_grad=True)
y = (x ** 2).mean()
y.backward()
print(y.item())
print(x.grad.tolist())

Expected output

30.0
[1.0, 2.0, 3.0, 4.0]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 For <code>y = x.mean()</code> (x has 4 elements), what is each element's gradient?

  • 0
  • 0.25 Correct answer
  • 1
  • 4

Explanation: Mean = Σx/4; each element enters the result with a coefficient of 1/4.

2 Reduce the vector result to a scalar.

Question 
y = (x ** 2).____()
y.backward()
Answer 
y = (x ** 2).sum()
y.backward()

Explanation: backward needs a scalar output.

3 Match the situations.

Term Meaning
scalar loss backward works
vector output sum/mean first
second backward Graph freed
retain_graph=True Keeps the graph

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4