forward
- Machine Learning course
- Advanced
- Lesson 31/78
- 5 min
The forward method defines how data passes through the model.
forward
forward defines how the input passes through the module. It is ordinary Python code: conditions, loops, several inputs and outputs can be used. The backward pass isn't written; autograd derives it from the forward pass by itself:
class Kalinti(nn.Module):
def __init__(self, d):
super().__init__()
self.l1 = nn.Linear(d, d)
self.l2 = nn.Linear(d, d)
def forward(self, x):
return x + self.l2(torch.relu(self.l1(x))) # residual connectionAdding its own transformation to the input (a residual connection) is the key idea that makes very deep networks trainable.
The functional interface
Operations without parameters (activation, pooling) can also be called as functions with torch.nn.functional. Layers with parameters, however, must be created as modules in __init__:
import torch.nn.functional as F
def forward(self, x):
x = F.relu(self.l1(x)) # the same as the nn.ReLU() module
x = F.dropout(x, 0.2, training=self.training)
return self.l2(x)Creating an nn.Linear inside forward produces new random weights on every call; the model can learn nothing.
Practice Residual block
In the given Kalinti block, zero l2's weight and bias (zero_() inside torch.no_grad()). In that case the block must pass the input through unchanged; print the result of torch.equal(blok(x), x) and the output shape on one line.
Starter code
import torch
from torch import nn
class Kalinti(nn.Module):
def __init__(self, d):
super().__init__()
self.l1 = nn.Linear(d, d)
self.l2 = nn.Linear(d, d)
def forward(self, x):
return x + self.l2(torch.relu(self.l1(x)))
blok = Kalinti(6)
x = torch.randn(4, 6)
# buraya yazSample solution
import torch
from torch import nn
class Kalinti(nn.Module):
def __init__(self, d):
super().__init__()
self.l1 = nn.Linear(d, d)
self.l2 = nn.Linear(d, d)
def forward(self, x):
return x + self.l2(torch.relu(self.l1(x)))
blok = Kalinti(6)
x = torch.randn(4, 6)
with torch.no_grad():
blok.l2.weight.zero_()
blok.l2.bias.zero_()
sonuc = blok(x)
print(torch.equal(sonuc, x), sonuc.shape)Expected output
True torch.Size([4, 6])
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How is the backward pass defined in a module?
- The derivatives are entered by hand
- The optimiser defines it
- It isn't; autograd derives it from forward Correct answer
- A backward method is written
Explanation: Every tensor operation in forward is recorded in the graph.
2 Apply a functional ReLU.
x = F.____(self.l1(x))x = F.relu(self.l1(x))Explanation: F.relu is a parameter-free function.
3 Match the concepts.
| Term | Meaning |
|---|---|
forward |
Forward computation |
F.relu |
Functional activation |
residual |
x + f(x) |
__init__ |
Layers with parameters |