Machine Learning
forward
The forward method defines how data passes through the model.
forward
forward defines how the input passes through the module. It is ordinary Python code: conditions, loops, several inputs and outputs can be used. The backward pass isn't written; autograd derives it from the forward pass by itself:
class Kalinti(nn.Module):
def __init__(self, d):
super().__init__()
self.l1 = nn.Linear(d, d)
self.l2 = nn.Linear(d, d)
def forward(self, x):
return x + self.l2(torch.relu(self.l1(x))) # residual connection
Adding its own transformation to the input (a residual connection) is the key idea that makes very deep networks trainable.
The functional interface
Operations without parameters (activation, pooling) can also be called as functions with torch.nn.functional. Layers with parameters, however, must be created as modules in __init__:
import torch.nn.functional as F
def forward(self, x):
x = F.relu(self.l1(x)) # the same as the nn.ReLU() module
x = F.dropout(x, 0.2, training=self.training)
return self.l2(x)
Creating an nn.Linear inside forward produces new random weights on every call; the model can learn nothing.
Exercise Residual block
In the given Kalinti block, zero l2's weight and bias (zero_() inside torch.no_grad()). In that case the block must pass the input through unchanged; print the result of torch.equal(blok(x), x) and the output shape on one line.
Show example solution
import torch
from torch import nn
class Kalinti(nn.Module):
def __init__(self, d):
super().__init__()
self.l1 = nn.Linear(d, d)
self.l2 = nn.Linear(d, d)
def forward(self, x):
return x + self.l2(torch.relu(self.l1(x)))
blok = Kalinti(6)
x = torch.randn(4, 6)
with torch.no_grad():
blok.l2.weight.zero_()
blok.l2.bias.zero_()
sonuc = blok(x)
print(torch.equal(sonuc, x), sonuc.shape)
Test yourself
-
How is the backward pass defined in a module?
- The derivatives are entered by hand
- The optimiser defines it
- It isn't; autograd derives it from forward
- A backward method is written
Show answer
It isn't; autograd derives it from forward — Every tensor operation in forward is recorded in the graph.
-
Apply a functional ReLU.
x = F.____(self.l1(x))
Show answer
x = F.relu(self.l1(x))
-
Match the concepts.
- forward
- Forward computation
- F.relu
- Functional activation
- residual
- x + f(x)
- __init__
- Layers with parameters
Next lesson: nn.Linear
