Machine Learning
nn.Linear
nn.Linear creates a fully connected layer.
nn.Linear
nn.Linear(in, out) computes y = x Wᵀ + b. The weight matrix has shape (out, in); careful, the opposite of the NumPy habit:
l = nn.Linear(3, 2)
print(l.weight.shape) # torch.Size([2, 3])
print(l.bias.shape) # torch.Size([2])
x = torch.randn(5, 3)
print(torch.allclose(l(x), x @ l.weight.T + l.bias)) # True
The weights are initialised with small random values by default (Kaiming uniform); bias=False removes the bias.
Weight initialisation
Initialisation affects the speed and stability of training. The nn.init module offers the common methods; Kaiming is preferred in deep ReLU networks, Xavier in tanh/sigmoid networks:
l = nn.Linear(256, 256)
nn.init.kaiming_normal_(l.weight, nonlinearity="relu")
nn.init.zeros_(l.bias)
# zeroing all weights can't break symmetry: all neurons learn the same
If all weights start with the same value, every neuron gets the same gradient and stays the same; randomness breaks this symmetry.
Exercise Compute the linear layer by hand
Set the weight of an nn.Linear(3, 2) layer to [[1, 0, -1], [2, 1, 0]] and its bias to [0.5, -1] (copy_ inside torch.no_grad()). For x = torch.tensor([[1., 2., 3.]]), print the output with tolist(), then print with torch.allclose that it equals x @ l.weight.T + l.bias.
Show example solution
import torch
from torch import nn
l = nn.Linear(3, 2)
x = torch.tensor([[1., 2., 3.]])
with torch.no_grad():
l.weight.copy_(torch.tensor([[1., 0., -1.], [2., 1., 0.]]))
l.bias.copy_(torch.tensor([0.5, -1.]))
y = l(x)
print(y.tolist())
print(torch.allclose(y, x @ l.weight.T + l.bias))
Test yourself
-
What is the weight shape of an <code>nn.Linear(128, 10)</code> layer?
- (128,)
- (10, 128)
- (128, 10)
- (10,)
Show answer
(10, 128) — PyTorch stores the weight as (out, in).
-
Build a linear layer without bias.
l = nn.Linear(10, 5, ____=False)
Show answer
l = nn.Linear(10, 5, bias=False)
-
Match the properties (nn.Linear(3, 2)).
- weight.shape
- (2, 3)
- bias.shape
- (2,)
- parameter count
- 8
- shape of l(x) for a (5, 3) input
- (5, 2)
Next lesson: parameters()
