nn.Linear
- Machine Learning course
- Advanced
- Lesson 32/78
- 5 min
nn.Linear creates a fully connected layer.
nn.Linear
nn.Linear(in, out) computes y = x Wᵀ + b. The weight matrix has shape (out, in); careful, the opposite of the NumPy habit:
l = nn.Linear(3, 2)
print(l.weight.shape) # torch.Size([2, 3])
print(l.bias.shape) # torch.Size([2])
x = torch.randn(5, 3)
print(torch.allclose(l(x), x @ l.weight.T + l.bias)) # TrueThe weights are initialised with small random values by default (Kaiming uniform); bias=False removes the bias.
Weight initialisation
Initialisation affects the speed and stability of training. The nn.init module offers the common methods; Kaiming is preferred in deep ReLU networks, Xavier in tanh/sigmoid networks:
l = nn.Linear(256, 256)
nn.init.kaiming_normal_(l.weight, nonlinearity="relu")
nn.init.zeros_(l.bias)
# zeroing all weights can't break symmetry: all neurons learn the sameIf all weights start with the same value, every neuron gets the same gradient and stays the same; randomness breaks this symmetry.
Practice Compute the linear layer by hand
Set the weight of an nn.Linear(3, 2) layer to [[1, 0, -1], [2, 1, 0]] and its bias to [0.5, -1] (copy_ inside torch.no_grad()). For x = torch.tensor([[1., 2., 3.]]), print the output with tolist(), then print with torch.allclose that it equals x @ l.weight.T + l.bias.
Starter code
import torch
from torch import nn
l = nn.Linear(3, 2)
x = torch.tensor([[1., 2., 3.]])
# buraya yazSample solution
import torch
from torch import nn
l = nn.Linear(3, 2)
x = torch.tensor([[1., 2., 3.]])
with torch.no_grad():
l.weight.copy_(torch.tensor([[1., 0., -1.], [2., 1., 0.]]))
l.bias.copy_(torch.tensor([0.5, -1.]))
y = l(x)
print(y.tolist())
print(torch.allclose(y, x @ l.weight.T + l.bias))Expected output
[[-1.5, 3.0]] True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What is the weight shape of an <code>nn.Linear(128, 10)</code> layer?
- (128,)
- (10, 128) Correct answer
- (128, 10)
- (10,)
Explanation: PyTorch stores the weight as (out, in).
2 Build a linear layer without bias.
l = nn.Linear(10, 5, ____=False)l = nn.Linear(10, 5, bias=False)Explanation: bias=False removes the bias parameter.
3 Match the properties (nn.Linear(3, 2)).
| Term | Meaning |
|---|---|
weight.shape |
(2, 3) |
bias.shape |
(2,) |
parameter count |
8 |
shape of l(x) for a (5, 3) input |
(5, 2) |