Forward propagation
- Machine Learning course
- Advanced
- Lesson 5/78
- 5 min
Forward propagation is the input passing through layers to produce output.
Forward propagation
Forward propagation is the input being transformed layer by layer until it reaches the output. In a two-layer classifier:
z1 = X @ W1 + b1 # (n, hidden)
h = np.maximum(0, z1) # ReLU
z2 = h @ W2 + b2 # (n, classes) the "logits"
p = softmax(z2) # probability per rowWhen the network is trained, a prediction is first made with forward propagation, then the error is distributed to the weights with backpropagation; you'll see this in the autograd group.
Cross-entropy loss
In classification the quality of a prediction is measured by the negative logarithm of the probability given to the correct class. A prediction giving the correct class a probability near 1 gets a low loss, one near 0 a high loss:
def capraz_entropi(p, y):
return -np.log(p[np.arange(len(y)), y]).mean()
p = np.array([[0.7, 0.2, 0.1], [0.1, 0.1, 0.8]])
print(round(capraz_entropi(p, np.array([0, 2])), 3)) # 0.290p[np.arange(n), y] is fancy indexing that picks the correct class's probability from each row.
Practice Two-layer forward pass
Do the forward pass of the two-layer network with the given weights: a ReLU hidden layer and a softmax output (per row, axis=1). Print the probabilities' shape and the row sums with np.round(..., 3), then the cross-entropy loss for y with round(..., 3).
Starter code
import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(size=(4, 3))
y = np.array([0, 1, 2, 1])
W1 = rng.normal(size=(3, 5))
b1 = np.zeros(5)
W2 = rng.normal(size=(5, 3))
b2 = np.zeros(3)
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(size=(4, 3))
y = np.array([0, 1, 2, 1])
W1 = rng.normal(size=(3, 5))
b1 = np.zeros(5)
W2 = rng.normal(size=(5, 3))
b2 = np.zeros(3)
h = np.maximum(0, X @ W1 + b1)
z = h @ W2 + b2
e = np.exp(z - z.max(axis=1, keepdims=True))
p = e / e.sum(axis=1, keepdims=True)
print(p.shape)
print(np.round(p.sum(axis=1), 3))
print(round(-np.log(p[np.arange(len(y)), y]).mean(), 3))Expected output
(4, 3) [1. 1. 1. 1.] 1.539
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What are the raw outputs before softmax called?
- Labels
- Logits Correct answer
- Probabilities
- Gradients
Explanation: Logits are unbounded scores; softmax turns them into probabilities.
2 Pick the correct class's probability from each row.
dogru = p[np.____(len(y)), y]dogru = p[np.arange(len(y)), y]Explanation: np.arange gives the row indices and y the column indices.
3 Match the steps.
| Term | Meaning |
|---|---|
X @ W1 + b1 |
First linear step |
np.maximum(0, ·) |
ReLU |
h @ W2 + b2 |
Logits |
-log p[correct] |
Cross-entropy |