Machine Learning
Forward propagation
Forward propagation is the input passing through layers to produce output.
Forward propagation
Forward propagation is the input being transformed layer by layer until it reaches the output. In a two-layer classifier:
z1 = X @ W1 + b1 # (n, hidden)
h = np.maximum(0, z1) # ReLU
z2 = h @ W2 + b2 # (n, classes) the "logits"
p = softmax(z2) # probability per row
When the network is trained, a prediction is first made with forward propagation, then the error is distributed to the weights with backpropagation; you'll see this in the autograd group.
Cross-entropy loss
In classification the quality of a prediction is measured by the negative logarithm of the probability given to the correct class. A prediction giving the correct class a probability near 1 gets a low loss, one near 0 a high loss:
def capraz_entropi(p, y):
return -np.log(p[np.arange(len(y)), y]).mean()
p = np.array([[0.7, 0.2, 0.1], [0.1, 0.1, 0.8]])
print(round(capraz_entropi(p, np.array([0, 2])), 3)) # 0.290
p[np.arange(n), y] is fancy indexing that picks the correct class's probability from each row.
Exercise Two-layer forward pass
Do the forward pass of the two-layer network with the given weights: a ReLU hidden layer and a softmax output (per row, axis=1). Print the probabilities' shape and the row sums with np.round(..., 3), then the cross-entropy loss for y with round(..., 3).
Show example solution
import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(size=(4, 3))
y = np.array([0, 1, 2, 1])
W1 = rng.normal(size=(3, 5))
b1 = np.zeros(5)
W2 = rng.normal(size=(5, 3))
b2 = np.zeros(3)
h = np.maximum(0, X @ W1 + b1)
z = h @ W2 + b2
e = np.exp(z - z.max(axis=1, keepdims=True))
p = e / e.sum(axis=1, keepdims=True)
print(p.shape)
print(np.round(p.sum(axis=1), 3))
print(round(-np.log(p[np.arange(len(y)), y]).mean(), 3))
Test yourself
-
What are the raw outputs before softmax called?
- Labels
- Logits
- Probabilities
- Gradients
Show answer
Logits — Logits are unbounded scores; softmax turns them into probabilities.
-
Pick the correct class's probability from each row.
dogru = p[np.____(len(y)), y]
Show answer
dogru = p[np.arange(len(y)), y]
-
Match the steps.
- X @ W1 + b1
- First linear step
- np.maximum(0, ·)
- ReLU
- h @ W2 + b2
- Logits
- -log p[correct]
- Cross-entropy
Next lesson: Summary - Neural network
