Activation

Activation functions add non-linearity, with ReLU being the most common.

Activation functions

An activation function gives the network non-linearity. The most common ones:

sigmoid = lambda z: 1 / (1 + np.exp(-z))   # (0, 1)
tanh    = np.tanh                            # (-1, 1)
relu    = lambda z: np.maximum(0, z)         # [0, ∞)
z = np.array([-2.0, 0.0, 2.0])
print(relu(z), np.tanh(z).round(3))

The default choice in hidden layers is ReLU: it is cheap to compute and its gradient doesn't vanish in the positive region.

Output activations

The output layer's activation is chosen per problem:

Regression: no activation (linear).
Binary classification: sigmoid → a single probability.
Multi-class: softmax → probabilities summing to 1.

def softmax(z):
    e = np.exp(z - z.max())   # subtract the max for numerical stability
    return e / e.sum()
print(softmax(np.array([2.0, 1.0, 0.1])).round(3))   # [0.659 0.242 0.099]

Subtracting the largest value doesn't change the result but prevents exp from overflowing.

Exercise Softmax

Write a numerically stable softmax(z) function (subtract the max). Print the result for [1, 2, 3] with np.round(..., 3), then print with np.allclose that the result for [1001, 1002, 1003] is the same.

Show example solution

import numpy as np

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

print(np.round(softmax(np.array([1.0, 2.0, 3.0])), 3))
print(np.allclose(softmax(np.array([1001.0, 1002.0, 1003.0])), softmax(np.array([1.0, 2.0, 3.0]))))

Test yourself

What is sigmoid's main problem in deep networks?
- It gives negative outputs
- It is linear
- Its gradient approaches zero at the extremes (vanishing)
- It can't be computed
Show answer

Its gradient approaches zero at the extremes (vanishing) — Sigmoid's derivative is at most 0.25; multiplied across layers the gradient shrinks fast.

Write ReLU.

relu = lambda z: np.____(0, z)

Show answer

relu = lambda z: np.maximum(0, z)

Match the activations.

ReLU

max(0, z)
sigmoid

(0, 1) range
tanh

(-1, 1) range
softmax

Probability distribution

Next lesson: Forward propagation

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies