CodeBlanks Logo Icon CodeBlanks
Login Register

Activation

Activation functions add non-linearity, with ReLU being the most common.

Activation functions

An activation function gives the network non-linearity. The most common ones:

sigmoid = lambda z: 1 / (1 + np.exp(-z))   # (0, 1)
tanh    = np.tanh                            # (-1, 1)
relu    = lambda z: np.maximum(0, z)         # [0, ∞)
z = np.array([-2.0, 0.0, 2.0])
print(relu(z), np.tanh(z).round(3))

The default choice in hidden layers is ReLU: it is cheap to compute and its gradient doesn't vanish in the positive region.

Output activations

The output layer's activation is chosen per problem:

  • Regression: no activation (linear).
  • Binary classification: sigmoid → a single probability.
  • Multi-class: softmax → probabilities summing to 1.
def softmax(z):
    e = np.exp(z - z.max())   # subtract the max for numerical stability
    return e / e.sum()
print(softmax(np.array([2.0, 1.0, 0.1])).round(3))   # [0.659 0.242 0.099]

Subtracting the largest value doesn't change the result but prevents exp from overflowing.

Practice Softmax

Write a numerically stable softmax(z) function (subtract the max). Print the result for [1, 2, 3] with np.round(..., 3), then print with np.allclose that the result for [1001, 1002, 1003] is the same.

Starter code

model.py
import numpy as np

# buraya yaz

Sample solution

model.py
import numpy as np

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

print(np.round(softmax(np.array([1.0, 2.0, 3.0])), 3))
print(np.allclose(softmax(np.array([1001.0, 1002.0, 1003.0])), softmax(np.array([1.0, 2.0, 3.0]))))

Expected output

[0.09  0.245 0.665]
True

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is sigmoid's main problem in deep networks?

  • It gives negative outputs
  • It is linear
  • Its gradient approaches zero at the extremes (vanishing) Correct answer
  • It can't be computed

Explanation: Sigmoid's derivative is at most 0.25; multiplied across layers the gradient shrinks fast.

2 Write ReLU.

Question 
relu = lambda z: np.____(0, z)
Answer 
relu = lambda z: np.maximum(0, z)

Explanation: np.maximum takes the element-wise maximum; np.max gives the array's maximum.

3 Match the activations.

Term Meaning
ReLU max(0, z)
sigmoid (0, 1) range
tanh (-1, 1) range
softmax Probability distribution

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4