Neuron

A neuron multiplies inputs by weights and applies an activation function.

The neuron

A single neuron multiplies its inputs by weights and sums them, adds a bias and passes the result through an activation function: a = f(w · x + b).

x = np.array([1.0, 2.0, 3.0])
w = np.array([0.5, -1.0, 0.25])
b = 0.5
z = w @ x + b          # 0.5 - 2 + 0.75 + 0.5 = -0.25
a = max(0.0, z)       # ReLU: 0.0

The weights set each input's importance, the bias how easily the neuron "fires".

Perceptron and its limits

A single neuron with a step activation (a perceptron) draws a linear decision boundary. It can learn logic gates like AND and OR but not XOR, because XOR's classes can't be separated by a single line:

# AND: w = [1, 1], b = -1.5  -> 1 only for (1, 1)
adim = lambda z: (z > 0).astype(int)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
print(adim(X @ np.array([1, 1]) - 1.5))   # [0 0 0 1]

XOR needs at least one hidden layer; that is where the power of multi-layer networks comes from.

Exercise The OR gate

Build the OR gate with a single step-function neuron: w = [1, 1], b = -0.5. Print the outputs for the four inputs. Then print that using b = -1.5 with the same weights gives the AND gate.

Show example solution

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
w = np.array([1, 1])

adim = lambda z: (z > 0).astype(int)
print(adim(X @ w - 0.5))
print(adim(X @ w - 1.5))

Test yourself

What is the ReLU output of a neuron with w = [2, 1], x = [1, 3], b = -1?
- 4
- 5
- 0
- 6
Show answer

4 — z = 2·1 + 1·3 - 1 = 4; ReLU(4) = 4.
Compute the neuron's weighted sum.
```
z = w @ x ____ b
```
Show answer
```
z = w @ x + b
```
Match the parts.

w

Input importances
b

Bias
z = w·x + b

Pre-activation
XOR

Unsolvable with one neuron

Next lesson: Layers

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies