What is a neural network?

A neural network is a layered model inspired by biological neural networks.

What is a neural network?

An artificial neural network is a model made of simple computing units (neurons) connected in layers. Each layer changes its input with a linear transformation and passes it through a non-linear function; stacked layers let the model learn very complex relationships.

# input (4 features) -> hidden layer (8 neurons) -> output (3 classes)
h = relu(X @ W1 + b1)    # (n, 8)
cikti = h @ W2 + b2      # (n, 3)

Logistic regression is a single-layer neural network; deep learning is the many-layered version of this structure.

Why now?

The idea of neural networks has existed since the 1950s. Recent success has three causes: large labelled datasets, the parallel matrix multiplication power of GPUs and better training techniques (ReLU, good initialisation, Adam, dropout).

Images: convolutional networks (CNN).
Sequences and text: RNN, LSTM and especially the transformer.
Tabular data: gradient boosting is often still better.

Exercise Composition of linear layers

Print with np.allclose that applying two linear layers (W1, W2) in a row without activation gives the same result as the single matrix W1 @ W2. Then print that the result is no longer the same once a ReLU (np.maximum(0, ·)) is put between them.

Show example solution

import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(5, 3))
W1 = rng.normal(size=(3, 4))
W2 = rng.normal(size=(4, 2))

print(np.allclose((X @ W1) @ W2, X @ (W1 @ W2)))
print(np.allclose(np.maximum(0, X @ W1) @ W2, X @ (W1 @ W2)))

Test yourself

What would happen without the non-linear function between hidden layers?
- Nothing would change
- The whole network would reduce to a single linear transformation
- The network would be more powerful
- Training would be impossible
Show answer

The whole network would reduce to a single linear transformation — The composition of two linear transformations is still linear: W2(W1x) = (W2W1)x.

Compute the hidden layer.

h = np.maximum(0, X ____ W1 + b1)

Show answer

h = np.maximum(0, X @ W1 + b1)

Match the concepts.

neuron

Simple computing unit
layer

Group of neurons
W

Learned weights
ReLU

Non-linear function

Next lesson: Neuron

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies