Summary - Neural network

In this group you learned neurons, layers, activation, and forward propagation.

Summary: neural networks

In this group:

Neuron: f(w·x + b); a single neuron draws a linear boundary and can't solve XOR.
Layer: weight matrix + bias; parameter count inputs·outputs + outputs.
Activations: ReLU (hidden), sigmoid/softmax (output); layers without activation collapse into one linear transformation.
Forward pass: X → linear → ReLU → linear → softmax; the loss is cross-entropy.

In the next groups we'll do the same computations with PyTorch, with automatic differentiation and on the GPU.

Exercise Solve XOR by hand

Solve XOR with a network of two hidden neurons: hidden layer W1 = [[1, 1], [1, 1]], b1 = [-0.5, -1.5] (OR and AND), step activation; output w2 = [1, -1], b2 = -0.5, again with a step. Print the outputs for the four inputs.

Show example solution

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
adim = lambda z: (z > 0).astype(int)

W1 = np.array([[1, 1], [1, 1]])
b1 = np.array([-0.5, -1.5])
h = adim(X @ W1 + b1)
print(adim(h @ np.array([1, -1]) - 0.5))

Test yourself

What is needed at minimum to solve the XOR problem?
- A hidden layer with a non-linear activation
- More data
- A larger learning rate
- A single neuron
Show answer

A hidden layer with a non-linear activation — The hidden layer moves the input into a new space where the classes are linearly separable.

Compute the probability sum per row.

p = e / e.sum(axis=1, ____=True)

Show answer

p = e / e.sum(axis=1, keepdims=True)

Match the concepts.

perceptron

Single step neuron
hidden layer

Solves XOR
logit

Score before softmax
cross-entropy

Classification loss

Next lesson: What is PyTorch?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies