CodeBlanks Logo Icon CodeBlanks
Login Register

Summary - Neural network

In this group you learned neurons, layers, activation, and forward propagation.

Summary: neural networks

In this group:

  • Neuron: f(w·x + b); a single neuron draws a linear boundary and can't solve XOR.
  • Layer: weight matrix + bias; parameter count inputs·outputs + outputs.
  • Activations: ReLU (hidden), sigmoid/softmax (output); layers without activation collapse into one linear transformation.
  • Forward pass: X → linear → ReLU → linear → softmax; the loss is cross-entropy.

In the next groups we'll do the same computations with PyTorch, with automatic differentiation and on the GPU.

Practice Solve XOR by hand

Solve XOR with a network of two hidden neurons: hidden layer W1 = [[1, 1], [1, 1]], b1 = [-0.5, -1.5] (OR and AND), step activation; output w2 = [1, -1], b2 = -0.5, again with a step. Print the outputs for the four inputs.

Starter code

model.py
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
adim = lambda z: (z > 0).astype(int)

# buraya yaz

Sample solution

model.py
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
adim = lambda z: (z > 0).astype(int)

W1 = np.array([[1, 1], [1, 1]])
b1 = np.array([-0.5, -1.5])
h = adim(X @ W1 + b1)
print(adim(h @ np.array([1, -1]) - 0.5))

Expected output

[0 1 1 0]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is needed at minimum to solve the XOR problem?

  • A hidden layer with a non-linear activation Correct answer
  • More data
  • A larger learning rate
  • A single neuron

Explanation: The hidden layer moves the input into a new space where the classes are linearly separable.

2 Compute the probability sum per row.

Question 
p = e / e.sum(axis=1, ____=True)
Answer 
p = e / e.sum(axis=1, keepdims=True)

Explanation: keepdims=True keeps the (n, 1) shape so broadcasting works.

3 Match the concepts.

Term Meaning
perceptron Single step neuron
hidden layer Solves XOR
logit Score before softmax
cross-entropy Classification loss

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4