CodeBlanks Logo Icon CodeBlanks
Login Register

Neuron

A neuron multiplies inputs by weights and applies an activation function.

The neuron

A single neuron multiplies its inputs by weights and sums them, adds a bias and passes the result through an activation function: a = f(w · x + b).

x = np.array([1.0, 2.0, 3.0])
w = np.array([0.5, -1.0, 0.25])
b = 0.5
z = w @ x + b          # 0.5 - 2 + 0.75 + 0.5 = -0.25
a = max(0.0, z)       # ReLU: 0.0

The weights set each input's importance, the bias how easily the neuron "fires".

Perceptron and its limits

A single neuron with a step activation (a perceptron) draws a linear decision boundary. It can learn logic gates like AND and OR but not XOR, because XOR's classes can't be separated by a single line:

# AND: w = [1, 1], b = -1.5  -> 1 only for (1, 1)
adim = lambda z: (z > 0).astype(int)
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
print(adim(X @ np.array([1, 1]) - 1.5))   # [0 0 0 1]

XOR needs at least one hidden layer; that is where the power of multi-layer networks comes from.

Practice The OR gate

Build the OR gate with a single step-function neuron: w = [1, 1], b = -0.5. Print the outputs for the four inputs. Then print that using b = -1.5 with the same weights gives the AND gate.

Starter code

model.py
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
w = np.array([1, 1])

# buraya yaz

Sample solution

model.py
import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
w = np.array([1, 1])

adim = lambda z: (z > 0).astype(int)
print(adim(X @ w - 0.5))
print(adim(X @ w - 1.5))

Expected output

[0 1 1 1]
[0 0 0 1]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 What is the ReLU output of a neuron with w = [2, 1], x = [1, 3], b = -1?

  • 4 Correct answer
  • 5
  • 0
  • 6

Explanation: z = 2·1 + 1·3 - 1 = 4; ReLU(4) = 4.

2 Compute the neuron's weighted sum.

Question 
z = w @ x ____ b
Answer 
z = w @ x + b

Explanation: The bias is added to the weighted sum.

3 Match the parts.

Term Meaning
w Input importances
b Bias
z = w·x + b Pre-activation
XOR Unsolvable with one neuron

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4