CodeBlanks Logo Icon CodeBlanks
Login Register

Layers

A neural network has input, hidden, and output layers.

Layers

A layer is a group of neurons receiving the same input. All the neurons' weights are collected in a matrix; the layer's output is then computed with a single matrix multiplication:

# 3 inputs, a 4-neuron layer
W = rng.normal(size=(3, 4))   # each column is a neuron
b = np.zeros(4)
h = X @ W + b                  # (n, 3) @ (3, 4) -> (n, 4)

The input layer takes the data, hidden layers produce intermediate representations, the output layer gives the prediction.

Depth and width

A network's width is the number of neurons per layer, its depth the number of layers. Deep networks learn hierarchical features: in images the first layers learn edges, later ones shapes, the top ones objects.

katmanlar = [784, 128, 64, 10]   # input, two hidden, output
param = sum(a * b + b for a, b in zip(katmanlar[:-1], katmanlar[1:]))
print(param)   # 109386

As the parameter count grows the model gets more flexible but needs more data and regularisation.

Practice Parameter count

For a network with layer sizes [784, 256, 128, 10], print each layer's parameter count (weights + biases) as a list, then the total.

Starter code

model.py
katmanlar = [784, 256, 128, 10]

# buraya yaz

Sample solution

model.py
katmanlar = [784, 256, 128, 10]

param = [a * b + b for a, b in zip(katmanlar[:-1], katmanlar[1:])]
print(param)
print(sum(param))

Expected output

[200960, 32896, 1290]
235146

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 How many parameters does a layer with 10 inputs and 32 neurons have?

  • 320
  • 42
  • 32
  • 352 Correct answer

Explanation: Weights 10 × 32 = 320, biases 32: 352 in total.

2 Compute the layer's output.

Question 
h = X @ W ____ b
Answer 
h = X @ W + b

Explanation: The bias vector is added to every row by broadcasting.

3 Match the layers.

Term Meaning
input layer Takes the data
hidden layer Intermediate representation
output layer Prediction
column of W One neuron

All lessons in Advanced

1 Neural network basics

  1. What is a neural network?
  2. Neuron
  3. Layers
  4. Activation
  5. Forward propagation
  6. Summary - Neural network

2 PyTorch basics

  1. What is PyTorch?
  2. Tensor
  3. Creating tensors
  4. Tensor properties
  5. GPU support
  6. Summary - PyTorch

3 Tensor operations

  1. Tensor operations
  2. Shape
  3. Dtype
  4. Reshape
  5. Indexing
  6. Summary - Tensor operations

4 Autograd

  1. What is autograd?
  2. requires_grad
  3. backward()
  4. grad
  5. Computational graph
  6. Summary - Autograd

5 Project: Neural network

  1. Preparing tensors
  2. Model definition
  3. Loss computation
  4. Optimizer

6 nn.Module

  1. What is nn.Module?
  2. __init__
  3. forward
  4. nn.Linear
  5. parameters()
  6. Summary - nn.Module

7 Training loop

  1. Training loop
  2. Epoch
  3. Loss
  4. backward()
  5. optimizer.step()
  6. Summary - Training loop

8 CNN

  1. What is CNN?
  2. Conv2d
  3. MaxPool2d
  4. Feature map
  5. Flatten
  6. Summary - CNN

9 RNN

  1. What is RNN?
  2. Hidden state
  3. Sequence
  4. Vanishing gradient
  5. Embedding
  6. Summary - RNN

10 LSTM

  1. What is LSTM?
  2. Gates
  3. Cell state
  4. input_size
  5. hidden_size
  6. Summary - LSTM

11 Project: CNN

  1. Preparing data
  2. CNN model
  3. Loss and optimizer
  4. Training

12 Transfer learning

  1. What is transfer learning?
  2. Pre-trained model
  3. Fine-tuning
  4. Feature extraction
  5. Advantages
  6. Summary - Transfer learning

13 Optimization

  1. What is an optimizer?
  2. SGD
  3. Adam
  4. Learning rate
  5. zero_grad()
  6. Summary - Optimization

14 Advanced final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4