CodeBlanks Logo Icon CodeBlanks
Login Register

TransformerEncoderLayer

nn.TransformerEncoderLayer represents a single encoder block.

nn.TransformerEncoderLayer

PyTorch offers the encoder block ready-made. With batch_first=True the input has shape (batch, sequence, d_model):

import torch
from torch import nn

katman = nn.TransformerEncoderLayer(d_model=16, nhead=4, dim_feedforward=32,
                                    dropout=0.0, batch_first=True)
encoder = nn.TransformerEncoder(katman, num_layers=2)
x = torch.randn(2, 5, 16)
print(encoder(x).shape)          # torch.Size([2, 5, 16])

d_model must be divisible by nhead; each head is d_model/nhead dimensional.

Counting the parameters

A layer's parameters:

  • Attention input projection (Q, K, V together): 3·d·d + 3·d
  • Attention output projection: d·d + d
  • FFN: d·f + f and f·d + d
  • Two LayerNorms: 2 · 2d
d, f = 16, 32
dikkat = 3*d*d + 3*d + d*d + d          # 1088
ffn = d*f + f + f*d + d                 # 1072
norm = 4 * d                            # 64
print(dikkat + ffn + norm)              # 2224
print(sum(p.numel() for p in katman.parameters()))   # 2224

When f = 4d, the FFN carries about twice as many parameters as attention.

Practice An encoder layer

Build a TransformerEncoderLayer with d_model=16, nhead=4, dim_feedforward=32, dropout=0.0, batch_first=True and put it in a 2-layer TransformerEncoder. Print the output shape for an input of shape (2, 5, 16), then the parameter count of a single layer.

Starter code

model.py
import torch
from torch import nn

torch.manual_seed(0)
x = torch.randn(2, 5, 16)

# buraya yaz

Sample solution

model.py
import torch
from torch import nn

torch.manual_seed(0)
x = torch.randn(2, 5, 16)

katman = nn.TransformerEncoderLayer(d_model=16, nhead=4, dim_feedforward=32,
                                    dropout=0.0, batch_first=True)
encoder = nn.TransformerEncoder(katman, num_layers=2)
print(encoder(x).shape)
print(sum(p.numel() for p in katman.parameters()))

Expected output

torch.Size([2, 5, 16])
2224

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 With d_model=16, nhead=4, what is each head's size?

  • 20
  • 4 Correct answer
  • 16
  • 64

Explanation: 16 / 4 = 4.

2 Make the batch the first axis.

Question 
nn.TransformerEncoderLayer(d_model=16, nhead=4, ____=True)
Answer 
nn.TransformerEncoderLayer(d_model=16, nhead=4, batch_first=True)

Explanation: The default is (sequence, batch, d) order; batch_first changes it.

3 Match the parameter counts (d=16, f=32).

Term Meaning
dikkat 1088
FFN 1072
LayerNorm'lar 64
toplam 2224

All lessons in Expert

1 Attention

  1. What is attention?
  2. Self-attention
  3. Query, Key, Value
  4. Scaled dot-product
  5. Multi-head
  6. Summary - Attention

2 Transformer

  1. What is a transformer?
  2. Encoder
  3. Decoder
  4. Positional encoding
  5. TransformerEncoderLayer
  6. Summary - Transformer

3 Project: Transformer

  1. Preparing data
  2. Attention layer
  3. Encoder block
  4. Results

4 BERT

  1. What is BERT?
  2. Masked LM
  3. Tokenization
  4. Fine-tuning
  5. CLS token
  6. Summary - BERT

5 GPT

  1. What is GPT?
  2. Autoregressive
  3. Decoder-only
  4. Causal mask
  5. Text generation
  6. Summary - GPT

6 GAN and VAE

  1. What is GAN?
  2. Generator
  3. Discriminator
  4. What is VAE?
  5. Encoder-Decoder
  6. Summary - GAN and VAE

7 Reinforcement learning

  1. What is RL?
  2. Agent
  3. Policy
  4. Reward
  5. Q-learning
  6. Summary - RL

8 Expert final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4