TransformerEncoderLayer

nn.TransformerEncoderLayer represents a single encoder block.

nn.TransformerEncoderLayer

PyTorch offers the encoder block ready-made. With batch_first=True the input has shape (batch, sequence, d_model):

import torch
from torch import nn

katman = nn.TransformerEncoderLayer(d_model=16, nhead=4, dim_feedforward=32,
                                    dropout=0.0, batch_first=True)
encoder = nn.TransformerEncoder(katman, num_layers=2)
x = torch.randn(2, 5, 16)
print(encoder(x).shape)          # torch.Size([2, 5, 16])

d_model must be divisible by nhead; each head is d_model/nhead dimensional.

Counting the parameters

A layer's parameters:

Attention input projection (Q, K, V together): 3·d·d + 3·d
Attention output projection: d·d + d
FFN: d·f + f and f·d + d
Two LayerNorms: 2 · 2d

d, f = 16, 32
dikkat = 3*d*d + 3*d + d*d + d          # 1088
ffn = d*f + f + f*d + d                 # 1072
norm = 4 * d                            # 64
print(dikkat + ffn + norm)              # 2224
print(sum(p.numel() for p in katman.parameters()))   # 2224

When f = 4d, the FFN carries about twice as many parameters as attention.

Exercise An encoder layer

Build a TransformerEncoderLayer with d_model=16, nhead=4, dim_feedforward=32, dropout=0.0, batch_first=True and put it in a 2-layer TransformerEncoder. Print the output shape for an input of shape (2, 5, 16), then the parameter count of a single layer.

Show example solution

import torch
from torch import nn

torch.manual_seed(0)
x = torch.randn(2, 5, 16)

katman = nn.TransformerEncoderLayer(d_model=16, nhead=4, dim_feedforward=32,
                                    dropout=0.0, batch_first=True)
encoder = nn.TransformerEncoder(katman, num_layers=2)
print(encoder(x).shape)
print(sum(p.numel() for p in katman.parameters()))

Test yourself

With d_model=16, nhead=4, what is each head's size?
- 20
- 4
- 16
- 64
Show answer

4 — 16 / 4 = 4.

Make the batch the first axis.

nn.TransformerEncoderLayer(d_model=16, nhead=4, ____=True)

Show answer

nn.TransformerEncoderLayer(d_model=16, nhead=4, batch_first=True)

Match the parameter counts (d=16, f=32).

dikkat

1088
FFN

1072
LayerNorm'lar

64
toplam

2224

Next lesson: Summary - Transformer

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies