Encoder block

Define a transformer encoder block.

Building the encoder block

We wrap the attention layer with a residual connection, LayerNorm and FFN to make a full encoder block. A positional code is also added to the input:

def encoder(X, pad):
    H = E[X] + konum_kodu(X.shape[1], d)
    Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
    H = layer_norm(H + Y)
    H = layer_norm(H + np.maximum(0, H @ W1) @ W2)
    return H, A

The output is a contextual vector for each word; the representation of "iyi" now also depends on the rest of the sentence.

The sentence vector: a masked mean

Classification needs a single vector per sentence. A simple way is averaging only the real words (BERT uses the [CLS] token instead):

m = (~pad)[:, :, None]                   # (5, 5, 1)
cumle = (H * m).sum(axis=1) / m.sum(axis=1)   # (5, d)

Including padding in the mean would pull short sentences' vectors toward the padding's representation.

Exercise The encoder and the sentence vector

Complete the encoder(X, pad) function and compute the sentence vectors with a masked mean. Print the shapes of H and the sentence vectors on one line. Then print on one line with np.allclose that the representations of "iyi" in sentence 1 (position 2) and sentence 3 (position 0) differ, and with np.allclose that padding doesn't affect the sentence vector (by changing the padding embedding and recomputing).

Show example solution

import numpy as np

def softmax(z, eksen=-1):
    e = np.exp(z - z.max(axis=eksen, keepdims=True))
    return e / e.sum(axis=eksen, keepdims=True)

metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
    for k in c.split():
        sozluk.setdefault(k, len(sozluk))

def kodla(c, n=5):
    ids = [sozluk[k] for k in c.split()][:n]
    return ids + [0] * (n - len(ids))

X = np.array([kodla(c) for c in metinler])
pad = X == 0

def dikkat(Q, K, V, pad):
    d = Q.shape[-1]
    skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
    skor = np.where(pad[:, None, :], -1e9, skor)
    A = softmax(skor)
    return A @ V, A

def layer_norm(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)

def konum_kodu(n, d):
    aci = np.arange(n)[:, None] / 10000 ** (np.arange(0, d, 2) / d)
    PE = np.zeros((n, d))
    PE[:, 0::2], PE[:, 1::2] = np.sin(aci), np.cos(aci)
    return PE

rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
W1 = rng.normal(size=(d, 4 * d)) * 0.3
W2 = rng.normal(size=(4 * d, d)) * 0.3

def encoder(X, pad):
    H = E[X] + konum_kodu(X.shape[1], d)
    Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
    H = layer_norm(H + Y)
    return layer_norm(H + np.maximum(0, H @ W1) @ W2)

def cumle_vektoru(H, pad):
    m = (~pad)[:, :, None]
    return (H * m).sum(axis=1) / m.sum(axis=1)

H = encoder(X, pad)
C = cumle_vektoru(H, pad)
print(H.shape, C.shape)
E[0] += 10
C2 = cumle_vektoru(encoder(X, pad), pad)
print(np.allclose(H[0, 2], H[2, 0]), np.allclose(C, C2))

Test yourself

Is the encoder output for "iyi" the same in the sentences "film çok iyi" and "iyi"?
- The encoder doesn't look at the word
- No; it changes because the context and position differ
- Yes, the same word is the same vector
- Only the padding differs
Show answer

No; it changes because the context and position differ — The embedding is the same, but attention and the positional code change the representation according to context.

Sum only the real words.

cumle = (H * m).sum(axis=1) / m.____(axis=1)

Show answer

cumle = (H * m).sum(axis=1) / m.sum(axis=1)

Match.

H + Y

Residual connection
konum_kodu

Order information
maskeli ortalama

Sentence vector
[CLS]

BERT's sentence token

Next lesson: Results

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies