Results

Print the model output.

Result: training the classifier

We don't train the encoder's weights in this project (that will be done with fine-tuning in the next BERT group). Instead we put a logistic regression on top of the sentence vectors; this is called linear probing:

from sklearn.linear_model import LogisticRegression

C = cumle_vektoru(encoder(X, pad), pad)
model = LogisticRegression(max_iter=1000).fit(C, y)
print(model.predict(cumle_vektoru(encoder(X_test, pad_test), pad_test)))

Probing is the standard way to measure whether a representation "carries" a piece of information.

Evaluation and limits

You need to test with sentences not seen in training. Even an untrained random encoder can catch distinctive words like "iyi"/"kötü", because the word embeddings are carried into the mean. Still, with a five-sentence training set some test sentences are misclassified; resolving negation like "iyi değil" requires training the encoder:

test = ["iyi film", "kötü oyuncular", "senaryo iyi", "film kötü"]
# if there are no out-of-vocabulary words, kodla() works directly

In real projects this step is done by fine-tuning on top of a pre-trained model (BERT).

Exercise Probing and prediction

Compute the vectors of the training sentences with the given encoder and cumle_vektoru and train a LogisticRegression(C=10, max_iter=1000). Print the training accuracy. Then encode the test sentences and print the predictions with .tolist() and the test accuracy.

Show example solution

import numpy as np

def softmax(z, eksen=-1):
    e = np.exp(z - z.max(axis=eksen, keepdims=True))
    return e / e.sum(axis=eksen, keepdims=True)

metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
    for k in c.split():
        sozluk.setdefault(k, len(sozluk))

def kodla(c, n=5):
    ids = [sozluk[k] for k in c.split()][:n]
    return ids + [0] * (n - len(ids))

X = np.array([kodla(c) for c in metinler])
pad = X == 0

def dikkat(Q, K, V, pad):
    d = Q.shape[-1]
    skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
    skor = np.where(pad[:, None, :], -1e9, skor)
    A = softmax(skor)
    return A @ V, A

def layer_norm(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)

rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5

def encoder(X, pad):
    H = E[X]
    Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
    return layer_norm(H + Y)

def cumle_vektoru(H, pad):
    m = (~pad)[:, :, None]
    return (H * m).sum(axis=1) / m.sum(axis=1)

from sklearn.linear_model import LogisticRegression
y = np.array([1, 0, 1, 0, 0])
test = ["iyi film", "kötü oyuncular", "senaryo iyi", "film kötü"]
y_test = np.array([1, 0, 1, 0])

C = cumle_vektoru(encoder(X, pad), pad)
model = LogisticRegression(C=10, max_iter=1000).fit(C, y)
print(model.score(C, y))
Xt = np.array([kodla(c) for c in test])
Ct = cumle_vektoru(encoder(Xt, Xt == 0), Xt == 0)
print(model.predict(Ct).tolist())
print(model.score(Ct, y_test))

Test yourself

Which parameters are trained in linear probing?
- Only those of the linear classifier on top
- The whole encoder
- Only the embedding table
- None
Show answer

Only those of the linear classifier on top — The encoder is frozen; only the top layer learns.

Train the probing model.

model = LogisticRegression(max_iter=1000).____(C, y)

Show answer

model = LogisticRegression(max_iter=1000).fit(C, y)

Match.

encoder

Frozen
LogisticRegression

Trained top layer
test sentences

Unseen data
negation

Requires training

Next lesson: What is BERT?

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies