Results
- Machine Learning course
- Expert
- Lesson 16/44
- 7 min
Print the model output.
Result: training the classifier
We don't train the encoder's weights in this project (that will be done with fine-tuning in the next BERT group). Instead we put a logistic regression on top of the sentence vectors; this is called linear probing:
from sklearn.linear_model import LogisticRegression
C = cumle_vektoru(encoder(X, pad), pad)
model = LogisticRegression(max_iter=1000).fit(C, y)
print(model.predict(cumle_vektoru(encoder(X_test, pad_test), pad_test)))Probing is the standard way to measure whether a representation "carries" a piece of information.
Evaluation and limits
You need to test with sentences not seen in training. Even an untrained random encoder can catch distinctive words like "iyi"/"kötü", because the word embeddings are carried into the mean. Still, with a five-sentence training set some test sentences are misclassified; resolving negation like "iyi değil" requires training the encoder:
test = ["iyi film", "kötü oyuncular", "senaryo iyi", "film kötü"]
# if there are no out-of-vocabulary words, kodla() works directlyIn real projects this step is done by fine-tuning on top of a pre-trained model (BERT).
Practice Probing and prediction
Compute the vectors of the training sentences with the given encoder and cumle_vektoru and train a LogisticRegression(C=10, max_iter=1000). Print the training accuracy. Then encode the test sentences and print the predictions with .tolist() and the test accuracy.
Starter code
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
def dikkat(Q, K, V, pad):
d = Q.shape[-1]
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, A
def layer_norm(x, eps=1e-5):
return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)
rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
def encoder(X, pad):
H = E[X]
Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
return layer_norm(H + Y)
def cumle_vektoru(H, pad):
m = (~pad)[:, :, None]
return (H * m).sum(axis=1) / m.sum(axis=1)
from sklearn.linear_model import LogisticRegression
y = np.array([1, 0, 1, 0, 0])
test = ["iyi film", "kötü oyuncular", "senaryo iyi", "film kötü"]
y_test = np.array([1, 0, 1, 0])
# buraya yazSample solution
import numpy as np
def softmax(z, eksen=-1):
e = np.exp(z - z.max(axis=eksen, keepdims=True))
return e / e.sum(axis=eksen, keepdims=True)
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
def dikkat(Q, K, V, pad):
d = Q.shape[-1]
skor = Q @ K.transpose(0, 2, 1) / np.sqrt(d)
skor = np.where(pad[:, None, :], -1e9, skor)
A = softmax(skor)
return A @ V, A
def layer_norm(x, eps=1e-5):
return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)
rng = np.random.default_rng(0)
d = 8
E = rng.normal(size=(len(sozluk), d)) * 0.5
Wq, Wk, Wv = rng.normal(size=(3, d, d)) * 0.5
def encoder(X, pad):
H = E[X]
Y, A = dikkat(H @ Wq, H @ Wk, H @ Wv, pad)
return layer_norm(H + Y)
def cumle_vektoru(H, pad):
m = (~pad)[:, :, None]
return (H * m).sum(axis=1) / m.sum(axis=1)
from sklearn.linear_model import LogisticRegression
y = np.array([1, 0, 1, 0, 0])
test = ["iyi film", "kötü oyuncular", "senaryo iyi", "film kötü"]
y_test = np.array([1, 0, 1, 0])
C = cumle_vektoru(encoder(X, pad), pad)
model = LogisticRegression(C=10, max_iter=1000).fit(C, y)
print(model.score(C, y))
Xt = np.array([kodla(c) for c in test])
Ct = cumle_vektoru(encoder(Xt, Xt == 0), Xt == 0)
print(model.predict(Ct).tolist())
print(model.score(Ct, y_test))Expected output
1.0 [1, 0, 0, 0] 0.75
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Which parameters are trained in linear probing?
- Only those of the linear classifier on top Correct answer
- The whole encoder
- Only the embedding table
- None
Explanation: The encoder is frozen; only the top layer learns.
2 Train the probing model.
model = LogisticRegression(max_iter=1000).____(C, y)model = LogisticRegression(max_iter=1000).fit(C, y)Explanation: fit trains; predict predicts.
3 Match.
| Term | Meaning |
|---|---|
encoder |
Frozen |
LogisticRegression |
Trained top layer |
test sentences |
Unseen data |
negation |
Requires training |