Machine Learning
Preparing data
Convert sequence data to tensors.
Project: a sentiment classifier
In this project you'll build a small transformer encoder step by step with NumPy and classify short Turkish sentences as positive/negative. The first step is turning the data into numbers:
- Build a vocabulary: give each word an id; 0 is reserved for padding (
<pad>). - Turn each sentence into a list of ids.
- Bring them all to the same length (pad or truncate).
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
setdefault adds the word with the next number if it's missing, and leaves it alone if present.
The padding mask
Padding positions are not real words; attention should not look at them. For this a boolean padding mask is kept:
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler]) # (5, 5)
pad = X == 0 # True = padding
In the next lesson this mask will zero the weights of padding columns by pulling their scores to a very small number.
Exercise Encode the sentences
Write the kodla(c, n=5) function with the given vocabulary (truncate and pad). Build the X matrix and the pad mask. Print the vocabulary size, X and the number of real words in each sentence ((~pad).sum(axis=1)) on separate lines.
Show example solution
import numpy as np
metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
for k in c.split():
sozluk.setdefault(k, len(sozluk))
def kodla(c, n=5):
ids = [sozluk[k] for k in c.split()][:n]
return ids + [0] * (n - len(ids))
X = np.array([kodla(c) for c in metinler])
pad = X == 0
print(len(sozluk))
print(X)
print((~pad).sum(axis=1))
Test yourself
-
Why are all sentences brought to the same length?
- To increase accuracy
- To shrink the vocabulary
- To turn off attention
- To process them as a single array (matrix) in a batch
Show answer
To process them as a single array (matrix) in a batch — NumPy arrays must be rectangular.
-
Fill the missing positions with 0.
return ids + [0] * (n - ____(ids))
Show answer
return ids + [0] * (n - len(ids))
-
Match.
- sozluk["<pad>"]
- 0
- X.shape
- (5, 5)
- pad
- Padding positions
- setdefault
- Adds if missing
Next lesson: Attention layer
