CodeBlanks Logo Icon CodeBlanks
Login Register

Preparing data

Convert sequence data to tensors.

Project: a sentiment classifier

In this project you'll build a small transformer encoder step by step with NumPy and classify short Turkish sentences as positive/negative. The first step is turning the data into numbers:

  1. Build a vocabulary: give each word an id; 0 is reserved for padding (<pad>).
  2. Turn each sentence into a list of ids.
  3. Bring them all to the same length (pad or truncate).
sozluk = {"<pad>": 0}
for c in metinler:
    for k in c.split():
        sozluk.setdefault(k, len(sozluk))

setdefault adds the word with the next number if it's missing, and leaves it alone if present.

The padding mask

Padding positions are not real words; attention should not look at them. For this a boolean padding mask is kept:

def kodla(c, n=5):
    ids = [sozluk[k] for k in c.split()][:n]
    return ids + [0] * (n - len(ids))

X = np.array([kodla(c) for c in metinler])   # (5, 5)
pad = X == 0                                   # True = padding

In the next lesson this mask will zero the weights of padding columns by pulling their scores to a very small number.

Practice Encode the sentences

Write the kodla(c, n=5) function with the given vocabulary (truncate and pad). Build the X matrix and the pad mask. Print the vocabulary size, X and the number of real words in each sentence ((~pad).sum(axis=1)) on separate lines.

Starter code

model.py
import numpy as np

metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
    for k in c.split():
        sozluk.setdefault(k, len(sozluk))

# buraya yaz

Sample solution

model.py
import numpy as np

metinler = ["film çok iyi", "çok kötü bir film", "iyi", "oyuncular iyi ama senaryo kötü", "kötü"]
sozluk = {"<pad>": 0}
for c in metinler:
    for k in c.split():
        sozluk.setdefault(k, len(sozluk))

def kodla(c, n=5):
    ids = [sozluk[k] for k in c.split()][:n]
    return ids + [0] * (n - len(ids))

X = np.array([kodla(c) for c in metinler])
pad = X == 0
print(len(sozluk))
print(X)
print((~pad).sum(axis=1))

Expected output

9
[[1 2 3 0 0]
 [2 4 5 1 0]
 [3 0 0 0 0]
 [6 3 7 8 4]
 [4 0 0 0 0]]
[3 4 1 5 1]

Questions and answers

Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.

1 Why are all sentences brought to the same length?

  • To increase accuracy
  • To shrink the vocabulary
  • To turn off attention
  • To process them as a single array (matrix) in a batch Correct answer

Explanation: NumPy arrays must be rectangular.

2 Fill the missing positions with 0.

Question 
return ids + [0] * (n - ____(ids))
Answer 
return ids + [0] * (n - len(ids))

Explanation: The number of padding items to add is n − length.

3 Match.

Term Meaning
sozluk["<pad>"] 0
X.shape (5, 5)
pad Padding positions
setdefault Adds if missing

All lessons in Expert

1 Attention

  1. What is attention?
  2. Self-attention
  3. Query, Key, Value
  4. Scaled dot-product
  5. Multi-head
  6. Summary - Attention

2 Transformer

  1. What is a transformer?
  2. Encoder
  3. Decoder
  4. Positional encoding
  5. TransformerEncoderLayer
  6. Summary - Transformer

3 Project: Transformer

  1. Preparing data
  2. Attention layer
  3. Encoder block
  4. Results

4 BERT

  1. What is BERT?
  2. Masked LM
  3. Tokenization
  4. Fine-tuning
  5. CLS token
  6. Summary - BERT

5 GPT

  1. What is GPT?
  2. Autoregressive
  3. Decoder-only
  4. Causal mask
  5. Text generation
  6. Summary - GPT

6 GAN and VAE

  1. What is GAN?
  2. Generator
  3. Discriminator
  4. What is VAE?
  5. Encoder-Decoder
  6. Summary - GAN and VAE

7 Reinforcement learning

  1. What is RL?
  2. Agent
  3. Policy
  4. Reward
  5. Q-learning
  6. Summary - RL

8 Expert final

  1. Challenge 1
  2. Challenge 2
  3. Challenge 3
  4. Challenge 4