Machine Learning
Embedding
The embedding layer converts word indices to dense vectors.
Embedding
Words aren't numbers; each word is given an index and nn.Embedding turns each index into a learned vector. This table is learned along with training; words with similar meanings get similar vectors:
gom = nn.Embedding(num_embeddings=1000, embedding_dim=16, padding_idx=0)
X = torch.tensor([[3, 1, 4, 0], [1, 5, 0, 0]]) # (batch, time) indices
print(gom(X).shape) # torch.Size([2, 4, 16])
print(gom.weight.shape) # torch.Size([1000, 16])
The vector of the index given as padding_idx stays fixed at zero and isn't updated.
Embedding + RNN classifier
A typical text classifier: indices → embedding → RNN → last hidden state → linear layer:
class MetinRNN(nn.Module):
def __init__(self, sozluk, gom, gizli, sinif):
super().__init__()
self.gom = nn.Embedding(sozluk, gom, padding_idx=0)
self.rnn = nn.RNN(gom, gizli, batch_first=True)
self.bas = nn.Linear(gizli, sinif)
def forward(self, x):
_, h_n = self.rnn(self.gom(x))
return self.bas(h_n[-1]) # (batch, classes)
h_n[-1] is the last layer's last hidden state.
Exercise Text classifier
Write the MetinRNN class from the card. For MetinRNN(1000, 16, 32, 2), print the output shape for a (2, 4) index input. Then print that the embedding vector of the padding index (0) is entirely zero.
Show example solution
import torch
from torch import nn
X = torch.tensor([[3, 1, 4, 0], [1, 5, 0, 0]])
class MetinRNN(nn.Module):
def __init__(self, sozluk, gom, gizli, sinif):
super().__init__()
self.gom = nn.Embedding(sozluk, gom, padding_idx=0)
self.rnn = nn.RNN(gom, gizli, batch_first=True)
self.bas = nn.Linear(gizli, sinif)
def forward(self, x):
_, h_n = self.rnn(self.gom(x))
return self.bas(h_n[-1])
m = MetinRNN(1000, 16, 32, 2)
print(m(X).shape)
print(bool((m.gom.weight[0] == 0).all()))
Test yourself
-
How many parameters does an embedding of a 5000-word vocabulary and 64 dimensions contain?
Show answer
320000 — An embedding is a lookup table: 5000 × 64.
-
Fix the vector of the padding index.
gom = nn.Embedding(1000, 16, ____=0)
Show answer
gom = nn.Embedding(1000, 16, padding_idx=0)
-
Match the parts.
- nn.Embedding
- Index to vector
- num_embeddings
- Vocabulary size
- embedding_dim
- Vector size
- h_n[-1]
- Last hidden state
Next lesson: Summary - RNN
