Embedding
- Machine Learning course
- Advanced
- Lesson 51/78
- 5 min
The embedding layer converts word indices to dense vectors.
Embedding
Words aren't numbers; each word is given an index and nn.Embedding turns each index into a learned vector. This table is learned along with training; words with similar meanings get similar vectors:
gom = nn.Embedding(num_embeddings=1000, embedding_dim=16, padding_idx=0)
X = torch.tensor([[3, 1, 4, 0], [1, 5, 0, 0]]) # (batch, time) indices
print(gom(X).shape) # torch.Size([2, 4, 16])
print(gom.weight.shape) # torch.Size([1000, 16])The vector of the index given as padding_idx stays fixed at zero and isn't updated.
Embedding + RNN classifier
A typical text classifier: indices → embedding → RNN → last hidden state → linear layer:
class MetinRNN(nn.Module):
def __init__(self, sozluk, gom, gizli, sinif):
super().__init__()
self.gom = nn.Embedding(sozluk, gom, padding_idx=0)
self.rnn = nn.RNN(gom, gizli, batch_first=True)
self.bas = nn.Linear(gizli, sinif)
def forward(self, x):
_, h_n = self.rnn(self.gom(x))
return self.bas(h_n[-1]) # (batch, classes)h_n[-1] is the last layer's last hidden state.
Practice Text classifier
Write the MetinRNN class from the card. For MetinRNN(1000, 16, 32, 2), print the output shape for a (2, 4) index input. Then print that the embedding vector of the padding index (0) is entirely zero.
Starter code
import torch
from torch import nn
X = torch.tensor([[3, 1, 4, 0], [1, 5, 0, 0]])
# buraya yazSample solution
import torch
from torch import nn
X = torch.tensor([[3, 1, 4, 0], [1, 5, 0, 0]])
class MetinRNN(nn.Module):
def __init__(self, sozluk, gom, gizli, sinif):
super().__init__()
self.gom = nn.Embedding(sozluk, gom, padding_idx=0)
self.rnn = nn.RNN(gom, gizli, batch_first=True)
self.bas = nn.Linear(gizli, sinif)
def forward(self, x):
_, h_n = self.rnn(self.gom(x))
return self.bas(h_n[-1])
m = MetinRNN(1000, 16, 32, 2)
print(m(X).shape)
print(bool((m.gom.weight[0] == 0).all()))Expected output
torch.Size([2, 2]) True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 How many parameters does an embedding of a 5000-word vocabulary and 64 dimensions contain?
- 5000
- 320000 Correct answer
- 5064
- 64
Explanation: An embedding is a lookup table: 5000 × 64.
2 Fix the vector of the padding index.
gom = nn.Embedding(1000, 16, ____=0)gom = nn.Embedding(1000, 16, padding_idx=0)Explanation: The padding_idx vector stays zero and gets no gradient.
3 Match the parts.
| Term | Meaning |
|---|---|
nn.Embedding |
Index to vector |
num_embeddings |
Vocabulary size |
embedding_dim |
Vector size |
h_n[-1] |
Last hidden state |