Machine Learning
Encoder-Decoder
VAE uses an encoder for distribution parameters and a decoder for data generation.
Encoder-decoder: the autoencoder
The basis of the VAE is the autoencoder: the encoder compresses x into a small code z, the decoder rebuilds x from z; the loss is the reconstruction error. The bottleneck (z size) forces the model to learn the data's most important structure:
enc = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 2))
dec = nn.Sequential(nn.Linear(2, 128), nn.ReLU(), nn.Linear(128, 784))
kayip = ((dec(enc(x)) - x) ** 2).mean()
The optimal solution of an autoencoder with linear layers and MSE loss finds the same subspace as PCA; non-linear layers capture more complex structure.
The VAE loss
The VAE loss is the sum of two terms: the reconstruction error and the KL. The balance between them matters; β-VAE weights the KL with a coefficient β:
kayip = yeniden_kurma + beta * kl
# large beta: smooth, disentangled latent space but blurry output
# small beta: sharp output but an irregular latent space
In this lesson you'll find the optimal solution of a linear autoencoder with SVD and measure the effect of the bottleneck.
Exercise A linear autoencoder
Center the given X (5-dimensional, really a 2-dimensional structure + small noise) and take the SVD. Have a yeniden_kur(k) function encode with the first k components and rebuild (Z = Xc @ V[:k].T, Xc_hat = Z @ V[:k]). Print the mean squared error for k = 1, 2, 3 with round(..., 4) on one line. Then print the shape of the k = 2 code.
Show example solution
import numpy as np
rng = np.random.default_rng(0)
gizli = rng.normal(size=(500, 2))
X = gizli @ rng.normal(size=(2, 5)) + 0.05 * rng.normal(size=(500, 5))
Xc = X - X.mean(axis=0)
_, _, V = np.linalg.svd(Xc, full_matrices=False)
def yeniden_kur(k):
Z = Xc @ V[:k].T
return Z, Z @ V[:k]
print(*[round(((yeniden_kur(k)[1] - Xc) ** 2).mean(), 4) for k in [1, 2, 3]])
print(yeniden_kur(2)[0].shape)
Test yourself
-
If the bottleneck size is increased, what usually happens to the reconstruction error?
- It decreases
- It increases
- It doesn't change
- It becomes undefined
Show answer
It decreases — A wider code carries more information.
-
Compute the reconstruction error.
kayip = ((dec(enc(x)) - x) ** ____).mean()
Show answer
kayip = ((dec(enc(x)) - x) ** 2).mean()
-
Match.
- encoder
- x → z
- decoder
- z → x̂
- bottleneck
- Small code size
- β
- KL weight
Next lesson: Summary - GAN and VAE
