Encoder-Decoder
- Machine Learning course
- Expert
- Lesson 33/44
- 5 min
VAE uses an encoder for distribution parameters and a decoder for data generation.
Encoder-decoder: the autoencoder
The basis of the VAE is the autoencoder: the encoder compresses x into a small code z, the decoder rebuilds x from z; the loss is the reconstruction error. The bottleneck (z size) forces the model to learn the data's most important structure:
enc = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 2))
dec = nn.Sequential(nn.Linear(2, 128), nn.ReLU(), nn.Linear(128, 784))
kayip = ((dec(enc(x)) - x) ** 2).mean()The optimal solution of an autoencoder with linear layers and MSE loss finds the same subspace as PCA; non-linear layers capture more complex structure.
The VAE loss
The VAE loss is the sum of two terms: the reconstruction error and the KL. The balance between them matters; β-VAE weights the KL with a coefficient β:
kayip = yeniden_kurma + beta * kl
# large beta: smooth, disentangled latent space but blurry output
# small beta: sharp output but an irregular latent spaceIn this lesson you'll find the optimal solution of a linear autoencoder with SVD and measure the effect of the bottleneck.
Practice A linear autoencoder
Center the given X (5-dimensional, really a 2-dimensional structure + small noise) and take the SVD. Have a yeniden_kur(k) function encode with the first k components and rebuild (Z = Xc @ V[:k].T, Xc_hat = Z @ V[:k]). Print the mean squared error for k = 1, 2, 3 with round(..., 4) on one line. Then print the shape of the k = 2 code.
Starter code
import numpy as np
rng = np.random.default_rng(0)
gizli = rng.normal(size=(500, 2))
X = gizli @ rng.normal(size=(2, 5)) + 0.05 * rng.normal(size=(500, 5))
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(0)
gizli = rng.normal(size=(500, 2))
X = gizli @ rng.normal(size=(2, 5)) + 0.05 * rng.normal(size=(500, 5))
Xc = X - X.mean(axis=0)
_, _, V = np.linalg.svd(Xc, full_matrices=False)
def yeniden_kur(k):
Z = Xc @ V[:k].T
return Z, Z @ V[:k]
print(*[round(((yeniden_kur(k)[1] - Xc) ** 2).mean(), 4) for k in [1, 2, 3]])
print(yeniden_kur(2)[0].shape)Expected output
0.2147 0.0015 0.001 (500, 2)
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 If the bottleneck size is increased, what usually happens to the reconstruction error?
- It decreases Correct answer
- It increases
- It doesn't change
- It becomes undefined
Explanation: A wider code carries more information.
2 Compute the reconstruction error.
kayip = ((dec(enc(x)) - x) ** ____).mean()kayip = ((dec(enc(x)) - x) ** 2).mean()Explanation: Mean squared error.
3 Match.
| Term | Meaning |
|---|---|
encoder |
x → z |
decoder |
z → x̂ |
bottleneck |
Small code size |
β |
KL weight |