Machine Learning
What is VAE?
VAE is a generative model that encodes data into a probabilistic distribution.
What is a VAE?
The variational autoencoder (VAE, 2013) is a probabilistic model that compresses data into a low-dimensional latent space and regenerates it from there. The encoder gives not a single point but a distribution for each sample: a mean μ and a log-variance log σ².
mu, logvar = encoder(x)
z = mu + np.exp(0.5 * logvar) * eps # eps ~ N(0, 1): reparameterization
x_hat = decoder(z)
After training, to generate a new sample it's enough to draw z ~ N(0, I) and give it to the decoder. Compared with a GAN its training is stable, but its samples are usually blurrier.
The KL divergence
The second term of the VAE loss keeps the encoder's distribution close to N(0, 1). For a Gaussian it has a closed form:
def kl(mu, logvar):
return -0.5 * np.sum(1 + logvar - mu ** 2 - np.exp(logvar))
print(kl(np.zeros(2), np.zeros(2))) # 0.0: exactly N(0, 1)
print(kl(np.array([1., 0.]), np.zeros(2))) # 0.5
Without the KL term, the encoder would put every sample at distant, narrow points and sampling from the latent space would be meaningless.
Exercise KL and reparameterization
Write the kl(mu, logvar) function. Print the KL value for three cases with round(..., 3) on one line: μ=[0,0], logvar=[0,0]; μ=[1,0], logvar=[0,0]; μ=[0,0], logvar=[log 0.25, 0]. Then draw 100 000 samples with reparameterization for μ=2, logvar=log 0.25; print the mean and the standard deviation with round(..., 2) on one line.
Show example solution
import numpy as np
rng = np.random.default_rng(0)
eps = rng.normal(size=100_000)
def kl(mu, logvar):
mu, logvar = np.asarray(mu, float), np.asarray(logvar, float)
return float(-0.5 * np.sum(1 + logvar - mu ** 2 - np.exp(logvar))) + 0.0
print(round(kl([0, 0], [0, 0]), 3), round(kl([1, 0], [0, 0]), 3), round(kl([0, 0], [np.log(0.25), 0]), 3))
z = 2 + np.exp(0.5 * np.log(0.25)) * eps
print(round(z.mean(), 2), round(z.std(), 2))
Test yourself
-
Why is the reparameterization trick needed?
- To freeze the decoder
- To let the gradient flow through the sampling step to μ and σ
- To save memory
- To increase the latent dimension
Show answer
To let the gradient flow through the sampling step to μ and σ — The randomness is moved into eps; z becomes a differentiable function of μ and σ.
-
Get the standard deviation from the log-variance.
z = mu + np.exp(____ * logvar) * eps
Show answer
z = mu + np.exp(0.5 * logvar) * eps
-
Match.
- μ
- Encoder mean
- log σ²
- Encoder log-variance
- eps
- N(0, 1) noise
- KL
- Closeness to N(0, 1)
Next lesson: Encoder-Decoder
