What is VAE?
- Machine Learning course
- Expert
- Lesson 32/44
- 5 min
VAE is a generative model that encodes data into a probabilistic distribution.
What is a VAE?
The variational autoencoder (VAE, 2013) is a probabilistic model that compresses data into a low-dimensional latent space and regenerates it from there. The encoder gives not a single point but a distribution for each sample: a mean μ and a log-variance log σ².
mu, logvar = encoder(x)
z = mu + np.exp(0.5 * logvar) * eps # eps ~ N(0, 1): reparameterization
x_hat = decoder(z)After training, to generate a new sample it's enough to draw z ~ N(0, I) and give it to the decoder. Compared with a GAN its training is stable, but its samples are usually blurrier.
The KL divergence
The second term of the VAE loss keeps the encoder's distribution close to N(0, 1). For a Gaussian it has a closed form:
def kl(mu, logvar):
return -0.5 * np.sum(1 + logvar - mu ** 2 - np.exp(logvar))
print(kl(np.zeros(2), np.zeros(2))) # 0.0: exactly N(0, 1)
print(kl(np.array([1., 0.]), np.zeros(2))) # 0.5Without the KL term, the encoder would put every sample at distant, narrow points and sampling from the latent space would be meaningless.
Practice KL and reparameterization
Write the kl(mu, logvar) function. Print the KL value for three cases with round(..., 3) on one line: μ=[0,0], logvar=[0,0]; μ=[1,0], logvar=[0,0]; μ=[0,0], logvar=[log 0.25, 0]. Then draw 100 000 samples with reparameterization for μ=2, logvar=log 0.25; print the mean and the standard deviation with round(..., 2) on one line.
Starter code
import numpy as np
rng = np.random.default_rng(0)
eps = rng.normal(size=100_000)
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(0)
eps = rng.normal(size=100_000)
def kl(mu, logvar):
mu, logvar = np.asarray(mu, float), np.asarray(logvar, float)
return float(-0.5 * np.sum(1 + logvar - mu ** 2 - np.exp(logvar))) + 0.0
print(round(kl([0, 0], [0, 0]), 3), round(kl([1, 0], [0, 0]), 3), round(kl([0, 0], [np.log(0.25), 0]), 3))
z = 2 + np.exp(0.5 * np.log(0.25)) * eps
print(round(z.mean(), 2), round(z.std(), 2))Expected output
0.0 0.5 0.318 2.0 0.5
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 Why is the reparameterization trick needed?
- To freeze the decoder
- To let the gradient flow through the sampling step to μ and σ Correct answer
- To save memory
- To increase the latent dimension
Explanation: The randomness is moved into eps; z becomes a differentiable function of μ and σ.
2 Get the standard deviation from the log-variance.
z = mu + np.exp(____ * logvar) * epsz = mu + np.exp(0.5 * logvar) * epsExplanation: σ = exp(½ log σ²).
3 Match.
| Term | Meaning |
|---|---|
μ |
Encoder mean |
log σ² |
Encoder log-variance |
eps |
N(0, 1) noise |
KL |
Closeness to N(0, 1) |