Challenge 3
- Machine Learning course
- Expert
- Lesson 43/44
- 8 min
Define a GAN generator.
Challenge: evaluating generative models
Evaluating generative models is hard: is it "realistic", is it "diverse"? In this challenge you'll implement two measures:
- Mode coverage: from how many of the clusters (modes) in the real data was at least one sample generated?
- Latent interpolation: when walking between two points in an autoencoder's latent space, do the outputs change smoothly?
# mode collapse: high quality, low coverage
# blurry VAE: high coverage, low sharpnessFor images, measures like FID and Precision/Recall take their place; the logic is the same.
Practice Mode coverage
8 mode centres lie on a circle. Write the kapsama(noktalar, merkezler, esik=0.5) function: assign each sample to the nearest centre and count those closer than the threshold as "high quality". Return the number of covered modes (at least one high-quality sample) and the share of high-quality samples. Print the results for the given three generators (good, collapsed, blurry) with round(..., 2) on one line each.
Starter code
import numpy as np
rng = np.random.default_rng(0)
aci = np.arange(8) * 2 * np.pi / 8
merkezler = np.c_[np.cos(aci), np.sin(aci)] * 3
iyi = merkezler[rng.integers(8, size=800)] + rng.normal(0, 0.15, (800, 2))
cokmus = merkezler[rng.integers(2, size=800)] + rng.normal(0, 0.15, (800, 2))
bulanik = merkezler[rng.integers(8, size=800)] + rng.normal(0, 0.8, (800, 2))
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(0)
aci = np.arange(8) * 2 * np.pi / 8
merkezler = np.c_[np.cos(aci), np.sin(aci)] * 3
iyi = merkezler[rng.integers(8, size=800)] + rng.normal(0, 0.15, (800, 2))
cokmus = merkezler[rng.integers(2, size=800)] + rng.normal(0, 0.15, (800, 2))
bulanik = merkezler[rng.integers(8, size=800)] + rng.normal(0, 0.8, (800, 2))
def kapsama(noktalar, merkezler, esik=0.5):
uzak = ((noktalar[:, None] - merkezler[None]) ** 2).sum(-1)
atama = uzak.argmin(axis=1)
kaliteli = np.sqrt(uzak.min(axis=1)) < esik
return len(set(atama[kaliteli])), kaliteli.mean()
for uretici in [iyi, cokmus, bulanik]:
k, q = kapsama(uretici, merkezler)
print(k, round(q, 2))Expected output
8 1.0 2 1.0 8 0.18
Practice Latent interpolation
Build a 2-dimensional linear autoencoder on the given data with SVD (kodla: (x - ort) @ V[:2].T, coz: z @ V[:2] + ort). Interpolate in 5 steps (np.linspace(0, 1, 5)) between the codes of the first two samples and decode. Print the shape of the decoded points. Then print on one line with np.allclose that the middle output equals the decoding of the mean of the endpoints and that the distances between consecutive steps are equal.
Starter code
import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(size=(300, 2)) @ rng.normal(size=(2, 6)) + 0.05 * rng.normal(size=(300, 6))
ort = X.mean(axis=0)
_, _, V = np.linalg.svd(X - ort, full_matrices=False)
# buraya yazSample solution
import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(size=(300, 2)) @ rng.normal(size=(2, 6)) + 0.05 * rng.normal(size=(300, 6))
ort = X.mean(axis=0)
_, _, V = np.linalg.svd(X - ort, full_matrices=False)
kodla = lambda x: (x - ort) @ V[:2].T
coz = lambda z: z @ V[:2] + ort
z0, z1 = kodla(X[0]), kodla(X[1])
t = np.linspace(0, 1, 5)[:, None]
yol = coz((1 - t) * z0 + t * z1)
print(yol.shape)
adimlar = np.linalg.norm(np.diff(yol, axis=0), axis=1)
print(np.allclose(yol[2], coz((z0 + z1) / 2)), np.allclose(adimlar, adimlar[0]))Expected output
(5, 6) True True
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 On which measure does a GAN with mode collapse score low?
- Training speed
- Parameter count
- Mode coverage (diversity) Correct answer
- Sample sharpness
Explanation: A model producing a single kind of sample misses most of the modes.
2 Assign each sample to the nearest mode centre.
atama = ((noktalar[:, None] - merkezler[None]) ** 2).sum(-1).____(axis=1)atama = ((noktalar[:, None] - merkezler[None]) ** 2).sum(-1).argmin(axis=1)Explanation: The index of the smallest distance.
3 Match.
| Term | Meaning |
|---|---|
coverage |
Diversity |
quality |
Realism |
interpolation |
Latent-space smoothness |
FID |
Image measure |