Challenge 3

Define a GAN generator.

Challenge: evaluating generative models

Evaluating generative models is hard: is it "realistic", is it "diverse"? In this challenge you'll implement two measures:

Mode coverage: from how many of the clusters (modes) in the real data was at least one sample generated?
Latent interpolation: when walking between two points in an autoencoder's latent space, do the outputs change smoothly?

# mode collapse: high quality, low coverage
# blurry VAE: high coverage, low sharpness

For images, measures like FID and Precision/Recall take their place; the logic is the same.

Exercise Mode coverage

8 mode centres lie on a circle. Write the kapsama(noktalar, merkezler, esik=0.5) function: assign each sample to the nearest centre and count those closer than the threshold as "high quality". Return the number of covered modes (at least one high-quality sample) and the share of high-quality samples. Print the results for the given three generators (good, collapsed, blurry) with round(..., 2) on one line each.

Show example solution

import numpy as np

rng = np.random.default_rng(0)
aci = np.arange(8) * 2 * np.pi / 8
merkezler = np.c_[np.cos(aci), np.sin(aci)] * 3
iyi = merkezler[rng.integers(8, size=800)] + rng.normal(0, 0.15, (800, 2))
cokmus = merkezler[rng.integers(2, size=800)] + rng.normal(0, 0.15, (800, 2))
bulanik = merkezler[rng.integers(8, size=800)] + rng.normal(0, 0.8, (800, 2))

def kapsama(noktalar, merkezler, esik=0.5):
    uzak = ((noktalar[:, None] - merkezler[None]) ** 2).sum(-1)
    atama = uzak.argmin(axis=1)
    kaliteli = np.sqrt(uzak.min(axis=1)) < esik
    return len(set(atama[kaliteli])), kaliteli.mean()

for uretici in [iyi, cokmus, bulanik]:
    k, q = kapsama(uretici, merkezler)
    print(k, round(q, 2))

Exercise Latent interpolation

Build a 2-dimensional linear autoencoder on the given data with SVD (kodla: (x - ort) @ V[:2].T, coz: z @ V[:2] + ort). Interpolate in 5 steps (np.linspace(0, 1, 5)) between the codes of the first two samples and decode. Print the shape of the decoded points. Then print on one line with np.allclose that the middle output equals the decoding of the mean of the endpoints and that the distances between consecutive steps are equal.

Show example solution

import numpy as np

rng = np.random.default_rng(0)
X = rng.normal(size=(300, 2)) @ rng.normal(size=(2, 6)) + 0.05 * rng.normal(size=(300, 6))
ort = X.mean(axis=0)
_, _, V = np.linalg.svd(X - ort, full_matrices=False)

kodla = lambda x: (x - ort) @ V[:2].T
coz = lambda z: z @ V[:2] + ort

z0, z1 = kodla(X[0]), kodla(X[1])
t = np.linspace(0, 1, 5)[:, None]
yol = coz((1 - t) * z0 + t * z1)
print(yol.shape)
adimlar = np.linalg.norm(np.diff(yol, axis=0), axis=1)
print(np.allclose(yol[2], coz((z0 + z1) / 2)), np.allclose(adimlar, adimlar[0]))

Test yourself

On which measure does a GAN with mode collapse score low?
- Training speed
- Parameter count
- Mode coverage (diversity)
- Sample sharpness
Show answer

Mode coverage (diversity) — A model producing a single kind of sample misses most of the modes.

Assign each sample to the nearest mode centre.

atama = ((noktalar[:, None] - merkezler[None]) ** 2).sum(-1).____(axis=1)

Show answer

atama = ((noktalar[:, None] - merkezler[None]) ** 2).sum(-1).argmin(axis=1)

Match.

coverage

Diversity
quality

Realism
interpolation

Latent-space smoothness
FID

Image measure

Next lesson: Challenge 4

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies