01 // THE ADVERSARIAL GAME
Goodfellow's 2014 GAN paper has a simple setup with complicated dynamics: train two networks against each other. The Generator G maps a random noise vector z ~ p(z) into data space. The DiscriminatorD receives either a real sample or G's output and estimates whether the input came from the data distribution.
The training objective is a two-player minimax game. D wants to maximize its classification accuracy; G wants to minimize it. The value function captures this tension:
Goodfellow showed that for a fixed G, the optimal discriminator is D*(x) = p_data(x) / (p_data(x) + p_g(x)). Substituting this back into V reveals that the minimax game implicitly minimizes the Jensen-Shannon divergence between the real data distribution and the generated distribution:
At the global optimum, p_g = p_data, the JSD is zero, and D outputs 0.5 everywhere. That is the clean theoretical picture. In practice, the optimizer has to maintain a useful training signal while both networks keep changing.
02 // WHY TRAINING IS HARD
The minimax game is not like training a normal classifier. Two objectives move at the same time, and the gradients depend on the relative strength of both models. If one network learns too fast, the other one stops receiving useful information.
The discriminator has to be calibrated enough to detect useful differences, but not so confident that G receives saturated gradients. If D is too weak, G learns from noisy feedback and sample quality stalls.
Mode collapse was the failure I saw most often. The generator would find one face or a tiny cluster of faces that fooled D, then produce that over and over. After D caught up, G would jump to another small cluster. The loss could look active while the samples were clearly bad.
"If every image in the batch looks the same, the loss curve is not telling the whole story."
03 // DCGAN ARCHITECTURE
Radford, Metz, and Chintala's DCGAN paper made my runs more stable. The useful lesson was that architecture choices were part of the optimization method, not just model capacity. These rules helped:
- Replace all pooling layers with strided convolutions (D) and fractional-strided convolutions (G) — let the network learn its own spatial downsampling and upsampling.
- Use batch normalization in both G and D (except G's output layer and D's input layer).
- Remove fully connected hidden layers — go all-convolutional.
- Use ReLU activation in G (except output, which uses Tanh) and LeakyReLU (slope 0.2) in D.
Batch normalization helped by keeping intermediate activations in a range where both networks could continue learning. Without it, my runs often collapsed after a few thousand iterations. With it, 64×64 face generators on CelebA became much easier to train and compare.
04 // TRAINING RECIPE
Architecture alone is not enough. GAN training is empirical, so I treated each run like an experiment: keep the dataset fixed, change one stabilization trick at a time, save random samples, and compare FID/IS instead of trusting the loss alone. These techniques made the biggest difference:
Use Adam with learning rate 0.0002 and β₁ = 0.5 (not the default 0.9). The reduced momentum prevents the optimizer from oscillating in the adversarial landscape. I've tried SGD with momentum, RMSProp, and vanilla Adam — β₁ = 0.5 Adam is the only one that consistently converges.
Instead of training D with hard labels (real=1, fake=0), use soft labels: real=0.9, fake=0.0 (or even 0.1). This prevents D from becoming overconfident, keeping its gradients informative for G. One-sided smoothing (only on the real label) works best.
real_labels = torch.full((batch_size,), 0.9) # not 1.0Train D for k steps, then G for 1 step. The original paper suggests k=1, but I've found k=1 works best for DCGAN. Some practitioners use k=5 with weight clipping (WGAN-style), but with the DCGAN architecture and Adam optimizer, one-to-one alternation is stable. The key is that D should always be slightly ahead of G — never too far ahead.
Instead of training G to maximize D's output directly, train G to match the statistics of intermediate features in D. Compute the L2 distance between the mean feature vectors of real and generated batches at some intermediate layer of D:
L_FM = ∥ 𝔼_x[f(x)] − 𝔼_z[f(G(z))] ∥² where f = D's intermediate featuresGive D the ability to compare samples within a batch. For each sample, compute a feature vector and measure its L1 distance to all other samples in the batch. Append this "closeness" score to D's features. This directly penalizes mode collapse: if all generated samples are identical, their closeness scores are maximal, and D can trivially reject the batch.
05 // DIAGNOSING FAILURE MODES
The hardest part of GAN training is knowing when something is going wrong. Unlike supervised learning, there's no single loss curve that monotonically decreases. Both D and G losses oscillate, and the relationship between loss values and sample quality is non-obvious. Here's what I've learned to watch for.
Loss curves: Healthy training shows D's loss hovering around ln(2) ≈ 0.693(random guessing) and G's loss oscillating in a bounded range. If D's loss drops to near zero and stays there, D has won — G is stuck. If both losses oscillate wildly with increasing amplitude, training is diverging.
Inception Score (IS) provides a quantitative measure of sample quality and diversity. It uses a pre-trained Inception network to classify generated images:
High IS means two things: the conditional distribution p(y|x) is peaked (each generated image clearly belongs to a class), and the marginal p(y) is spread across classes (the generator produces diverse outputs). IS on real CIFAR-10 is ~11.2; a well-trained GAN achieves 7–8.
Fréchet Inception Distance (FID) is a more robust metric that compares the statistics of generated samples to real samples in Inception feature space:
FID models both feature spaces as multivariate Gaussians and measures the Fréchet distance between them. Lower is better. FID captures both quality (mean alignment) and diversity (covariance alignment), making it more sensitive to mode collapse than IS. I log FID every 1,000 iterations and save checkpoints when it hits a new minimum.
"Never trust the loss curve alone. Always look at the samples. Always compute FID. Your eyes are the ultimate discriminator."
06 // WASSERSTEIN DISTANCE
Wasserstein GAN (WGAN) from Arjovsky et al. is important because it gives a better-behaved training signal when real and generated distributions have little overlap. Early in training, G often produces noise, so the Jensen-Shannon signal can saturate at log(2) and stop telling G which direction to move.
The Wasserstein-1 distance (Earth Mover's distance) doesn't have this problem. Intuitively, it measures the minimum cost of transporting mass from one distribution to another. Its dual formulation gives us a tractable objective:
The supremum is over all 1-Lipschitz functions f. In practice, D (now called the "critic") approximates f. The Lipschitz constraint is enforced either by weight clipping (original WGAN — clip all weights to [-0.01, 0.01]) or by a gradient penalty (WGAN-GP):
The gradient penalty samples random interpolations between real and generated points and penalizes the critic's gradient norm for deviating from 1. This is far more stable than weight clipping, which tends to push weights to the clipping boundary and causes capacity underuse.
In my experiments, WGAN-GP made the critic loss more useful. It still did not replace looking at samples or computing FID, but the loss curve became less misleading than the vanilla GAN objective.
07 // RESULTS & GENERATED SAMPLES
After several hyperparameter searches, my best DCGAN model on CelebA (64×64, ~200k images) produced recognizable faces with reasonable global structure. The failure cases were still obvious: melted ear regions, asymmetric glasses, and bad teeth.
On CIFAR-10 (32×32), the results were weaker but useful for diagnostics. The generator learned rough class-like modes: cars, planes, and animals were visible, but fine structure was poor. FID on my best CIFAR-10 run was around 37.
Cherry-picking vs. random samples: This is the elephant in the room of GAN research. Published papers show carefully selected best-case outputs. Random batches tell a different story — maybe 60–70% of generated faces look plausible, 20% have minor artifacts, and 10% are clearly broken. Always demand random samples when evaluating a GAN.
Latent space interpolation is a useful sanity check. Take two noise vectors z₁ and z₂, interpolate linearly between them with z_t = (1−t)·z₁ + t·z₂, and generate images at each step. If training is healthy, the images change smoothly instead of jumping between unrelated samples.
Latent space arithmetic also works: z(man with glasses) − z(man) + z(woman) ≈ z(woman with glasses). This vector arithmetic on noise vectors produces semantically meaningful transformations in image space — visual evidence that the latent space has captured disentangled factors of variation.
08 // LEGACY
We are only two years into the GAN era, and the research direction is becoming clearer. DCGAN improved stability. WGAN improved the training signal. Conditional GANs made generation controllable. The open problems are still diversity, resolution, and reliable evaluation.
A few directions look promising right now. Progressive training grows G and D from low resolution to high resolution. Pix2Pix applies the adversarial loss to paired image translation. CycleGAN tries the same idea without paired examples by adding a cycle-consistency loss.
Even if future models replace adversarial training, these lessons are useful: training dynamics matter as much as architecture, sample quality needs both visual inspection and metrics, and distribution matching is a more useful framing than per-pixel reconstruction.
"The training loop is short. The science is in the diagnostics: samples, metrics, ablations, and knowing when the loss is lying."