Full search page →

GAN — a distribution game and its idealized guarantee

Optional · paper guide 2 of 2 for ML 07 · Generative Models: Autoencoders, VAEs, GANs and Flows. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Question and thesis

Can a generator learn by competing with a classifier instead of specifying a tractable observation likelihood? Goodfellow et al. introduce a game in which a discriminator separates real from generated samples and a generator tries to defeat it [11]. Read binary cross-entropy and probability densities first. The paper combines an idealized distribution argument with finite neural-network experiments; these must be evaluated separately.

Method and figure

The schematic follows the data density, generator density and discriminator across alternating updates. It illustrates the distribution-level argument, not a measured training trajectory.
Source: Goodfellow et al., NeurIPS 2014 proceedings. Figure 1, PDF p. 4. The schematic follows the data density, generator density and discriminator across alternating updates. It illustrates the distribution-level argument, not a measured training trajectory. [11]

The original objective is

min⁡Gmax⁡DV(D,G)=Ex∼pdatalog⁡D(x)+Ez∼pzlog⁡(1−D(G(z))).\min_G\max_D V(D,G)=\mathbb E_{x\sim p_{\mathrm{data}}}\log D(x)+\mathbb E_{z\sim p_z}\log(1-D(G(z))).

Here xx is a real observation drawn from data distribution pdatap_{\mathrm{data}}, zz is noise from prior pzp_z, GG maps noise to a synthetic observation, and DD predicts the probability that an input is real. VV is the adversarial objective. Alternating updates train the two networks. The paper also recommends maximizing log⁡D(G(z))\log D(G(z)) for the generator early in training to avoid weak gradients from the saturating form.

Figure 1 follows schematic densities and the decision function. With an optimal discriminator, D∗(x)=pdata(x)/(pdata(x)+pg(x))D^*(x)=p_{\mathrm{data}}(x)/(p_{\mathrm{data}}(x)+p_g(x)), where pgp_g is the generator’s induced density. Under the ideal assumptions, matching distributions is the global solution. The figure is not evidence that every practical optimizer reaches it.

Evidence

The experiments show generated samples and quantitative comparisons using a Parzen-window density estimate fitted to generated samples. Table 1 therefore reports a surrogate evaluation, not the exact likelihood of the implicit generator. Kernel bandwidth and estimation behavior affect that comparison. Figure 2 includes nearest training examples as a useful visual check, but a few nearest neighbors cannot establish that an entire model never memorizes.

Design analysis

The discriminator learns a task-specific signal instead of using a fixed reconstruction distance. That is useful, but also means the generator’s objective changes as its opponent changes. The theoretical optimum assumes sufficient distributional flexibility and appropriate discriminator optimization. It does not prove convergence of finite networks trained by alternating minibatch updates. Keep the optimum’s existence separate from an algorithm’s ability to find it.

Reading checkpoint

If real and generated densities agree everywhere they have support, compute the optimal discriminator output. Explain why an observed discriminator accuracy near chance alone is insufficient to demonstrate this ideal state: a weak or poorly trained discriminator can also fail to distinguish inputs.

Limits and next step

Training imbalance and missing modes can coexist with visually plausible samples. Inspect coverage as well as sample quality, and avoid comparing surrogate density scores as if they were exact log likelihoods. Next read DDPM in ML 08: contrast a learned adversarial signal with an explicit noise-prediction objective and a multi-step sampling procedure.

Continue reading: DDPM — noise prediction and the quality–likelihood tradeoff

References

  1. C. M. Bishop and H. Bishop, Deep Learning: Foundations and Concepts, Springer, 2024. Book
  2. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book and notebooks
  3. S. Bond-Taylor, A. Leach, Y. Long and C. G. Willcocks, “Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. Paper
  4. G. Papamakarios et al., “Normalizing Flows for Probabilistic Modeling and Inference,” JMLR, 2021. Review
  5. L. Theis, A. van den Oord and M. Bethge, “A note on the evaluation of generative models,” ICLR, 2016. Paper
  6. D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” ICLR, 2014. Paper
  7. S. R. Bowman et al., “Generating Sentences from a Continuous Space,” CoNLL, 2016. Paper
  8. I. Higgins et al., “β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework,” ICLR, 2017. Paper
  9. D. P. Kingma and M. Welling, “An Introduction to Variational Autoencoders,” Foundations and Trends in Machine Learning, 2019. Review
  10. A. Vahdat and J. Kautz, “NVAE: A Deep Hierarchical Variational Autoencoder,” NeurIPS, 2020. Paper
  11. I. Goodfellow et al., “Generative Adversarial Nets,” NeurIPS, 2014. Paper
  12. T. Salimans et al., “Improved Techniques for Training GANs,” NeurIPS, 2016. Paper
  13. M. Arjovsky, S. Chintala and L. Bottou, “Wasserstein GAN,” ICML, 2017. Paper
  14. I. Gulrajani et al., “Improved Training of Wasserstein GANs,” NeurIPS, 2017. Paper
  15. T. Miyato et al., “Spectral Normalization for Generative Adversarial Networks,” ICLR, 2018. Paper
  16. L. Mescheder, A. Geiger and S. Nowozin, “Which Training Methods for GANs do actually Converge?” ICML, 2018. Paper
  17. T. Karras, S. Laine and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. Paper
  18. L. Dinh, J. Sohl-Dickstein and S. Bengio, “Density estimation using Real NVP,” ICLR, 2017. Paper
  19. D. P. Kingma and P. Dhariwal, “Glow: Generative Flow with Invertible 1x1 Convolutions,” NeurIPS, 2018. Paper
  20. D. J. Rezende and S. Mohamed, “Variational Inference with Normalizing Flows,” ICML, 2015. Paper
  21. R. T. Q. Chen, Y. Rubanova, J. Bettencourt and D. Duvenaud, “Neural Ordinary Differential Equations,” NeurIPS, 2018. Paper
  22. Y. Lipman et al., “Flow Matching for Generative Modeling,” ICLR, 2023. Paper
  23. I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, ch. 18–20. Book
  24. Y. Song and D. P. Kingma, “How to Train Your Energy-Based Models,” arXiv, 2021. Paper
  25. A. van den Oord, N. Kalchbrenner and K. Kavukcuoglu, “Pixel Recurrent Neural Networks,” ICML, 2016. Paper
  26. J. Ho, A. Jain and P. Abbeel, “Denoising Diffusion Probabilistic Models,” NeurIPS, 2020. Paper
  27. L. Yang et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. Survey
  28. E. Nalisnick et al., “Do Deep Generative Models Know What They Don’t Know?” ICLR, 2019. Paper
  29. M. Heusel et al., “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. Paper
  30. M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet and S. Gelly, “Assessing Generative Models via Precision and Recall,” NeurIPS, 2018. Paper
  31. T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen and T. Aila, “Improved Precision and Recall Metric for Assessing Generative Models,” NeurIPS, 2019. Paper
  32. G. Stein et al., “Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models,” NeurIPS, 2023. Paper
  33. S. Jayasumana et al., “Rethinking FID: Towards a Better Evaluation Metric for Image Generation,” CVPR, 2024. Paper
  34. A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” NeurIPS, 2017. Paper
  35. R. Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. Paper
  36. M. Kang et al., “Scaling up GANs for Text-to-Image Synthesis,” CVPR, 2023. Paper
  37. Y. Huang, A. Gokaslan, V. Kuleshov and J. Tompkin, “The GAN is dead; long live the GAN! A Modern GAN Baseline,” NeurIPS, 2024. Paper