VQ-VAE — discrete codes and a separately learned prior
Optional · paper guide 1 of 1 for ML 09 · Autoregressive Vision: Pixels, Tokens and Scales. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Question and thesis
Can an autoencoder learn useful discrete latent variables without differentiating through an ordinary categorical sample? Van den Oord, Vinyals and Kavukcuoglu quantize encoder outputs with a learned codebook, then learn a prior over the resulting codes [4]. Read nearest-neighbor quantization, autoencoders and autoregressive factorization first. Separating representation learning from prior learning is essential to understanding the original procedure.
Method and figure

For one latent position, select a code and train with
Here is the input, is the encoder output, is codebook entry , indexes the nearest entry and is the quantized vector. The decoder likelihood is , stops gradients and weights commitment. The second term moves selected embeddings toward encoder outputs; the third discourages the encoder from drifting away from its selected codes.
Figure 1 shows the discrete bottleneck and gradient path. Reconstruction gradients are passed to the encoder with a straight-through estimator. That is a deliberate gradient approximation, not the exact derivative of nearest-neighbor selection. The original procedure subsequently trains an autoregressive prior on codes from the learned representation.
Evidence
Section 4.1 compares continuous VAE, VQ-VAE and a discrete alternative on CIFAR-10 under the described architecture comparison. The reported variational coding scores are 4.51, 4.67 and 5.14 bits/dim respectively, with lower better. These are bound-based scores, not exact marginal likelihoods. The discrete representation therefore does not beat the continuous VAE on that reported measure; the paper’s contribution also concerns useful discrete structure and learned generation.
Design analysis
Reconstruction trains the encoder–decoder pair to preserve information, while the prior learns dependencies among the resulting symbols. Sampling independent uniform code indices would skip those dependencies and is not equivalent to sampling the trained generative model. The codebook and commitment terms have different recipients because of their stop-gradient operators; deleting those operators changes the optimization problem.
Reading checkpoint
For each of the three loss terms, identify which parameters receive gradients. Then describe the sequence for generating without an input image: draw codes from the learned prior, look up embeddings, and decode. Explain why a good reconstruction alone does not evaluate that prior.
Limits and next step
Discrete codes do not guarantee that every codebook entry is used. Underused codes are a different issue from a decoder ignoring its latent variable. Quantization also creates a rate–distortion tradeoff. Continue with this chapter’s autoregressive token models, tracking reconstruction quality and prior-model quality separately before comparing complete generators.
Continue reading: ML 09 · Return to the chapter
References
- J. Xiong, G. Liu, L. Huang et al., “Autoregressive Models in Vision: A Survey,” Transactions on Machine Learning Research (TMLR), 2025 (arXiv:2411.05902, 2024). arXiv
- A. van den Oord, N. Kalchbrenner and K. Kavukcuoglu, “Pixel Recurrent Neural Networks,” in Proc. ICML, PMLR 48, 2016. PMLR
- A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves and K. Kavukcuoglu, “Conditional Image Generation with PixelCNN Decoders,” in NeurIPS, 2016. NeurIPS
- A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” in NeurIPS, 2017. NeurIPS
- P. Esser, R. Rombach and B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” in Proc. IEEE/CVF CVPR, 2021. CVF
- K. Tian, Y. Jiang, Z. Yuan, B. Peng and L. Wang, “Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction,” in NeurIPS, 2024 (Best Paper Award). arXiv
- T. Li, Y. Tian, H. Li, M. Deng and K. He, “Autoregressive Image Generation without Vector Quantization,” in NeurIPS, 2024. arXiv
- H. Chang, H. Zhang, L. Jiang, C. Liu and W. T. Freeman, “MaskGIT: Masked Generative Image Transformer,” in Proc. IEEE/CVF CVPR, 2022. CVF
- S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book
- A. Razavi, A. van den Oord and O. Vinyals, “Generating Diverse High-Fidelity Images with VQ-VAE-2,” in NeurIPS, 2019. NeurIPS
- M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan and I. Sutskever, “Generative Pretraining From Pixels,” in Proc. ICML, PMLR 119, 2020. PMLR
- J. Yu, X. Li, J. Y. Koh et al., “Vector-quantized Image Modeling with Improved VQGAN,” in ICLR, 2022. arXiv
- J. Yu, Y. Xu, J. Y. Koh et al., “Scaling Autoregressive Models for Content-Rich Text-to-Image Generation,” Transactions on Machine Learning Research, 2022. arXiv
- P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo and Z. Yuan, “Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation,” arXiv:2406.06525, 2024. arXiv
- D. Lee, C. Kim, S. Kim, M. Cho and W.-S. Han, “Autoregressive Image Generation using Residual Quantization,” in Proc. IEEE/CVF CVPR, 2022. CVF
- F. Mentzer, D. Minnen, E. Agustsson and M. Tschannen, “Finite Scalar Quantization: VQ-VAE Made Simple,” in Proc. International Conference on Learning Representations (ICLR), 2024. arXiv
- H. Chang, H. Zhang, J. Barber et al., “Muse: Text-To-Image Generation via Masked Generative Transformers,” in Proc. ICML, PMLR 202, 2023. PMLR
- J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv:2207.12598, 2022 (short version at the NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications). arXiv
- S. Bond-Taylor, A. Leach, Y. Long and C. G. Willcocks, “Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,” IEEE TPAMI, vol. 44, no. 11, 2022. doi
- J. Li, Y. Fu, J. Liu, L. Cao, W. Ji, M. Yang, I. King and M.-H. Yang, “Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey,” IEEE TPAMI, 2026 (arXiv:2507.22920). doi
- S. Zhao, X. Zhang, J. Guo et al., “Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities,” arXiv:2505.02567, 2025; revised 2026. arXiv
- J. Jia, J. Gao, B. Xue et al., “From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval,” arXiv:2502.12448, 2025. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
- H. Touvron, T. Lavril, G. Izacard et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv:2302.13971, 2023. arXiv