VQ-VAE:離散編碼與另行學習的先驗
選讀・ML 09・視覺自迴歸:像素、Tokens 與尺度 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
問題與論點
自編碼器能否學到有用的離散潛在變數,又不必直接對一般類別取樣微分?Van den Oord、Vinyals 與 Kavukcuoglu 用可學習碼本量化編碼器輸出,再學習編碼上的先驗 [4]。先讀最近鄰量化、自編碼器與自迴歸分解。區分表示學習及先驗學習,才能理解原始流程。
方法與圖解

對單一潛在位置,選取編碼並使用下列目標:
其中 為輸入, 為編碼器輸出, 為第 個碼本向量, 為最近向量索引, 為量化向量。 是解碼器概似, 阻斷梯度, 是承諾項權重。第二項讓選中的嵌入靠近編碼器輸出,第三項避免編碼器遠離選中的編碼。
圖 1 顯示離散瓶頸及梯度路徑。重建梯度透過直通估計器傳回編碼器;這是刻意採用的梯度近似,不是最近鄰選擇的精確導數。原始流程之後才在學得表示的編碼上,訓練自迴歸先驗。
證據
第 4.1 節依所述架構比較,在 CIFAR-10 上評估連續 VAE、VQ-VAE 與另一離散方法。報告的變分編碼分數依序為 4.51、4.67、5.14 bits/dim,越低越好。這些基於界限的分數,不是精確邊際概似。因此離散表示在這項量測上沒有勝過連續 VAE;貢獻還包含有用離散結構及學得的生成能力。
設計分析
重建訓練編碼器與解碼器保存資訊;先驗則學習所得符號間的相依性。獨立均勻抽取碼本索引會略過這些相依關係,不等於從訓練好的生成模型取樣。碼本項與承諾項因梯度阻斷而更新不同對象,刪掉阻斷操作就改變了最佳化問題。
閱讀檢查點
逐項指出三個損失各讓哪些參數收到梯度。再描述沒有輸入影像時的生成順序:由學得先驗抽取編碼、查找嵌入、解碼。說明為何良好重建本身沒有評估那個先驗。
限制與下一步
離散編碼不保證每個碼本向量都被使用。編碼使用不足,與解碼器忽略潛在變數,是不同問題;量化也帶來率失真取捨。接著讀本章自迴歸 token 模型,在比較完整生成器以前,分別追蹤重建品質及先驗模型品質。
接著閱讀: ML 09・回到章節
參考文獻
- J. Xiong, G. Liu, L. Huang et al., “Autoregressive Models in Vision: A Survey,” Transactions on Machine Learning Research (TMLR), 2025 (arXiv:2411.05902, 2024). arXiv
- A. van den Oord, N. Kalchbrenner and K. Kavukcuoglu, “Pixel Recurrent Neural Networks,” in Proc. ICML, PMLR 48, 2016. PMLR
- A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves and K. Kavukcuoglu, “Conditional Image Generation with PixelCNN Decoders,” in NeurIPS, 2016. NeurIPS
- A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” in NeurIPS, 2017. NeurIPS
- P. Esser, R. Rombach and B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” in Proc. IEEE/CVF CVPR, 2021. CVF
- K. Tian, Y. Jiang, Z. Yuan, B. Peng and L. Wang, “Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction,” in NeurIPS, 2024 (Best Paper Award). arXiv
- T. Li, Y. Tian, H. Li, M. Deng and K. He, “Autoregressive Image Generation without Vector Quantization,” in NeurIPS, 2024. arXiv
- H. Chang, H. Zhang, L. Jiang, C. Liu and W. T. Freeman, “MaskGIT: Masked Generative Image Transformer,” in Proc. IEEE/CVF CVPR, 2022. CVF
- S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book
- A. Razavi, A. van den Oord and O. Vinyals, “Generating Diverse High-Fidelity Images with VQ-VAE-2,” in NeurIPS, 2019. NeurIPS
- M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan and I. Sutskever, “Generative Pretraining From Pixels,” in Proc. ICML, PMLR 119, 2020. PMLR
- J. Yu, X. Li, J. Y. Koh et al., “Vector-quantized Image Modeling with Improved VQGAN,” in ICLR, 2022. arXiv
- J. Yu, Y. Xu, J. Y. Koh et al., “Scaling Autoregressive Models for Content-Rich Text-to-Image Generation,” Transactions on Machine Learning Research, 2022. arXiv
- P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo and Z. Yuan, “Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation,” arXiv:2406.06525, 2024. arXiv
- D. Lee, C. Kim, S. Kim, M. Cho and W.-S. Han, “Autoregressive Image Generation using Residual Quantization,” in Proc. IEEE/CVF CVPR, 2022. CVF
- F. Mentzer, D. Minnen, E. Agustsson and M. Tschannen, “Finite Scalar Quantization: VQ-VAE Made Simple,” in Proc. International Conference on Learning Representations (ICLR), 2024. arXiv
- H. Chang, H. Zhang, J. Barber et al., “Muse: Text-To-Image Generation via Masked Generative Transformers,” in Proc. ICML, PMLR 202, 2023. PMLR
- J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv:2207.12598, 2022 (short version at the NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications). arXiv
- S. Bond-Taylor, A. Leach, Y. Long and C. G. Willcocks, “Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,” IEEE TPAMI, vol. 44, no. 11, 2022. doi
- J. Li, Y. Fu, J. Liu, L. Cao, W. Ji, M. Yang, I. King and M.-H. Yang, “Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey,” IEEE TPAMI, 2026 (arXiv:2507.22920). doi
- S. Zhao, X. Zhang, J. Guo et al., “Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities,” arXiv:2505.02567, 2025; revised 2026. arXiv
- J. Jia, J. Gao, B. Xue et al., “From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval,” arXiv:2502.12448, 2025. arXiv
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
- H. Touvron, T. Lavril, G. Izacard et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv:2302.13971, 2023. arXiv