完整搜尋頁面 →

VQ-VAE:離散編碼與另行學習的先驗

選讀・ML 09・視覺自迴歸:像素、Tokens 與尺度 的論文導讀(第 1/1 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

問題與論點

自編碼器能否學到有用的離散潛在變數,又不必直接對一般類別取樣微分?Van den Oord、Vinyals 與 Kavukcuoglu 用可學習碼本量化編碼器輸出,再學習編碼上的先驗 [4]。先讀最近鄰量化、自編碼器與自迴歸分解。區分表示學習及先驗學習,才能理解原始流程。

方法與圖解

編碼器選取最近的碼本向量,解碼器據此重建;穿過離散選擇的直通梯度是估計方式。
來源:van den Oord, Vinyals and Kavukcuoglu, NeurIPS 2017 proceedings. 圖 1,PDF 第 4 頁。編碼器選取最近的碼本向量,解碼器據此重建;穿過離散選擇的直通梯度是估計方式。 [4]

對單一潛在位置,選取編碼並使用下列目標:

k=arg⁡min⁡j∥ze(x)−ej∥2,zq=ek,L=−log⁡p(x∣zq)+∥sg⁡[ze(x)]−ek∥2+β∥ze(x)−sg⁡[ek]∥2.\begin{aligned} k&=\arg\min_j\|z_e(x)-e_j\|^2,\qquad z_q=e_k,\\ L&=-\log p(x\mid z_q)+\|\operatorname{sg}[z_e(x)]-e_k\|^2+\beta\|z_e(x)-\operatorname{sg}[e_k]\|^2. \end{aligned}

其中 xx 為輸入,ze(x)z_e(x) 為編碼器輸出,eje_j 為第 jj 個碼本向量,kk 為最近向量索引,zqz_q 為量化向量。p(x∣zq)p(x\mid z_q) 是解碼器概似,sg⁡\operatorname{sg} 阻斷梯度,β\beta 是承諾項權重。第二項讓選中的嵌入靠近編碼器輸出,第三項避免編碼器遠離選中的編碼。

圖 1 顯示離散瓶頸及梯度路徑。重建梯度透過直通估計器傳回編碼器;這是刻意採用的梯度近似,不是最近鄰選擇的精確導數。原始流程之後才在學得表示的編碼上,訓練自迴歸先驗。

證據

第 4.1 節依所述架構比較,在 CIFAR-10 上評估連續 VAE、VQ-VAE 與另一離散方法。報告的變分編碼分數依序為 4.51、4.67、5.14 bits/dim,越低越好。這些基於界限的分數,不是精確邊際概似。因此離散表示在這項量測上沒有勝過連續 VAE;貢獻還包含有用離散結構及學得的生成能力。

設計分析

重建訓練編碼器與解碼器保存資訊;先驗則學習所得符號間的相依性。獨立均勻抽取碼本索引會略過這些相依關係,不等於從訓練好的生成模型取樣。碼本項與承諾項因梯度阻斷而更新不同對象,刪掉阻斷操作就改變了最佳化問題。

閱讀檢查點

逐項指出三個損失各讓哪些參數收到梯度。再描述沒有輸入影像時的生成順序:由學得先驗抽取編碼、查找嵌入、解碼。說明為何良好重建本身沒有評估那個先驗。

限制與下一步

離散編碼不保證每個碼本向量都被使用。編碼使用不足,與解碼器忽略潛在變數,是不同問題;量化也帶來率失真取捨。接著讀本章自迴歸 token 模型,在比較完整生成器以前,分別追蹤重建品質及先驗模型品質。

接著閱讀: ML 09・回到章節

參考文獻

  1. J. Xiong, G. Liu, L. Huang et al., “Autoregressive Models in Vision: A Survey,” Transactions on Machine Learning Research (TMLR), 2025 (arXiv:2411.05902, 2024). arXiv
  2. A. van den Oord, N. Kalchbrenner and K. Kavukcuoglu, “Pixel Recurrent Neural Networks,” in Proc. ICML, PMLR 48, 2016. PMLR
  3. A. van den Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves and K. Kavukcuoglu, “Conditional Image Generation with PixelCNN Decoders,” in NeurIPS, 2016. NeurIPS
  4. A. van den Oord, O. Vinyals and K. Kavukcuoglu, “Neural Discrete Representation Learning,” in NeurIPS, 2017. NeurIPS
  5. P. Esser, R. Rombach and B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” in Proc. IEEE/CVF CVPR, 2021. CVF
  6. K. Tian, Y. Jiang, Z. Yuan, B. Peng and L. Wang, “Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction,” in NeurIPS, 2024 (Best Paper Award). arXiv
  7. T. Li, Y. Tian, H. Li, M. Deng and K. He, “Autoregressive Image Generation without Vector Quantization,” in NeurIPS, 2024. arXiv
  8. H. Chang, H. Zhang, L. Jiang, C. Liu and W. T. Freeman, “MaskGIT: Masked Generative Image Transformer,” in Proc. IEEE/CVF CVPR, 2022. CVF
  9. S. J. D. Prince, Understanding Deep Learning, MIT Press, 2023. Book
  10. A. Razavi, A. van den Oord and O. Vinyals, “Generating Diverse High-Fidelity Images with VQ-VAE-2,” in NeurIPS, 2019. NeurIPS
  11. M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan and I. Sutskever, “Generative Pretraining From Pixels,” in Proc. ICML, PMLR 119, 2020. PMLR
  12. J. Yu, X. Li, J. Y. Koh et al., “Vector-quantized Image Modeling with Improved VQGAN,” in ICLR, 2022. arXiv
  13. J. Yu, Y. Xu, J. Y. Koh et al., “Scaling Autoregressive Models for Content-Rich Text-to-Image Generation,” Transactions on Machine Learning Research, 2022. arXiv
  14. P. Sun, Y. Jiang, S. Chen, S. Zhang, B. Peng, P. Luo and Z. Yuan, “Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation,” arXiv:2406.06525, 2024. arXiv
  15. D. Lee, C. Kim, S. Kim, M. Cho and W.-S. Han, “Autoregressive Image Generation using Residual Quantization,” in Proc. IEEE/CVF CVPR, 2022. CVF
  16. F. Mentzer, D. Minnen, E. Agustsson and M. Tschannen, “Finite Scalar Quantization: VQ-VAE Made Simple,” in Proc. International Conference on Learning Representations (ICLR), 2024. arXiv
  17. H. Chang, H. Zhang, J. Barber et al., “Muse: Text-To-Image Generation via Masked Generative Transformers,” in Proc. ICML, PMLR 202, 2023. PMLR
  18. J. Ho and T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv:2207.12598, 2022 (short version at the NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications). arXiv
  19. S. Bond-Taylor, A. Leach, Y. Long and C. G. Willcocks, “Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,” IEEE TPAMI, vol. 44, no. 11, 2022. doi
  20. J. Li, Y. Fu, J. Liu, L. Cao, W. Ji, M. Yang, I. King and M.-H. Yang, “Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey,” IEEE TPAMI, 2026 (arXiv:2507.22920). doi
  21. S. Zhao, X. Zhang, J. Guo et al., “Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities,” arXiv:2505.02567, 2025; revised 2026. arXiv
  22. J. Jia, J. Gao, B. Xue et al., “From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval,” arXiv:2502.12448, 2025. arXiv
  23. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, pp. 248–255. doi
  24. H. Touvron, T. Lavril, G. Izacard et al., “LLaMA: Open and Efficient Foundation Language Models,” arXiv:2302.13971, 2023. arXiv