完整搜尋頁面 →

剪枝耦合的通道

選讀・DLCV 01C・模型壓縮與高效推論 的補充資料(第 2/6 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 6

  1. N:M 半結構化稀疏
  2. 剪枝耦合的通道
  3. 漸進式剪枝排程
  4. 用定點乘數重新量化
  5. 選擇截斷範圍:min-max、百分位數與 MSE
  6. 超越四捨五入:AdaRound、BRECQ 與 LSQ

論文導讀 · 2

  1. The Lottery Ticket Hypothesis(Frankle 與 Carbin,ICLR 2019)
  2. A White Paper on Neural Network Quantization(Nagel 等人,2021)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則列出決定哪些通道必須一起剪的相依規則,這是結構化剪枝中錯誤的主要來源。

當某一層的輸出通道 jj 被移除時,每個使用或產生通道 jj 的張量都必須一致地改變。常見視覺區塊的規則如下:

  • 連續的卷積。 移除卷積 ℓ\ell 的輸出通道 jj,會移除它的核與偏置的第 jj 列、它後面任何批次正規化的對應項(γj\gamma_j、βj\beta_j、μj\mu_j、σj2\sigma_j^2),以及卷積 ℓ+1\ell + 1 的輸入通道 jj。剪枝程式碼對 MLP 做的就是這件事。
  • 殘差相加。 在 y=x+F(x)y = x + F(x) 中,xx 的通道 jj 與 F(x)F(x) 的通道 jj 會被相加,所以它們構成一個群組:通道 jj 只能同時從區塊輸入、FF 的最後一層、區塊輸出,以及每個共用同一條殘差流的後續區塊一次移除。這個群組的重要性必須共同判斷,例如把所有成員的範數加總。FF 的內部通道(瓶頸的中間)則是自由的。
  • 深度卷積。 輸入通道 jj 與輸出通道 jj 是同一個濾波器,所以移除一個就會移除另一個,這會耦合前後的 1×11 \times 1 層。
  • 串接。 在 concat⁡(a,b)\operatorname{concat}(a, b) 之後,bb 的通道索引會偏移 aa 的寬度;移除 aa 的一個通道,會讓下一層輸入中 bb 的索引位移。
  • 注意力頭。 移除一個注意力頭,會移除它的查詢、鍵與值投影,以及輸出投影中對應的輸入片段。

Fang 等人的 DepGraph 從計算圖自動建立這些群組,並顯示在群組正確之後,即使是簡單的、以範數為基礎的準則也很有效,涵蓋 ResNe(X)t、DenseNet、MobileNet 與視覺 Transformer 等架構 [9]。任何結構化剪枝之後,一個實用的檢查是:匯出剪枝後的模型,跑一個批次,並與遮罩版(尚未切除)的模型比較;如程式碼那樣輸出完全相同,就證明切除的動作遵守了每一個相依關係。

參考文獻

  1. M. Dehghani, A. Arnab, L. Beyer, A. Vaswani and Y. Tay, “The efficiency misnomer,” ICLR, 2022. arXiv
  2. S. Williams, A. Waterman and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Communications of the ACM, vol. 52, no. 4, pp. 65–76, 2009. doi
  3. Z. Liu, M. Sun, T. Zhou, G. Huang and T. Darrell, “Rethinking the value of network pruning,” ICLR, 2019. arXiv
  4. D. Blalock, J. J. Gonzalez Ortiz, J. Frankle and J. Guttag, “What is the state of neural network pruning?,” MLSys, 2020. arXiv
  5. H. Li, A. Kadav, I. Durdanovic, H. Samet and H. P. Graf, “Pruning filters for efficient ConvNets,” ICLR, 2017. arXiv
  6. A. Mishra, J. Albericio Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu and P. Micikevicius, “Accelerating sparse deep neural networks,” arXiv:2104.08378, 2021. arXiv
  7. Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan and C. Zhang, “Learning efficient convolutional networks through network slimming,” ICCV, 2017. arXiv
  8. T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden and A. Peste, “Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks,” Journal of Machine Learning Research, vol. 22, no. 241, pp. 1–124, 2021. paper
  9. G. Fang, X. Ma, M. Song, M. B. Mi and X. Wang, “DepGraph: towards any structural pruning,” CVPR, 2023. doi arXiv
  10. S. Han, J. Pool, J. Tran and W. J. Dally, “Learning both weights and connections for efficient neural networks,” NeurIPS (NIPS), 2015. arXiv
  11. S. Han, H. Mao and W. J. Dally, “Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding,” ICLR, 2016. arXiv
  12. M. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression,” arXiv:1710.01878, 2017. arXiv
  13. J. Frankle and M. Carbin, “The lottery ticket hypothesis: finding sparse, trainable neural networks,” ICLR, 2019. arXiv
  14. J. Frankle, G. K. Dziugaite, D. M. Roy and M. Carbin, “Linear mode connectivity and the lottery ticket hypothesis,” ICML, 2020. arXiv
  15. M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. van Baalen and T. Blankevoort, “A white paper on neural network quantization,” arXiv:2106.08295, 2021. arXiv
  16. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” CVPR, 2018. doi arXiv
  17. M. Nagel, M. van Baalen, T. Blankevoort and M. Welling, “Data-free quantization through weight equalization and bias correction,” ICCV, 2019. arXiv
  18. T. Dettmers, M. Lewis, Y. Belkada and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” NeurIPS, 2022. arXiv
  19. G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth and S. Han, “SmoothQuant: accurate and efficient post-training quantization for large language models,” ICML, 2023. arXiv
  20. M. Nagel, R. A. Amjad, M. van Baalen, C. Louizos and T. Blankevoort, “Up or down? Adaptive rounding for post-training quantization,” ICML, 2020. arXiv
  21. Y. Bengio, N. Léonard and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv:1308.3432, 2013. arXiv
  22. Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney and K. Keutzer, “HAWQ: Hessian aware quantization of neural networks with mixed-precision,” ICCV, 2019. arXiv
  23. K. Wang, Z. Liu, Y. Lin, J. Lin and S. Han, “HAQ: hardware-aware automated quantization with mixed precision,” CVPR, 2019. arXiv
  24. P. Micikevicius et al., “FP8 formats for deep learning,” arXiv:2209.05433, 2022. arXiv
  25. G. Hinton, O. Vinyals and J. Dean, “Distilling the knowledge in a neural network,” NIPS Deep Learning Workshop, 2014; arXiv:1503.02531. arXiv
  26. L. J. Ba and R. Caruana, “Do deep nets really need to be deep?,” NeurIPS (NIPS), 2014. arXiv
  27. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta and Y. Bengio, “FitNets: hints for thin deep nets,” ICLR, 2015. arXiv
  28. S. Zagoruyko and N. Komodakis, “Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer,” ICLR, 2017. arXiv
  29. J. Gou, B. Yu, S. J. Maybank and D. Tao, “Knowledge distillation: a survey,” International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021. doi
  30. L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil and A. Kolesnikov, “Knowledge distillation: a good teacher is patient and consistent,” CVPR, 2022. arXiv
  31. J. H. Cho and B. Hariharan, “On the efficacy of knowledge distillation,” ICCV, 2019. arXiv
  32. H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles and H. Jégou, “Training data-efficient image transformers & distillation through attention,” ICML, 2021. arXiv
  33. R. Denton, W. Zaremba, J. Bruna, Y. LeCun and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” NeurIPS (NIPS), 2014. arXiv
  34. M. Jaderberg, A. Vedaldi and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” BMVC, 2014. arXiv
  35. X. Zhang, J. Zou, K. He and J. Sun, “Accelerating very deep convolutional networks for classification and detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 10, pp. 1943–1955, 2016. doi arXiv
  36. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto and H. Adam, “MobileNets: efficient convolutional neural networks for mobile vision applications,” arXiv:1704.04861, 2017. arXiv
  37. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L.-C. Chen, “MobileNetV2: inverted residuals and linear bottlenecks,” CVPR, 2018. arXiv
  38. M. Tan and Q. V. Le, “EfficientNet: rethinking model scaling for convolutional neural networks,” ICML, 2019. arXiv
  39. X. Ding, X. Zhang, N. Ma, J. Han, G. Ding and J. Sun, “RepVGG: making VGG-style ConvNets great again,” CVPR, 2021. arXiv
  40. C. Li et al., “YOLOv6: a single-stage object detection framework for industrial applications,” arXiv:2209.02976, 2022. arXiv
  41. T. Elsken, J. H. Metzen and F. Hutter, “Neural architecture search: a survey,” Journal of Machine Learning Research, vol. 20, pp. 1–21, 2019. arXiv
  42. B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” ICLR, 2017. arXiv
  43. H. Pham, M. Y. Guan, B. Zoph, Q. V. Le and J. Dean, “Efficient neural architecture search via parameter sharing,” ICML, 2018. arXiv
  44. H. Liu, K. Simonyan and Y. Yang, “DARTS: differentiable architecture search,” ICLR, 2019. arXiv
  45. H. Cai, C. Gan, T. Wang, Z. Zhang and S. Han, “Once-for-all: train one network and specialize it for efficient deployment,” ICLR, 2020. arXiv
  46. M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard and Q. V. Le, “MnasNet: platform-aware neural architecture search for mobile,” CVPR, 2019. arXiv
  47. H. Cai, L. Zhu and S. Han, “ProxylessNAS: direct neural architecture search on target task and hardware,” ICLR, 2019. arXiv
  48. B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia and K. Keutzer, “FBNet: hardware-aware efficient ConvNet design via differentiable neural architecture search,” CVPR, 2019. doi arXiv
  49. L. Li and A. Talwalkar, “Random search and reproducibility for neural architecture search,” UAI, 2019. arXiv
  50. S. Ioffe and C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift,” ICML, 2015. arXiv
  51. T. Wang, L. Yuan, X. Zhang and J. Feng, “Distilling object detectors with fine-grained feature imitation,” CVPR, 2019. arXiv
  52. H. Li, C. Yan, S. Lin, X. Zheng, Y. Li, B. Zhang, F. Yang and R. Ji, “PAMS: quantized super-resolution via parameterized max scale,” ECCV, 2020. arXiv
  53. C. Hong, H. Kim, S. Baik, J. Oh and K. M. Lee, “DAQ: channel-wise distribution-aware quantization for deep image super-resolution networks,” WACV, 2022. arXiv
  54. B. Ren, Y. Li, N. Mehta, R. Timofte et al., “The ninth NTIRE 2024 efficient super-resolution challenge report,” CVPR Workshops, 2024. doi arXiv
  55. Z. Yuan, C. Xue, Y. Chen, Q. Wu and G. Sun, “PTQ4ViT: post-training quantization for vision transformers with twin uniform quantization,” ECCV, 2022. doi arXiv
  56. Z. Liu, Y. Wang, K. Han, S. Ma and W. Gao, “Post-training quantization for vision transformer,” NeurIPS, 2021. arXiv
  57. X. Li, Y. Liu, L. Lian, H. Yang, Z. Dong, D. Kang, S. Zhang and K. Keutzer, “Q-Diffusion: quantizing diffusion models,” ICCV, 2023. doi arXiv
  58. Y. Shang, Z. Yuan, B. Xie, B. Wu and Y. Yan, “Post-training quantization on diffusion models,” CVPR, 2023. arXiv
  59. A. Kirillov et al., “Segment anything,” ICCV, 2023. doi arXiv
  60. C. Zhang, D. Han, Y. Qiao, J. U. Kim, S.-H. Bae, S. Lee and C. S. Hong, “Faster segment anything: towards lightweight SAM for mobile applications,” arXiv:2306.14289, 2023. arXiv
  61. Y. Xiong et al., “EfficientSAM: leveraged masked image pretraining for efficient segment anything,” CVPR, 2024. doi arXiv
  62. Y. Cheng, D. Wang, P. Zhou and T. Zhang, “Model compression and acceleration for deep neural networks: the principles, progress, and challenges,” IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 126–136, 2018. doi arXiv
  63. G. Menghani, “Efficient deep learning: a survey on making deep learning models smaller, faster, and better,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–37, 2023. doi arXiv
  64. A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” book chapter in Low-Power Computer Vision; arXiv:2103.13630, 2021. arXiv
  65. H. Cheng, M. Zhang and J. Q. Shi, “A survey on deep neural network pruning: taxonomy, comparison, analysis, and recommendations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10558–10578, 2024. doi arXiv
  66. L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: a review and new outlooks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 6, pp. 3048–3068, 2022. doi arXiv
  67. C. White, M. Safari, R. Sukthanker, B. Ru, T. Elsken, A. Zela, D. Dey and F. Hutter, “Neural architecture search: insights from 1000 papers,” arXiv:2301.08727, 2023. arXiv
  68. E. Frantar, S. Ashkboos, T. Hoefler and D. Alistarh, “GPTQ: accurate post-training quantization for generative pre-trained transformers,” ICLR, 2023. arXiv
  69. J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan and S. Han, “AWQ: activation-aware weight quantization for LLM compression and acceleration,” MLSys, 2024. arXiv
  70. M. Li, Y. Lin, Z. Zhang, T. Cai, X. Li, J. Guo, E. Xie, C. Meng, J.-Y. Zhu and S. Han, “SVDQuant: absorbing outliers by low-rank components for 4-bit diffusion models,” ICLR, 2025. arXiv
  71. Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang and S. Gu, “BRECQ: pushing the limit of post-training quantization by block reconstruction,” ICLR, 2021. arXiv
  72. S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy and D. S. Modha, “Learned step size quantization,” ICLR, 2020. arXiv