完整搜尋頁面 →

N:M 半結構化稀疏

選讀・DLCV 01C・模型壓縮與高效推論 的補充資料(第 1/6 則)。主線不依賴這則補充;引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 6

  1. N:M 半結構化稀疏
  2. 剪枝耦合的通道
  3. 漸進式剪枝排程
  4. 用定點乘數重新量化
  5. 選擇截斷範圍:min-max、百分位數與 MSE
  6. 超越四捨五入:AdaRound、BRECQ 與 LSQ

論文導讀 · 2

  1. The Lottery Ticket Hypothesis(Frankle 與 Carbin,ICLR 2019)
  2. A White Paper on Neural Network Quantization(Nagel 等人,2021)

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

這則說明稀疏 GPU 硬體使用的 2:4 樣式,以及它與非結構化剪枝相比要付出什麼代價。

在 N:M 稀疏中,每連續 MM 個權重(沿著矩陣乘法的輸入維度)最多包含 NN 個非零值;2:4 是 NVIDIA 稀疏 Tensor Core 所支援的情況,作者報告這種樣式讓稠密單元的數學吞吐量加倍 [6]。每組只儲存兩個非零值,另外每個儲存值加 2 位元的中繼資料,說明它在組內的位置;對 16 位元的值,一組四個用 36 位元儲存而不是 64 位元,少約 44% [6]。他們的流程是:先訓練稠密網路,把每組中量值最小的兩個權重移除而剪成 2:4,再用相同的最佳化器與排程重新訓練,同時保持樣式固定 [6]。因為這個限制是局部的,剪枝可能會損失一些相對大的權重;作者指出,剪枝前先置換輸入通道,可以保留更多權重量值,不過在他們測試的大多數網路中並不需要這樣做 [6]。

下面的程式碼在一個隨機矩陣上量測這個代價:只要四個一組中有三個大的,2:4 就必須丟掉其中一個大權重,而在同樣 50% 下的非結構化剪枝會把它留下。

程式碼 · Python · 20 行
import numpy as np

rng = np.random.default_rng(0)
W = rng.normal(size=(256, 512))

def prune_2_4(W):
    """Keep the two largest |w| in every group of four consecutive weights along a row."""
    groups = W.reshape(W.shape[0], -1, 4)
    order = np.argsort(np.abs(groups), axis=-1)                # positions sorted by magnitude
    mask = np.ones_like(groups, dtype=bool)
    np.put_along_axis(mask, order[..., :2], False, axis=-1)     # drop the two smallest
    return (groups * mask).reshape(W.shape)

W_24 = prune_2_4(W)
W_un = W * (np.abs(W) > np.median(np.abs(W)))                    # unstructured, also 50%
kept = lambda V: (V**2).sum() / (W**2).sum()
print("zeros: 2:4", (W_24 == 0).mean(), "| unstructured", (W_un == 0).mean())
print(f"squared magnitude kept: 2:4 {kept(W_24):.3f} | unstructured {kept(W_un):.3f}")
print("every group of four has exactly two nonzeros:",
      bool(((W_24.reshape(256, -1, 4) != 0).sum(-1) == 2).all()))

輸出:

程式碼 · 3 行
zeros: 2:4 0.5 | unstructured 0.5
squared magnitude kept: 2:4 0.867 | unstructured 0.928
every group of four has exactly two nonzeros: True

對隨機高斯矩陣,2:4 樣式保留了 86.7% 的權重平方量值,非結構化剪枝在相同稀疏度下則是 92.8%;重新訓練才是彌補準確率差距的關鍵。邊界情況:這個樣式是沿著歸約維度套用的,所以對卷積而言,各組是依核的排列方式,沿著輸入通道乘以核位置展開;而加速只出現在一層中矩陣乘法的部分,且要在支援它的硬體與函式庫上。

參考文獻

  1. M. Dehghani, A. Arnab, L. Beyer, A. Vaswani and Y. Tay, “The efficiency misnomer,” ICLR, 2022. arXiv
  2. S. Williams, A. Waterman and D. Patterson, “Roofline: an insightful visual performance model for multicore architectures,” Communications of the ACM, vol. 52, no. 4, pp. 65–76, 2009. doi
  3. Z. Liu, M. Sun, T. Zhou, G. Huang and T. Darrell, “Rethinking the value of network pruning,” ICLR, 2019. arXiv
  4. D. Blalock, J. J. Gonzalez Ortiz, J. Frankle and J. Guttag, “What is the state of neural network pruning?,” MLSys, 2020. arXiv
  5. H. Li, A. Kadav, I. Durdanovic, H. Samet and H. P. Graf, “Pruning filters for efficient ConvNets,” ICLR, 2017. arXiv
  6. A. Mishra, J. Albericio Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu and P. Micikevicius, “Accelerating sparse deep neural networks,” arXiv:2104.08378, 2021. arXiv
  7. Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan and C. Zhang, “Learning efficient convolutional networks through network slimming,” ICCV, 2017. arXiv
  8. T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden and A. Peste, “Sparsity in deep learning: pruning and growth for efficient inference and training in neural networks,” Journal of Machine Learning Research, vol. 22, no. 241, pp. 1–124, 2021. paper
  9. G. Fang, X. Ma, M. Song, M. B. Mi and X. Wang, “DepGraph: towards any structural pruning,” CVPR, 2023. doi arXiv
  10. S. Han, J. Pool, J. Tran and W. J. Dally, “Learning both weights and connections for efficient neural networks,” NeurIPS (NIPS), 2015. arXiv
  11. S. Han, H. Mao and W. J. Dally, “Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding,” ICLR, 2016. arXiv
  12. M. Zhu and S. Gupta, “To prune, or not to prune: exploring the efficacy of pruning for model compression,” arXiv:1710.01878, 2017. arXiv
  13. J. Frankle and M. Carbin, “The lottery ticket hypothesis: finding sparse, trainable neural networks,” ICLR, 2019. arXiv
  14. J. Frankle, G. K. Dziugaite, D. M. Roy and M. Carbin, “Linear mode connectivity and the lottery ticket hypothesis,” ICML, 2020. arXiv
  15. M. Nagel, M. Fournarakis, R. A. Amjad, Y. Bondarenko, M. van Baalen and T. Blankevoort, “A white paper on neural network quantization,” arXiv:2106.08295, 2021. arXiv
  16. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” CVPR, 2018. doi arXiv
  17. M. Nagel, M. van Baalen, T. Blankevoort and M. Welling, “Data-free quantization through weight equalization and bias correction,” ICCV, 2019. arXiv
  18. T. Dettmers, M. Lewis, Y. Belkada and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” NeurIPS, 2022. arXiv
  19. G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth and S. Han, “SmoothQuant: accurate and efficient post-training quantization for large language models,” ICML, 2023. arXiv
  20. M. Nagel, R. A. Amjad, M. van Baalen, C. Louizos and T. Blankevoort, “Up or down? Adaptive rounding for post-training quantization,” ICML, 2020. arXiv
  21. Y. Bengio, N. Léonard and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv:1308.3432, 2013. arXiv
  22. Z. Dong, Z. Yao, A. Gholami, M. W. Mahoney and K. Keutzer, “HAWQ: Hessian aware quantization of neural networks with mixed-precision,” ICCV, 2019. arXiv
  23. K. Wang, Z. Liu, Y. Lin, J. Lin and S. Han, “HAQ: hardware-aware automated quantization with mixed precision,” CVPR, 2019. arXiv
  24. P. Micikevicius et al., “FP8 formats for deep learning,” arXiv:2209.05433, 2022. arXiv
  25. G. Hinton, O. Vinyals and J. Dean, “Distilling the knowledge in a neural network,” NIPS Deep Learning Workshop, 2014; arXiv:1503.02531. arXiv
  26. L. J. Ba and R. Caruana, “Do deep nets really need to be deep?,” NeurIPS (NIPS), 2014. arXiv
  27. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta and Y. Bengio, “FitNets: hints for thin deep nets,” ICLR, 2015. arXiv
  28. S. Zagoruyko and N. Komodakis, “Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer,” ICLR, 2017. arXiv
  29. J. Gou, B. Yu, S. J. Maybank and D. Tao, “Knowledge distillation: a survey,” International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021. doi
  30. L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil and A. Kolesnikov, “Knowledge distillation: a good teacher is patient and consistent,” CVPR, 2022. arXiv
  31. J. H. Cho and B. Hariharan, “On the efficacy of knowledge distillation,” ICCV, 2019. arXiv
  32. H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles and H. Jégou, “Training data-efficient image transformers & distillation through attention,” ICML, 2021. arXiv
  33. R. Denton, W. Zaremba, J. Bruna, Y. LeCun and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” NeurIPS (NIPS), 2014. arXiv
  34. M. Jaderberg, A. Vedaldi and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” BMVC, 2014. arXiv
  35. X. Zhang, J. Zou, K. He and J. Sun, “Accelerating very deep convolutional networks for classification and detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 10, pp. 1943–1955, 2016. doi arXiv
  36. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto and H. Adam, “MobileNets: efficient convolutional neural networks for mobile vision applications,” arXiv:1704.04861, 2017. arXiv
  37. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L.-C. Chen, “MobileNetV2: inverted residuals and linear bottlenecks,” CVPR, 2018. arXiv
  38. M. Tan and Q. V. Le, “EfficientNet: rethinking model scaling for convolutional neural networks,” ICML, 2019. arXiv
  39. X. Ding, X. Zhang, N. Ma, J. Han, G. Ding and J. Sun, “RepVGG: making VGG-style ConvNets great again,” CVPR, 2021. arXiv
  40. C. Li et al., “YOLOv6: a single-stage object detection framework for industrial applications,” arXiv:2209.02976, 2022. arXiv
  41. T. Elsken, J. H. Metzen and F. Hutter, “Neural architecture search: a survey,” Journal of Machine Learning Research, vol. 20, pp. 1–21, 2019. arXiv
  42. B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” ICLR, 2017. arXiv
  43. H. Pham, M. Y. Guan, B. Zoph, Q. V. Le and J. Dean, “Efficient neural architecture search via parameter sharing,” ICML, 2018. arXiv
  44. H. Liu, K. Simonyan and Y. Yang, “DARTS: differentiable architecture search,” ICLR, 2019. arXiv
  45. H. Cai, C. Gan, T. Wang, Z. Zhang and S. Han, “Once-for-all: train one network and specialize it for efficient deployment,” ICLR, 2020. arXiv
  46. M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard and Q. V. Le, “MnasNet: platform-aware neural architecture search for mobile,” CVPR, 2019. arXiv
  47. H. Cai, L. Zhu and S. Han, “ProxylessNAS: direct neural architecture search on target task and hardware,” ICLR, 2019. arXiv
  48. B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia and K. Keutzer, “FBNet: hardware-aware efficient ConvNet design via differentiable neural architecture search,” CVPR, 2019. doi arXiv
  49. L. Li and A. Talwalkar, “Random search and reproducibility for neural architecture search,” UAI, 2019. arXiv
  50. S. Ioffe and C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift,” ICML, 2015. arXiv
  51. T. Wang, L. Yuan, X. Zhang and J. Feng, “Distilling object detectors with fine-grained feature imitation,” CVPR, 2019. arXiv
  52. H. Li, C. Yan, S. Lin, X. Zheng, Y. Li, B. Zhang, F. Yang and R. Ji, “PAMS: quantized super-resolution via parameterized max scale,” ECCV, 2020. arXiv
  53. C. Hong, H. Kim, S. Baik, J. Oh and K. M. Lee, “DAQ: channel-wise distribution-aware quantization for deep image super-resolution networks,” WACV, 2022. arXiv
  54. B. Ren, Y. Li, N. Mehta, R. Timofte et al., “The ninth NTIRE 2024 efficient super-resolution challenge report,” CVPR Workshops, 2024. doi arXiv
  55. Z. Yuan, C. Xue, Y. Chen, Q. Wu and G. Sun, “PTQ4ViT: post-training quantization for vision transformers with twin uniform quantization,” ECCV, 2022. doi arXiv
  56. Z. Liu, Y. Wang, K. Han, S. Ma and W. Gao, “Post-training quantization for vision transformer,” NeurIPS, 2021. arXiv
  57. X. Li, Y. Liu, L. Lian, H. Yang, Z. Dong, D. Kang, S. Zhang and K. Keutzer, “Q-Diffusion: quantizing diffusion models,” ICCV, 2023. doi arXiv
  58. Y. Shang, Z. Yuan, B. Xie, B. Wu and Y. Yan, “Post-training quantization on diffusion models,” CVPR, 2023. arXiv
  59. A. Kirillov et al., “Segment anything,” ICCV, 2023. doi arXiv
  60. C. Zhang, D. Han, Y. Qiao, J. U. Kim, S.-H. Bae, S. Lee and C. S. Hong, “Faster segment anything: towards lightweight SAM for mobile applications,” arXiv:2306.14289, 2023. arXiv
  61. Y. Xiong et al., “EfficientSAM: leveraged masked image pretraining for efficient segment anything,” CVPR, 2024. doi arXiv
  62. Y. Cheng, D. Wang, P. Zhou and T. Zhang, “Model compression and acceleration for deep neural networks: the principles, progress, and challenges,” IEEE Signal Processing Magazine, vol. 35, no. 1, pp. 126–136, 2018. doi arXiv
  63. G. Menghani, “Efficient deep learning: a survey on making deep learning models smaller, faster, and better,” ACM Computing Surveys, vol. 55, no. 12, pp. 1–37, 2023. doi arXiv
  64. A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney and K. Keutzer, “A survey of quantization methods for efficient neural network inference,” book chapter in Low-Power Computer Vision; arXiv:2103.13630, 2021. arXiv
  65. H. Cheng, M. Zhang and J. Q. Shi, “A survey on deep neural network pruning: taxonomy, comparison, analysis, and recommendations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 10558–10578, 2024. doi arXiv
  66. L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: a review and new outlooks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 6, pp. 3048–3068, 2022. doi arXiv
  67. C. White, M. Safari, R. Sukthanker, B. Ru, T. Elsken, A. Zela, D. Dey and F. Hutter, “Neural architecture search: insights from 1000 papers,” arXiv:2301.08727, 2023. arXiv
  68. E. Frantar, S. Ashkboos, T. Hoefler and D. Alistarh, “GPTQ: accurate post-training quantization for generative pre-trained transformers,” ICLR, 2023. arXiv
  69. J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan and S. Han, “AWQ: activation-aware weight quantization for LLM compression and acceleration,” MLSys, 2024. arXiv
  70. M. Li, Y. Lin, Z. Zhang, T. Cai, X. Li, J. Guo, E. Xie, C. Meng, J.-Y. Zhu and S. Han, “SVDQuant: absorbing outliers by low-rank components for 4-bit diffusion models,” ICLR, 2025. arXiv
  71. Y. Li, R. Gong, X. Tan, Y. Yang, P. Hu, Q. Zhang, F. Yu, W. Wang and S. Gu, “BRECQ: pushing the limit of post-training quantization by block reconstruction,” ICLR, 2021. arXiv
  72. S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy and D. S. Modha, “Learned step size quantization,” ICLR, 2020. arXiv