完整搜尋頁面 →

PiSA-SR:分開像素與語意調適(CVPR 2025)

選讀・DLCV 09A.2・單步擴散超解析度 的論文導讀(第 7/8 篇)。引用編號對應本頁底部的參考文獻。

導覽:本頁與本系列: 導覽:本頁與本系列

本頁內容

補充資料 · 2

  1. 潛在起點不等於加雜訊方程式
  2. 分數正規化與逐圖量測一致性

論文導讀 · 8

  1. ResShift:復原條件的短鏈(NeurIPS 2023)
  2. SinSR:蒸餾確定性的影像條件映射(CVPR 2024)
  3. OSEDiff:LR 潛在起點與變分分數正規化(NeurIPS 2024)
  4. S3Diff:退化引導的低秩調適
  5. TSD-SR:目標感知分數正規化(CVPR 2025)
  6. InvSR:學得的影像條件起點(CVPR 2025)
  7. PiSA-SR:分開像素與語意調適(CVPR 2025)
  8. AddSR:預測細化的對抗復原蒸餾

電腦視覺深度學習

  1. 02 · 深度影像分類
  2. 03 · 物件偵測:從 R-CNN 到 DETR
  3. 09 · 深度影像復原:低階視覺
查看完整課程樹 →

挑戰

像素重建與生成細節常在同一調適中競爭。PiSA-SR [8] 分成兩組低秩更新,讓使用者不用每種偏好都重訓,也能調節保真與外觀。

它們是學得的控制方向,不是影像真實/虛構的物理拆解。語意增強仍可能改字,像素模型也可能保留模糊,不能只靠控制名稱判定效果。

方法

像素與語意低秩支路接受不同訓練訊號,推論時可調組合呈現忠實度與感知品質取捨。
原論文圖 3 — 像素與語意低秩支路接受不同訓練訊號,推論時可調組合呈現忠實度與感知品質取捨。

圖片來源: [8], arXiv:2412.03017v2, PDF 第 5 頁. 原始 PDF.

潛在殘差。 固定自動編碼器把放大 LR 編為 zLz_L;簡化固定時間式為 z^H=zL−ϵθ(zL)\widehat z_H=z_L-\epsilon_\theta(z_L)。這是調適的影像殘差,不主張 zLz_L 真含該時間的獨立高斯雜訊。

像素轉接器。 對凍結去雜訊器加入 Δθpix\Delta\theta_{\mathrm{pix}},以 L2 訓練成 θpix\theta_{\mathrm{pix}},專注參考對齊結構。編碼器及解碼器均固定。

語意轉接器。 固定已學像素更新,再加 Δθsem\Delta\theta_{\mathrm{sem}},以 LPIPS 和分類器分數蒸餾訓練成 θPiSA\theta_{\mathrm{PiSA}}。兩階段依序訓練,語意階段不能任意重寫像素轉接器。

語意先驗。 固定文字擴散的有條件/無條件預測差,提供無分類器引導導出的方向 [36]。它不訓練 OSEDiff 式生成分布分數模型,減少輔助最佳化,也改變統計目標,兩種正規化不完全相同。

預設推論。 合併兩組轉接器,做一次殘差預測,不必線上執行訓練提示擷取器。固定編解碼仍要執行,大型骨幹也仍計入總參數。

可調推論。 分別評估像素模型與合併模型,再形成:

ϵadj=λpixϵθpix(zL)+λsem(ϵθPiSA(zL)−ϵθpix(zL)).\epsilon_{\mathrm{adj}}=\lambda_{\mathrm{pix}}\epsilon_{\theta_{\mathrm{pix}}}(z_L) +\lambda_{\mathrm{sem}}\bigl(\epsilon_{\theta_{\mathrm{PiSA}}}(z_L)-\epsilon_{\theta_{\mathrm{pix}}}(z_L)\bigr).

λpix\lambda_{\mathrm{pix}} 控制像素修正,λsem\lambda_{\mathrm{sem}} 控制語意調適帶來的差異。此版需要兩次去雜訊預測,成本高於預設;非線性網路下,內插轉接器權重也不等價於此輸出差構造。

資料。 LSDIR 加 FFHQ 前 10,000 張 [32]、[33],Real-ESRGAN 退化 [27],選定秩 4。訓練語意條件被蒸餾入更新,不表示部署需要描述器。

結果

預設 RealSR 的 PSNR/LPIPS/CLIPIQA/MUSIQ 為 25.50/0.2672/0.6702/70.15。固定像素強度 1,語意 0.2 的 PSNR/MUSIQ 26.95/53.64,語意 1.2 為 24.59/71.60。這是偏好取捨,不是新增細節的獨立真實性證據。

A100 80 GB、×4、128 × 128 輸入,預設 0.09 秒/一次預測,可調 0.13 秒/兩次。介面不能把所有可調設定都宣傳成預設單次延遲。

消融實驗

像素單獨/語意單獨/兩階段的 RealSR PSNR 27.28/24.13/25.50,LPIPS 0.3090/0.2803/0.2672,MUSIQ 49.02/70.69/70.15。這些是各自訓練消融,不都等於最後模型直接把係數歸零。

語意秩 4→32,PSNR 25.50→25.40,LPIPS 0.2672→0.2713,MUSIQ 相近。容量更多不保證更好,控制掃描的最佳感知值也可能落在中間強度。

結論

分開訓練目標提供可用增強方向;合併預設快,輸出差控制較彈性但多花推論成本。

限制與問題

像素/語意未必完全解耦,固定像素轉接器不一定涵蓋所有損傷。強細節設定可改掉內容,可調版也較貴並繼承先驗偏差。應評估每個操作點,不能用一個分數保證整段控制範圍。

參考文獻

  1. R. Wu, L. Sun, Z. Ma, L. Zhang, “One-Step Effective Diffusion Network for Real-World Image Super-Resolution,” NeurIPS, 2024. source
  2. Z. Yue, J. Wang, C. C. Loy, “ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting,” NeurIPS, 2023. source
  3. Y. Wang, W. Yang, X. Chen, et al., “SinSR: Diffusion-Based Image Super-Resolution in a Single Step,” CVPR, 2024. source · supplementary ablations
  4. A. Zhang, Z. Yue, R. Pei, W. Ren, X. Cao, “Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors,” arXiv preprint, 2024. paper
  5. Y. Tai, R. Xie, C. Zhao, K. Zhang, Z. Zhang, J. Zhou, J. Yang, “AddSR: Accelerating diffusion-based blind super-resolution with adversarial diffusion distillation,” Pattern Recognition, 2026. published version · manuscript supplying the discussed tables
  6. L. Dong, Q. Fan, Y. Guo, Z. Wang, Q. Zhang, J. Chen, Y. Luo, C. Zou, “TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution,” CVPR, 2025. paper
  7. Z. Yue, K. Liao, C. C. Loy, “Arbitrary-steps Image Super-resolution via Diffusion Inversion,” CVPR, 2025. paper
  8. L. Sun, R. Wu, Z. Ma, S. Liu, Q. Yi, L. Zhang, “Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach,” CVPR, 2025. paper
  9. A. Liu, Y. Liu, J. Gu, Y. Qiao, C. Dong, “Blind Image Super-Resolution: A Survey and Beyond,” IEEE TPAMI (early access), 2022. source
  10. B. B. Moser, A. S. Shanbhag, F. Raue, S. Frolov, S. Palacio and A. Dengel, “Diffusion Models, Image Super-Resolution, and Everything: A Survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 7, pp. 11793–11813, 2025. doi · arXiv
  11. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. source
  12. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  13. Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, J. Zhu, “ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation,” NeurIPS, 2023. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. J. Cai, H. Zeng, H. Yong, Z. Cao, L. Zhang, “Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. source
  16. P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, L. Lin, “Component Divide-and-Conquer for Real-World Image Super-Resolution,” ECCV, 2020. source
  17. E. Agustsson and R. Timofte, “NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study,” in Proc. IEEE CVPR Workshops, 2017, pp. 1122–1131. doi · CVF
  18. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  19. K. Ding, K. Ma, S. Wang, E. P. Simoncelli, “Image Quality Assessment: Unifying Structure and Texture Similarity,” IEEE TPAMI, 2022. source
  20. A. Mittal, R. Soundararajan and A. C. Bovik, “Making a “Completely Blind” Image Quality Analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. doi
  21. J. Ke, Q. Wang, Y. Wang, P. Milanfar, F. Yang, “MUSIQ: Multi-scale Image Quality Transformer,” ICCV, 2021. source
  22. S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, Y. Yang, “MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment,” CVPR Workshops, 2022. source
  23. J. Wang, K. C. K. Chan, C. C. Loy, “Exploring CLIP for Assessing the Look and Feel of Images,” AAAI, 2023. source
  24. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  25. Z. Wu, Y. Dong, X. Wei, “TUDSR: Twice Upsampling-Diffusion for Higher Super-Resolution,” CVPR, 2026. paper
  26. P. Esser, R. Rombach, B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” CVPR, 2021. paper
  27. X. Wang, L. Xie, C. Dong, Y. Shan, “Real-ESRGAN: Training Real-World Blind Super-Resolution With Pure Synthetic Data,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. source
  28. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  29. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  30. R. Wu, T. Yang, L. Sun, Z. Zhang, S. Li, L. Zhang, “SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. source · supplementary experiments
  31. H. Liu, C. Li, Q. Wu, Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. source
  32. Y. Li, K. Zhang, J. Liang, et al., “LSDIR: A Large Scale Dataset for Image Restoration,” CVPR Workshops, 2023. paper
  33. T. Karras, S. Laine, T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. paper
  34. I. Loshchilov, F. Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. paper
  35. P. Esser, S. Kulal, A. Blattmann, et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, 2024. paper
  36. J. Ho, T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv preprint, 2022. paper
  37. R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang et al., “NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results,” in Proc. IEEE CVPR Workshops, 2017, pp. 1110–1121. doi · CVF