BasicVSR 與 IconVSR
選讀・DLCV 09A.3・影片超解析度:用時間找回細節 的論文導讀(第 1/2 篇)。引用編號對應本頁底部的參考文獻。
導覽:本頁與本系列: 導覽:本頁與本系列
挑戰
到了 2020 年,VSR 方法已變得複雜而且彼此差異很大(EDVR 的多尺度可變形對齊加注意力、RBPN 的多個投影模組),這讓它們很慢、難以重新實作,也很難公平比較。BasicVSR [7] 問的是:哪些元件真正重要?

圖片來源: [7], Published paper PDF, PDF 第 3 頁. 原始 PDF.
方法
把 VSR 方法分解成四個功能:傳播、對齊、聚合與上取樣,並用最精簡的設計測試每個功能的各種選項。論文的表 1 就是這樣分類現有方法的(局部、單向或雙向傳播;無對齊、光流或可變形對齊;以串接為基礎的聚合;pixel shuffle 上取樣)。BasicVSR 本身是效果最好的最簡單組合:雙向傳播、以光流(SPyNet)對特徵做對齊、串接與 pixel shuffle。
結果
消融實驗是這篇論文的核心,這裡的數字是論文自己的。把測試序列切成獨立片段會降低 PSNR,在片段末端降得最多,這支持長距離傳播(它的圖 3)。單向版本在早期影格明顯較差,整體而言比雙向模型差約 0.5 dB(圖 4)。移除對齊損失 1.19 dB,而扭曲影像而不是特徵損失 0.17 dB。延伸版 IconVSR 加入關鍵影格的資訊補充與耦合傳播以減少誤差累積,回報比 BasicVSR 最多高 0.31 dB。
判讀證據
從表 1 與 3.1 節讀起,兩者合起來就是這個領域的精簡地圖。在主要的比較表中,要檢查:(a) 每一欄的退化(REDS4 只有 BI,UDM10 只有 BD,Vimeo-90K-T 與 Vid4 兩者都有);(b) 通道(除了用 RGB 的 REDS4 之外,都用 Y);(c) 每一欄背後的訓練集(REDS4 用 REDS,其餘用 Vimeo-90K);(d) 執行時間,它取決於硬體與 LR 大小。在消融實驗中,要注意每一個都是修改 BasicVSR 的某個變體,所以下降的幅度衡量的是該元件在這個設計內的價值,而不是一般情況下的價值。
限制與問題
雙向傳播需要整段序列,所以 BasicVSR 是離線方法。以光流為基礎的對齊會承襲大運動與遮擋下的光流誤差,BasicVSR++ 用光流引導的可變形對齊來處理這一點 [20]。這項研究是在乾淨的 BI/BD 退化上進行的;RealBasicVSR [27] 顯示,同樣的長距離傳播可能放大嚴重的真實世界瑕疵。
結論
它提供了本篇通篇使用的詞彙,也是 BasicVSR++、RealBasicVSR 與數個擴散式 VSR 模型所建立的基線。
參考文獻
- S. Farsiu, M. D. Robinson, M. Elad and P. Milanfar, “Fast and Robust Multiframe Super Resolution,” IEEE Transactions on Image Processing, vol. 13, no. 10, pp. 1327–1344, 2004. doi
- J. Caballero, C. Ledig, A. Aitken, A. Acosta, J. Totz, Z. Wang and W. Shi, “Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation,” in Proc. IEEE CVPR, 2017. CVF · arXiv
- M. S. M. Sajjadi, R. Vemulapalli and M. Brown, “Frame-Recurrent Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- M. Irani and S. Peleg, “Improving Resolution by Image Registration,” CVGIP: Graphical Models and Image Processing, vol. 53, no. 3, pp. 231–239, 1991. doi
- M. Haris, G. Shakhnarovich and N. Ukita, “Recurrent Back-Projection Network for Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2019. CVF · arXiv
- M. Guizar-Sicairos, S. T. Thurman and J. R. Fienup, “Efficient Subpixel Image Registration Algorithms,” Optics Letters, vol. 33, no. 2, p. 156, 2008. doi
- K. C. K. Chan, X. Wang, K. Yu, C. Dong and C. C. Loy, “BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond,” in Proc. IEEE/CVF CVPR, 2021. CVF · arXiv
- B. K. P. Horn and B. G. Schunck, “Determining Optical Flow,” Artificial Intelligence, vol. 17, no. 1–3, pp. 185–203, 1981. doi
- G. Farnebäck, “Two-Frame Motion Estimation Based on Polynomial Expansion,” in Image Analysis (Lecture Notes in Computer Science), pp. 363–370, Springer, 2003. doi
- A. Ranjan and M. J. Black, “Optical Flow Estimation Using a Spatial Pyramid Network,” in Proc. IEEE CVPR, 2017. CVF · arXiv
- Z. Teed and J. Deng, “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 402–419, 2020. doi · arXiv
- M. Jaderberg, K. Simonyan, A. Zisserman and K. Kavukcuoglu, “Spatial Transformer Networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2015. proceedings · arXiv
- W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert and Z. Wang, “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in Proc. IEEE CVPR, 2016. arXiv
- T. Xue, B. Chen, J. Wu, D. Wei and W. T. Freeman, “Video Enhancement with Task-Oriented Flow,” International Journal of Computer Vision, vol. 127, no. 8, pp. 1106–1125, 2019. doi · arXiv
- J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu and Y. Wei, “Deformable Convolutional Networks,” in Proc. IEEE ICCV, 2017. CVF · arXiv
- X. Zhu, H. Hu, S. Lin and J. Dai, “Deformable ConvNets V2: More Deformable, Better Results,” in Proc. IEEE/CVF CVPR, 2019. CVF · arXiv
- Y. Tian, Y. Zhang, Y. Fu and C. Xu, “TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2020. CVF · arXiv
- X. Wang, K. C. K. Chan, K. Yu, C. Dong and C. C. Loy, “EDVR: Video Restoration with Enhanced Deformable Convolutional Networks,” in Proc. IEEE/CVF CVPR Workshops, 2019. CVF · arXiv
- K. C. K. Chan, X. Wang, K. Yu, C. Dong and C. C. Loy, “Understanding Deformable Alignment in Video Super-Resolution,” in Proc. AAAI Conference on Artificial Intelligence, vol. 35, no. 2, pp. 973–981, 2021. doi · arXiv
- K. C. K. Chan, S. Zhou, X. Xu and C. C. Loy, “BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
- Y. Jo, S. W. Oh, J. Kang and S. J. Kim, “Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation,” in Proc. IEEE/CVF CVPR, 2018. CVF
- P. Yi, Z. Wang, K. Jiang, J. Jiang and J. Ma, “Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations,” in Proc. IEEE/CVF ICCV, 2019. CVF · code and test data (Vid4, UDM10)
- J. Liang, J. Cao, Y. Fan, K. Zhang, R. Ranjan, Y. Li, R. Timofte and L. Van Gool, “VRT: A Video Restoration Transformer,” IEEE Transactions on Image Processing, vol. 33, pp. 2171–2182, 2024. doi · arXiv
- J. Liang, Y. Fan, X. Xiang, R. Ranjan, E. Ilg, S. Green, J. Cao, K. Zhang, R. Timofte and L. Van Gool, “Recurrent Video Restoration Transformer with Guided Deformable Attention,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 378–393, 2022. arXiv
- T. Wiegand, G. J. Sullivan, G. Bjøntegaard and A. Luthra, “Overview of the H.264/AVC Video Coding Standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560–576, 2003. doi
- Y. Li, P. Jin, F. Yang, C. Liu, M.-H. Yang and P. Milanfar, “COMISR: Compression-Informed Video Super-Resolution,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
- K. C. K. Chan, S. Zhou, X. Xu and C. C. Loy, “Investigating Tradeoffs in Real-World Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
- S. Zhou, P. Yang, J. Wang, Y. Luo and C. C. Loy, “Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution,” in Proc. IEEE/CVF CVPR, pp. 2535–2545, 2024. CVF · arXiv
- X. Yang, C. He, J. Ma and L. Zhang, “Motion-Guided Latent Diffusion for Temporally Consistent Real-World Video Super-Resolution,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 224–242, 2024. doi · arXiv
- R. Xie, Y. Liu, P. Zhou, C. Zhao, J. Zhou, K. Zhang, Z. Zhang, J. Yang, Z. Yang and Y. Tai, “STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution,” in Proc. IEEE/CVF ICCV, 2025. CVF · arXiv
- J. Wang, Z. Lin, M. Wei, Y. Zhao, C. Yang, C. C. Loy and L. Jiang, “SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
- Y. Sun, L. Sun, S. Liu, R. Wu, Z. Zhang and L. Zhang, “One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 38, 2025. arXiv
- W.-S. Lai, J.-B. Huang, O. Wang, E. Shechtman, E. Yumer and M.-H. Yang, “Learning Blind Video Temporal Consistency,” in Proc. ECCV, 2018. CVF · arXiv
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- ITU-R, “Recommendation ITU-R BT.601-7: Studio Encoding Parameters of Digital Television for Standard 4:3 and Wide-Screen 16:9 Aspect Ratios,” International Telecommunication Union, 2011. ITU
- M. Chu, Y. Xie, J. Mayer, L. Leal-Taixé and N. Thuerey, “Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation,” ACM Transactions on Graphics, vol. 39, no. 4, article 75, 2020. doi · arXiv
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
- S. Nah, S. Baik, S. Hong, G. Moon, S. Son, R. Timofte and K. M. Lee, “NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study,” in Proc. IEEE/CVF CVPR Workshops, 2019. CVF
- C. Liu and D. Sun, “On Bayesian Adaptive Video Super Resolution,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 2, pp. 346–360, 2014. doi
- H. Liu, Z. Ruan, P. Zhao, C. Dong, F. Shang, Y. Liu, L. Yang and R. Timofte, “Video Super-Resolution Based on Deep Learning: A Comprehensive Survey,” Artificial Intelligence Review, vol. 55, no. 8, pp. 5981–6035, 2022. doi · arXiv
- S. Nah, R. Timofte, S. Gu, S. Baik, S. Hong, G. Moon, S. Son and K. M. Lee, “NTIRE 2019 Challenge on Video Super-Resolution: Methods and Results,” in Proc. IEEE/CVF CVPR Workshops, 2019. CVF
- S. Son, S. Lee, S. Nah, R. Timofte and K. M. Lee, “NTIRE 2021 Challenge on Video Super-Resolution,” in Proc. IEEE/CVF CVPR Workshops, 2021. CVF · arXiv
- A. Ignatov, R. Timofte et al., “Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report,” arXiv:2211.05256, 2022. arXiv
- M. V. Conde, Z. Lei, W. Li, C. Bampis, I. Katsavounidis, R. Timofte et al., “AIM 2024 Challenge on Efficient Video Super-Resolution for AV1 Compressed Content,” in Proc. ECCV Workshops (Lecture Notes in Computer Science), pp. 304–325, 2025. doi · arXiv
- Z. Tu, H. Li, W. Xie, Y. Liu, S. Zhang, B. Li and J. Yuan, “Optical Flow for Video Super-Resolution: A Survey,” Artificial Intelligence Review, vol. 55, no. 8, pp. 6505–6546, 2022. doi · arXiv
- A. A. Baniya, T.-K. Lee, P. W. Eklund and S. Aryal, “A Survey of Deep Learning Video Super-Resolution,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 8, no. 4, pp. 2655–2676, 2024. doi · arXiv
- B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.-K. Liang, M. Levoy and P. Milanfar, “Handheld Multi-Frame Super-Resolution,” ACM Transactions on Graphics, vol. 38, no. 4, 2019. doi · arXiv