Full search page →

RealBasicVSR

Optional · paper guide 2 of 2 for DLCV 09A.3 · Video Super-Resolution: Using Time to Recover Detail. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 4

  1. Classical multi-frame super-resolution
  2. Brightness constancy and the classical flow solvers
  3. Deformable convolution as warping plus convolution
  4. Diffusion-based VSR models

Paper guides · 2

  1. BasicVSR and IconVSR
  2. RealBasicVSR

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

Models trained on BI or BD degradations fail on real videos, and simply training BasicVSR-style models on random real-world degradations raises new trade-offs that earlier work ignored [27].

Clean each frame before bidirectional propagation so corruption is not repeatedly carried through time; dynamic refinement can repeat the cleaning stage.
Original paper figure 3 — Clean each frame before bidirectional propagation so corruption is not repeatedly carried through time; dynamic refinement can repeat the cleaning stage.

Figure source: [27], Published paper PDF, PDF page 4. Original PDF.

Method

The paper identifies three trade-offs and proposes a simple fix for each. (1) Detail synthesis versus artifact suppression: long-term propagation spreads noise and artifacts along with detail, so an image pre-cleaning module (a stack of residual blocks) cleans each LR frame independently before a BasicVSR network propagates them. It is trained with an extra cleaning loss that compares its output with a downsampled ground truth, and at test time a dynamic refinement scheme applies it repeatedly until the change falls below a threshold. The paper shows that a recurrent cleaning module, or no cleaning loss, fails to remove the artifacts. (2) Speed versus performance in training: loading many long, degraded clips makes training slow, so a stochastic degradation scheme loads half of each sequence and flips it in time, and lets the degradation parameters change from frame to frame as a random walk to keep the data diverse; this reduces training time by up to 40% without loss of quality. (3) Batch versus length: under a fixed memory budget, longer sequences use temporal information more effectively than larger batches and make inference more stable.

Results

RealBasicVSR, a BasicVSR with the cleaning module, fine-tuned with perceptual and adversarial losses (with the cleaning module detached from those two losses), compares favorably in quality and efficiency with real-world image and video SR models on real videos. The paper also releases the VideoLQ dataset of real low-quality videos as a common benchmark.

Reading the evidence

Begin with the analysis of the cleaning module (what happens when propagation is applied to uncleaned inputs) and the batch-versus-length experiments, since they explain the design. For VideoLQ, check that the comparison uses no-reference metrics and visual results because there is no ground truth, and read the visual comparisons for the trade-off between sharper texture and invented artifacts.

Problems

No-reference metrics on VideoLQ are only weakly tied to human judgment, so the comparison relies partly on visual examples. The degradation model is synthetic, so codecs and camera pipelines outside it can still break the model. A cleaning module that removes too much also removes real detail.

Conclusion

It explains why real-world VSR cannot just reuse the clean-benchmark recipe, and it is the standard non-diffusion baseline that diffusion VSR papers such as Upscale-A-Video compare with [28].

References

  1. S. Farsiu, M. D. Robinson, M. Elad and P. Milanfar, “Fast and Robust Multiframe Super Resolution,” IEEE Transactions on Image Processing, vol. 13, no. 10, pp. 1327–1344, 2004. doi
  2. J. Caballero, C. Ledig, A. Aitken, A. Acosta, J. Totz, Z. Wang and W. Shi, “Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation,” in Proc. IEEE CVPR, 2017. CVF · arXiv
  3. M. S. M. Sajjadi, R. Vemulapalli and M. Brown, “Frame-Recurrent Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
  4. M. Irani and S. Peleg, “Improving Resolution by Image Registration,” CVGIP: Graphical Models and Image Processing, vol. 53, no. 3, pp. 231–239, 1991. doi
  5. M. Haris, G. Shakhnarovich and N. Ukita, “Recurrent Back-Projection Network for Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2019. CVF · arXiv
  6. M. Guizar-Sicairos, S. T. Thurman and J. R. Fienup, “Efficient Subpixel Image Registration Algorithms,” Optics Letters, vol. 33, no. 2, p. 156, 2008. doi
  7. K. C. K. Chan, X. Wang, K. Yu, C. Dong and C. C. Loy, “BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond,” in Proc. IEEE/CVF CVPR, 2021. CVF · arXiv
  8. B. K. P. Horn and B. G. Schunck, “Determining Optical Flow,” Artificial Intelligence, vol. 17, no. 1–3, pp. 185–203, 1981. doi
  9. G. Farnebäck, “Two-Frame Motion Estimation Based on Polynomial Expansion,” in Image Analysis (Lecture Notes in Computer Science), pp. 363–370, Springer, 2003. doi
  10. A. Ranjan and M. J. Black, “Optical Flow Estimation Using a Spatial Pyramid Network,” in Proc. IEEE CVPR, 2017. CVF · arXiv
  11. Z. Teed and J. Deng, “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 402–419, 2020. doi · arXiv
  12. M. Jaderberg, K. Simonyan, A. Zisserman and K. Kavukcuoglu, “Spatial Transformer Networks,” in Advances in Neural Information Processing Systems (NeurIPS), 2015. proceedings · arXiv
  13. W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert and Z. Wang, “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” in Proc. IEEE CVPR, 2016. arXiv
  14. T. Xue, B. Chen, J. Wu, D. Wei and W. T. Freeman, “Video Enhancement with Task-Oriented Flow,” International Journal of Computer Vision, vol. 127, no. 8, pp. 1106–1125, 2019. doi · arXiv
  15. J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu and Y. Wei, “Deformable Convolutional Networks,” in Proc. IEEE ICCV, 2017. CVF · arXiv
  16. X. Zhu, H. Hu, S. Lin and J. Dai, “Deformable ConvNets V2: More Deformable, Better Results,” in Proc. IEEE/CVF CVPR, 2019. CVF · arXiv
  17. Y. Tian, Y. Zhang, Y. Fu and C. Xu, “TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2020. CVF · arXiv
  18. X. Wang, K. C. K. Chan, K. Yu, C. Dong and C. C. Loy, “EDVR: Video Restoration with Enhanced Deformable Convolutional Networks,” in Proc. IEEE/CVF CVPR Workshops, 2019. CVF · arXiv
  19. K. C. K. Chan, X. Wang, K. Yu, C. Dong and C. C. Loy, “Understanding Deformable Alignment in Video Super-Resolution,” in Proc. AAAI Conference on Artificial Intelligence, vol. 35, no. 2, pp. 973–981, 2021. doi · arXiv
  20. K. C. K. Chan, S. Zhou, X. Xu and C. C. Loy, “BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
  21. Y. Jo, S. W. Oh, J. Kang and S. J. Kim, “Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation,” in Proc. IEEE/CVF CVPR, 2018. CVF
  22. P. Yi, Z. Wang, K. Jiang, J. Jiang and J. Ma, “Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations,” in Proc. IEEE/CVF ICCV, 2019. CVF · code and test data (Vid4, UDM10)
  23. J. Liang, J. Cao, Y. Fan, K. Zhang, R. Ranjan, Y. Li, R. Timofte and L. Van Gool, “VRT: A Video Restoration Transformer,” IEEE Transactions on Image Processing, vol. 33, pp. 2171–2182, 2024. doi · arXiv
  24. J. Liang, Y. Fan, X. Xiang, R. Ranjan, E. Ilg, S. Green, J. Cao, K. Zhang, R. Timofte and L. Van Gool, “Recurrent Video Restoration Transformer with Guided Deformable Attention,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 378–393, 2022. arXiv
  25. T. Wiegand, G. J. Sullivan, G. Bjøntegaard and A. Luthra, “Overview of the H.264/AVC Video Coding Standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560–576, 2003. doi
  26. Y. Li, P. Jin, F. Yang, C. Liu, M.-H. Yang and P. Milanfar, “COMISR: Compression-Informed Video Super-Resolution,” in Proc. IEEE/CVF ICCV, 2021. CVF · arXiv
  27. K. C. K. Chan, S. Zhou, X. Xu and C. C. Loy, “Investigating Tradeoffs in Real-World Video Super-Resolution,” in Proc. IEEE/CVF CVPR, 2022. CVF · arXiv
  28. S. Zhou, P. Yang, J. Wang, Y. Luo and C. C. Loy, “Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution,” in Proc. IEEE/CVF CVPR, pp. 2535–2545, 2024. CVF · arXiv
  29. X. Yang, C. He, J. Ma and L. Zhang, “Motion-Guided Latent Diffusion for Temporally Consistent Real-World Video Super-Resolution,” in Proc. ECCV (Lecture Notes in Computer Science), pp. 224–242, 2024. doi · arXiv
  30. R. Xie, Y. Liu, P. Zhou, C. Zhao, J. Zhou, K. Zhang, Z. Zhang, J. Yang, Z. Yang and Y. Tai, “STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution,” in Proc. IEEE/CVF ICCV, 2025. CVF · arXiv
  31. J. Wang, Z. Lin, M. Wei, Y. Zhao, C. Yang, C. C. Loy and L. Jiang, “SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration,” in Proc. IEEE/CVF CVPR, 2025. CVF · arXiv
  32. Y. Sun, L. Sun, S. Liu, R. Wu, Z. Zhang and L. Zhang, “One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 38, 2025. arXiv
  33. W.-S. Lai, J.-B. Huang, O. Wang, E. Shechtman, E. Yumer and M.-H. Yang, “Learning Blind Video Temporal Consistency,” in Proc. ECCV, 2018. CVF · arXiv
  34. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  35. ITU-R, “Recommendation ITU-R BT.601-7: Studio Encoding Parameters of Digital Television for Standard 4:3 and Wide-Screen 16:9 Aspect Ratios,” International Telecommunication Union, 2011. ITU
  36. M. Chu, Y. Xie, J. Mayer, L. Leal-Taixé and N. Thuerey, “Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation,” ACM Transactions on Graphics, vol. 39, no. 4, article 75, 2020. doi · arXiv
  37. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018. CVF · arXiv
  38. S. Nah, S. Baik, S. Hong, G. Moon, S. Son, R. Timofte and K. M. Lee, “NTIRE 2019 Challenge on Video Deblurring and Super-Resolution: Dataset and Study,” in Proc. IEEE/CVF CVPR Workshops, 2019. CVF
  39. C. Liu and D. Sun, “On Bayesian Adaptive Video Super Resolution,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 36, no. 2, pp. 346–360, 2014. doi
  40. H. Liu, Z. Ruan, P. Zhao, C. Dong, F. Shang, Y. Liu, L. Yang and R. Timofte, “Video Super-Resolution Based on Deep Learning: A Comprehensive Survey,” Artificial Intelligence Review, vol. 55, no. 8, pp. 5981–6035, 2022. doi · arXiv
  41. S. Nah, R. Timofte, S. Gu, S. Baik, S. Hong, G. Moon, S. Son and K. M. Lee, “NTIRE 2019 Challenge on Video Super-Resolution: Methods and Results,” in Proc. IEEE/CVF CVPR Workshops, 2019. CVF
  42. S. Son, S. Lee, S. Nah, R. Timofte and K. M. Lee, “NTIRE 2021 Challenge on Video Super-Resolution,” in Proc. IEEE/CVF CVPR Workshops, 2021. CVF · arXiv
  43. A. Ignatov, R. Timofte et al., “Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report,” arXiv:2211.05256, 2022. arXiv
  44. M. V. Conde, Z. Lei, W. Li, C. Bampis, I. Katsavounidis, R. Timofte et al., “AIM 2024 Challenge on Efficient Video Super-Resolution for AV1 Compressed Content,” in Proc. ECCV Workshops (Lecture Notes in Computer Science), pp. 304–325, 2025. doi · arXiv
  45. Z. Tu, H. Li, W. Xie, Y. Liu, S. Zhang, B. Li and J. Yuan, “Optical Flow for Video Super-Resolution: A Survey,” Artificial Intelligence Review, vol. 55, no. 8, pp. 6505–6546, 2022. doi · arXiv
  46. A. A. Baniya, T.-K. Lee, P. W. Eklund and S. Aryal, “A Survey of Deep Learning Video Super-Resolution,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 8, no. 4, pp. 2655–2676, 2024. doi · arXiv
  47. B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.-K. Liang, M. Levoy and P. Milanfar, “Handheld Multi-Frame Super-Resolution,” ACM Transactions on Graphics, vol. 38, no. 4, 2019. doi · arXiv