Full search page →

ResShift: a short restoration-conditioned chain (NeurIPS 2023)

Optional · paper guide 1 of 8 for DLCV 09A.2 · One-Step Diffusion Super-Resolution. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Why a latent start is not a noising equation
  2. Score regularization and per-image measurement agreement

Paper guides · 8

  1. ResShift: a short restoration-conditioned chain (NeurIPS 2023)
  2. SinSR: distilling a deterministic image-conditioned map (CVPR 2024)
  3. OSEDiff: LR-latent start and variational score regularization (NeurIPS 2024)
  4. S3Diff: degradation-guided low-rank adaptation
  5. TSD-SR: target-aware score regularization (CVPR 2025)
  6. InvSR: a learned image-conditioned start (CVPR 2025)
  7. PiSA-SR: separate pixel and semantic adaptation (CVPR 2025)
  8. AddSR: prediction-refined adversarial restoration distillation

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

A diffusion restorer need not reconstruct the complete scene from pure noise when an LR image already supplies much of its structure. ResShift builds a forward process that gradually moves an HR target toward the enlarged LR observation [2]. Reversing this process starts near the observation, making a shorter chain plausible.

The contribution is a restoration-specific stochastic process and schedule, not distillation of a large text-to-image teacher. Its shorter chain and smaller task-specific model must therefore be evaluated together with their training data and representation. Comparing it with a pretrained text-to-image prior does not isolate only the number of steps.

Method

The forward chain shifts clean content toward the observed low-resolution condition while adding noise; reversing that chain is different from unconditional generation from pure noise.
Original paper figure 2 — The forward chain shifts clean content toward the observed low-resolution condition while adding noise; reversing that chain is different from unconditional generation from pure noise.

Figure source: [2], arXiv:2307.12348v3, PDF page 3. Original PDF.

Residual and forward transition. Let x0x_0 be HR, yy the enlarged LR observation, and e=y−x0e=y-x_0 their residual. Choose increasing shift fractions ηt\eta_t, with the endpoints near zero and one. Set δt=ηt−ηt−1\delta_t=\eta_t-\eta_{t-1}. A forward transition adds δte\delta_te plus Gaussian noise of covariance κ2δtI\kappa^2\delta_tI, where κ\kappa controls noise. Using δ\delta here avoids confusing this residual increment with the signal coefficient used by other diffusion papers.

Closed-form marginal. The summed shifts and Gaussian increments give

q(xt∣x0,y)=N(x0+ηt(y−x0),κ2ηtI).q(x_t\mid x_0,y)=\mathcal N(x_0+\eta_t(y-x_0),\kappa^2\eta_tI).

The state has the image/latent shape. At small ηt\eta_t, it is near HR; at large ηt\eta_t, its mean is near LR. Training can sample a state directly at any time without simulating the full forward chain. This makes the process tractable even though the original image is unknown during deployment.

Reverse target. The tractable posterior between adjacent forward states has a mean that combines xtx_t with the HR target. Replace that unknown target by a network prediction fθ(xt,y,t)f_\theta(x_t,y,t). One reverse mean is

μθ=ηt−1ηtxt+δtηtfθ(xt,y,t),\mu_\theta=\frac{\eta_{t-1}}{\eta_t}x_t+\frac{\delta_t}{\eta_t}f_\theta(x_t,y,t),

with covariance κ2ηt−1δtI/ηt\kappa^2\eta_{t-1}\delta_tI/\eta_t. Here fθf_\theta predicts HR from the current state, observation and time. The resulting training objective reduces to a weighted reconstruction loss; the published implementation drops the derived time weight after an empirical comparison. That practical simplification should be distinguished from the exact variational expression.

Latent domain. The process is implemented using an image autoencoder associated with VQGAN [26], compressing spatial dimensions by four in the paper’s setting. Both HR and enlarged LR images occupy compatible latent grids. The model learns its restoration prior from its training task; it does not obtain the same broad prior as freezing a large text-to-image network.

Schedule. A geometric-style schedule for ηt\sqrt{\eta_t} includes a shape parameter pp, while κ\kappa sets noise intensity. The published default uses 15 steps, p=0.3p=0.3 and κ=2\kappa=2. These are fitted design choices. Changing the process parameters can change what the network sees, so a schedule sweep should not automatically be interpreted as free inference-time control for one unchanged model.

Data and inference. Training uses synthetic degradations on image data, with a Real-ESRGAN-style observation model [27]. Inference begins approximately at y+κϵy+\kappa\epsilon and follows the learned reverse transitions. A smaller noise scale keeps states closer to the observation, but closeness alone cannot identify the missing high-frequency content or compensate for an incorrect degradation model. Randomness remains in the starting state and stochastic reverse updates. The decoder converts the resulting latent to pixels.

Results

On the paper’s ImageNet-Test protocol [28], Table 3 reports PSNR 25.01, SSIM 0.677, LPIPS 0.231, CLIPIQA 0.592 and MUSIQ 53.660 for the selected model. The comparison includes a general latent diffusion model shortened to the same 15-step budget, helping separate the benefit from merely choosing fewer steps. The results are still specific to that training and test setup.

The paper also evaluates real images with no-reference metrics and visual comparisons. Those results support practical appearance quality under the tested degradations; they do not prove exact reconstruction of missing scene details. No-reference scores are particularly important to read alongside the reference-based trade-offs in the schedule study.

Ablations

Table 1 varies step count while retaining the listed process parameters. At 10, 15 and 50 steps, PSNR is 25.20, 25.01 and 24.22, while LPIPS is 0.2517, 0.2312 and 0.2212. More steps improve this perceptual distance while reducing PSNR; “more computation improves quality” needs a metric attached.

At 15 steps and κ=2\kappa=2, changing pp from 0.3 to 1 changes LPIPS from 0.2312 to 0.2613 and MUSIQ from 53.6596 to 48.4964. A larger pp can suppress detail. Varying κ\kappa also changes the trade-off: at p=0.3p=0.3, κ=0.5\kappa=0.5 gives LPIPS 0.2437 and κ=8\kappa=8 gives 0.2592, versus 0.2312 at 2. These comparisons demonstrate schedule sensitivity, not a monotonic realism dial.

Conclusion

A chain centered on the observed LR image is a useful alternative to starting restoration from pure noise. The residual process supplies tractable training and reverse transitions, while the schedule determines a fidelity/appearance trade.

Problems

The method still needs repeated network calls and cannot identify unmeasured detail. Its task-specific prior can be less expressive than larger pretrained generative models. Synthetic degradation coverage and autoencoder reconstruction remain limitations. SinSR subsequently learns a deterministic endpoint of this conditional process in one student call [3].

References

  1. R. Wu, L. Sun, Z. Ma, L. Zhang, “One-Step Effective Diffusion Network for Real-World Image Super-Resolution,” NeurIPS, 2024. source
  2. Z. Yue, J. Wang, C. C. Loy, “ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting,” NeurIPS, 2023. source
  3. Y. Wang, W. Yang, X. Chen, et al., “SinSR: Diffusion-Based Image Super-Resolution in a Single Step,” CVPR, 2024. source · supplementary ablations
  4. A. Zhang, Z. Yue, R. Pei, W. Ren, X. Cao, “Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors,” arXiv preprint, 2024. paper
  5. Y. Tai, R. Xie, C. Zhao, K. Zhang, Z. Zhang, J. Zhou, J. Yang, “AddSR: Accelerating diffusion-based blind super-resolution with adversarial diffusion distillation,” Pattern Recognition, 2026. published version · manuscript supplying the discussed tables
  6. L. Dong, Q. Fan, Y. Guo, Z. Wang, Q. Zhang, J. Chen, Y. Luo, C. Zou, “TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution,” CVPR, 2025. paper
  7. Z. Yue, K. Liao, C. C. Loy, “Arbitrary-steps Image Super-resolution via Diffusion Inversion,” CVPR, 2025. paper
  8. L. Sun, R. Wu, Z. Ma, S. Liu, Q. Yi, L. Zhang, “Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach,” CVPR, 2025. paper
  9. A. Liu, Y. Liu, J. Gu, Y. Qiao, C. Dong, “Blind Image Super-Resolution: A Survey and Beyond,” IEEE TPAMI (early access), 2022. source
  10. B. B. Moser, A. S. Shanbhag, F. Raue, S. Frolov, S. Palacio and A. Dengel, “Diffusion Models, Image Super-Resolution, and Everything: A Survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 7, pp. 11793–11813, 2025. doi · arXiv
  11. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. source
  12. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  13. Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, J. Zhu, “ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation,” NeurIPS, 2023. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. J. Cai, H. Zeng, H. Yong, Z. Cao, L. Zhang, “Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. source
  16. P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, L. Lin, “Component Divide-and-Conquer for Real-World Image Super-Resolution,” ECCV, 2020. source
  17. E. Agustsson and R. Timofte, “NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study,” in Proc. IEEE CVPR Workshops, 2017, pp. 1122–1131. doi · CVF
  18. Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
  19. K. Ding, K. Ma, S. Wang, E. P. Simoncelli, “Image Quality Assessment: Unifying Structure and Texture Similarity,” IEEE TPAMI, 2022. source
  20. A. Mittal, R. Soundararajan and A. C. Bovik, “Making a “Completely Blind” Image Quality Analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. doi
  21. J. Ke, Q. Wang, Y. Wang, P. Milanfar, F. Yang, “MUSIQ: Multi-scale Image Quality Transformer,” ICCV, 2021. source
  22. S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, Y. Yang, “MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment,” CVPR Workshops, 2022. source
  23. J. Wang, K. C. K. Chan, C. C. Loy, “Exploring CLIP for Assessing the Look and Feel of Images,” AAAI, 2023. source
  24. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  25. Z. Wu, Y. Dong, X. Wei, “TUDSR: Twice Upsampling-Diffusion for Higher Super-Resolution,” CVPR, 2026. paper
  26. P. Esser, R. Rombach, B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” CVPR, 2021. paper
  27. X. Wang, L. Xie, C. Dong, Y. Shan, “Real-ESRGAN: Training Real-World Blind Super-Resolution With Pure Synthetic Data,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. source
  28. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  29. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  30. R. Wu, T. Yang, L. Sun, Z. Zhang, S. Li, L. Zhang, “SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. source · supplementary experiments
  31. H. Liu, C. Li, Q. Wu, Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. source
  32. Y. Li, K. Zhang, J. Liang, et al., “LSDIR: A Large Scale Dataset for Image Restoration,” CVPR Workshops, 2023. paper
  33. T. Karras, S. Laine, T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. paper
  34. I. Loshchilov, F. Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. paper
  35. P. Esser, S. Kulal, A. Blattmann, et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, 2024. paper
  36. J. Ho, T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv preprint, 2022. paper
  37. R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang et al., “NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results,” in Proc. IEEE CVPR Workshops, 2017, pp. 1110–1121. doi · CVF