OSEDiff: LR-latent start and variational score regularization (NeurIPS 2024)
Optional · paper guide 3 of 8 for DLCV 09A.2 · One-Step Diffusion Super-Resolution. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- ResShift: a short restoration-conditioned chain (NeurIPS 2023)
- SinSR: distilling a deterministic image-conditioned map (CVPR 2024)
- OSEDiff: LR-latent start and variational score regularization (NeurIPS 2024)
- S3Diff: degradation-guided low-rank adaptation
- TSD-SR: target-aware score regularization (CVPR 2025)
- InvSR: a learned image-conditioned start (CVPR 2025)
- PiSA-SR: separate pixel and semantic adaptation (CVPR 2025)
- AddSR: prediction-refined adversarial restoration distillation
Deep Learning for Computer Vision
Challenge
OSEDiff asks how to retain a large pretrained image prior without repeatedly running its diffusion sampler [1]. Multi-step text-to-image-prior restorers can produce appealing detail, but need repeated denoising and often use an input-control network. A conventional paired reconstruction model is fast, yet may return smooth outputs when the HR target is ambiguous. The proposed system combines a direct LR-to-HR transformation with a distribution-oriented regularizer during training.
Its central decisions are connected. Start from an LR image latent rather than random noise. Adapt the pretrained encoder and denoiser with LoRA. Keep the decoder fixed so that generated latents remain interpretable in the original representation. Supply image-derived text prompts. During training, use both paired reconstruction and a latent-space score-difference signal. These are separate components with separate evidence; “one-step diffusion” alone does not explain the method.
Method

Figure source: [1], arXiv:2406.08177v3, PDF page 4. Original PDF.
Inputs and representation. Let be the low-quality image enlarged to the working output resolution and its paired high-quality training target. The pretrained model supplies a variational-autoencoder encoder , a text-conditioned diffusion denoiser and decoder [29]. For a 512 × 512 image under the model’s factor-eight spatial compression, the latent grid is 64 × 64; the representation includes multiple latent channels. Spatial dimensions must match the generator’s intended output, so a latent start is not equivalent to feeding an unresized LR grid and expecting an implicit arbitrary scale change.
LoRA adaptation. Train low-rank additions to the encoder and denoiser, obtaining and [11]. The underlying pretrained weights remain fixed, and the selected low-rank modules supply the adaptation. A matrix update has the form with rank bounded by the adapter rank; this changes the effective transformation without learning every backbone parameter. The reported setting uses rank 4 in the VAE encoder, generator denoiser and adapted regularizer. A small adaptation does not make the complete execution model equally small.
Fixed decoder. Set . Freezing the decoder avoids letting it freely redefine how the predicted latent becomes an image while the prior regularizer still evaluates the pretrained latent coordinates. This is a consistency of representation, not a guarantee that the encoder/decoder round trip is exact. An adapted decoder can improve some pixel scores while weakening the intended prior alignment; the paper tests that choice explicitly.
Image-derived prompt. A prompt extractor processes the low-quality input. The main implementation uses the degradation-aware prompt extraction approach from SeeSR [30]. Its tags are embedded for the text-conditioned denoiser. A tag is an inferred semantic condition and can be wrong. This is additional computation at deployment, even though the denoiser is called only once. The paper compares a null prompt, the lightweight extractor and a larger multimodal captioner [31].
One fixed-timestep transformation. Define and . The generator predicts
Here is the selected fixed model timestep, and are its signal/noise coefficients in the paper’s notation. There is no newly sampled Gaussian noise added to in this deployed path. The pretrained denoising expression is used as a parameterization of a newly adapted image-to-image map. The LR latent is not claimed to obey the original forward corruption law at that timestep. Given a fixed input and deterministic extraction/execution, the output is repeatable; repeatability does not establish correctness of generated detail.
Paired reconstruction. The data term is
MSE penalizes pixel discrepancies; LPIPS compares learned perceptual features [12]. The scalar balances their scales. The paired training data tell the model what an HR result should be for the sampled synthetic degradation. They do not reveal the true HR original of an arbitrary internet image at inference.
Two training regularizers. Keep a frozen pretrained denoiser as the prior model. Maintain a second denoiser , adapted with LoRA to the generator’s changing output distribution. This second model is distinct from the generator denoiser . The diagram therefore contains three denoiser roles: the deployed generator, a fixed prior estimator and an adapted generated-distribution estimator. Confusing the last two with a second inference stage obscures the main computational trade.
Noised latent comparison. Sample a regularization time and Gaussian noise , and form . The regularization gradient is written as
Here is the time-dependent weight, and the bracket denotes a vector–Jacobian product through the generator latent. The frozen model supplies a target-prior prediction; the adapted model estimates how the current generated samples differ from it. The score-difference direction is treated as a training signal rather than differentiating a tractable, explicitly evaluated image-density KL divergence. VSD originates in variational score-distillation work [13]; using it here does not establish exact posterior sampling for the unknown SR degradation.
Why latent VSD. A pixel-space route would decode and re-encode the generated image before regularizing it. OSEDiff approximates and works directly on . This removes repeated encoding/decoding from the regularization path. The approximation is most credible when latents stay compatible with the fixed representation. It is not an exact algebraic cancellation for every latent vector.
Adapted-regularizer update. Train to predict the known added noise on generated latents:
This loss updates the adapted regularizer. It is not an extra reconstruction loss on the deployed generator. During its update, generator samples serve as training data; the optimizers and stopped-gradient boundaries must keep the two roles separate. A poorly tracked generated distribution can make the prior-difference direction inaccurate.
Complete training and deployment. The generator uses , with the reported and . The dataset is LSDIR [32] plus the first 10,000 FFHQ images [33], with synthetic pairs from the Real-ESRGAN pipeline [27]. The paper reports AdamW adaptation [34], learning rate and batch size 16 on four A100 GPUs. Deployment retains the prompt extractor, adapted encoder, adapted generator denoiser and fixed decoder. The two regularizer denoisers are discarded.
Results
Table 1 uses 3,000 synthetic DIV2K validation crops and captured RealSR/DRealSR pairs [17], [15], [16]. OSEDiff’s selected rows are:
| Test set | PSNR ↑ | LPIPS ↓ | DISTS ↓ | FID ↓ | MUSIQ ↑ | CLIPIQA ↑ |
|---|---|---|---|---|---|---|
| DIV2K-Val | 23.72 | 0.2941 | 0.1976 | 26.32 | 67.97 | 0.6683 |
| RealSR | 25.15 | 0.2921 | 0.2128 | 123.49 | 69.09 | 0.6693 |
The paper finds favorable perceptual/reference comparisons on real pairs, while some multi-step methods obtain better no-reference scores and restoration-specific models obtain higher PSNR. The table is not an across-the-board victory for one-step prediction. Metrics measure different things, and crop/degradation conventions belong with the numbers.
Table 2 reports 0.11 seconds on an A100 for its stated 512 × 512 input-size test, compared with 11.50 seconds for StableSR and 0.13 for SinSR in that experiment. It lists 8.5 million trainable parameters but 1,775 million total parameters for OSEDiff. Reporting only “8.5 million parameters” as its complete model size would be wrong. The paper also separately times prompt extraction; the inclusion boundary should be checked when reproducing its latency rather than assumed to match every other paper.
Ablations
Prior objective, Table 3. On RealSR, no VSD gives PSNR 25.42, LPIPS 0.2934, MUSIQ 65.23 and CLIPIQA 0.5876. Replacing it with GAN training gives 25.00/0.2760/67.29/0.6254. Pixel-domain VSD gives 25.05/0.2759/67.90/0.6256. Latent-domain OSEDiff gives 25.15/0.2921/69.09/0.6693. Latent VSD leads those two NR scores but does not lead every fidelity/perceptual metric: its LPIPS is worse than the GAN and image-domain variants. The exact table is more informative than saying the regularizer improves everything.
Prompt extraction, Table 4. On DRealSR, the null prompt gives PSNR 28.51, LPIPS 0.2896, MUSIQ 62.13 and CLIPIQA 0.6599. DAPE gives 27.92/0.2968/64.65/0.6963. The larger captioner gives 27.72/0.3109/65.70/0.7033. Extractor times are 0, 0.02 and 3.53 seconds. Richer prompting increases NR scores while reducing reference fidelity in this study. The larger captioner’s cost is substantial compared with the generator’s reported runtime.
Adapter rank, Tables 5–6. Encoder rank 2 does not converge in the reported experiment. Rank 4 gives PSNR 25.15 and MUSIQ 69.09, while rank 8 gives 24.86 and 68.37. For the generator denoiser, ranks 2, 4 and 8 give PSNR 25.28, 25.15 and 24.87, with DISTS 0.2154, 0.2128 and 0.2115. The chosen rank is a balance, not the best value in every column. More adapter capacity is not guaranteed to improve generalization.
Encoder/decoder, Table 7. With neither adapted, MUSIQ is 58.99 and CLIPIQA 0.5303. Adapting only the decoder gives 65.83/0.5604; adapting both gives 65.92/0.5778. Adapting only the encoder, the selected method, gives 69.09/0.6693. The both-adapted variant has higher PSNR, 25.59 versus 25.15, so freezing the decoder supports the selected perceptual/prior trade rather than dominating all metrics. These controlled rows directly support the design distinction between removing degradation in the encoder and preserving the decoder’s latent interpretation.
Conclusion
OSEDiff turns a pretrained latent diffusion backbone into a direct image-conditioned restorer. Paired losses preserve training-target agreement, while two training-only score models supply a natural-image regularizer. Encoder adaptation and a fixed decoder make the latent-space shortcut practical; prompt extraction adds semantics and deployment cost.
Problems
An LR-latent start removes random sampling from the default generator but does not prevent semantic invention. Synthetic degradations do not cover every real camera pipeline. Prompt errors can steer a plausible reconstruction toward the wrong content, and higher NR scores can accompany lower reference fidelity. Latent VSD relies on approximate representation consistency and a well-tracked generated distribution. The total pretrained model remains large despite small adapters. Later work investigates degradation-guided updates, target-aware scores and separate enhancement controls [4], [6], [8]; those are changed objectives and systems, not proof that this ambiguity has been solved.
References
- R. Wu, L. Sun, Z. Ma, L. Zhang, “One-Step Effective Diffusion Network for Real-World Image Super-Resolution,” NeurIPS, 2024. source
- Z. Yue, J. Wang, C. C. Loy, “ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting,” NeurIPS, 2023. source
- Y. Wang, W. Yang, X. Chen, et al., “SinSR: Diffusion-Based Image Super-Resolution in a Single Step,” CVPR, 2024. source · supplementary ablations
- A. Zhang, Z. Yue, R. Pei, W. Ren, X. Cao, “Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors,” arXiv preprint, 2024. paper
- Y. Tai, R. Xie, C. Zhao, K. Zhang, Z. Zhang, J. Zhou, J. Yang, “AddSR: Accelerating diffusion-based blind super-resolution with adversarial diffusion distillation,” Pattern Recognition, 2026. published version · manuscript supplying the discussed tables
- L. Dong, Q. Fan, Y. Guo, Z. Wang, Q. Zhang, J. Chen, Y. Luo, C. Zou, “TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution,” CVPR, 2025. paper
- Z. Yue, K. Liao, C. C. Loy, “Arbitrary-steps Image Super-resolution via Diffusion Inversion,” CVPR, 2025. paper
- L. Sun, R. Wu, Z. Ma, S. Liu, Q. Yi, L. Zhang, “Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach,” CVPR, 2025. paper
- A. Liu, Y. Liu, J. Gu, Y. Qiao, C. Dong, “Blind Image Super-Resolution: A Survey and Beyond,” IEEE TPAMI (early access), 2022. source
- B. B. Moser, A. S. Shanbhag, F. Raue, S. Frolov, S. Palacio and A. Dengel, “Diffusion Models, Image Super-Resolution, and Everything: A Survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 7, pp. 11793–11813, 2025. doi · arXiv
- E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. source
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, J. Zhu, “ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation,” NeurIPS, 2023. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- J. Cai, H. Zeng, H. Yong, Z. Cao, L. Zhang, “Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. source
- P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, L. Lin, “Component Divide-and-Conquer for Real-World Image Super-Resolution,” ECCV, 2020. source
- E. Agustsson and R. Timofte, “NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study,” in Proc. IEEE CVPR Workshops, 2017, pp. 1122–1131. doi · CVF
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- K. Ding, K. Ma, S. Wang, E. P. Simoncelli, “Image Quality Assessment: Unifying Structure and Texture Similarity,” IEEE TPAMI, 2022. source
- A. Mittal, R. Soundararajan and A. C. Bovik, “Making a “Completely Blind” Image Quality Analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. doi
- J. Ke, Q. Wang, Y. Wang, P. Milanfar, F. Yang, “MUSIQ: Multi-scale Image Quality Transformer,” ICCV, 2021. source
- S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, Y. Yang, “MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment,” CVPR Workshops, 2022. source
- J. Wang, K. C. K. Chan, C. C. Loy, “Exploring CLIP for Assessing the Look and Feel of Images,” AAAI, 2023. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- Z. Wu, Y. Dong, X. Wei, “TUDSR: Twice Upsampling-Diffusion for Higher Super-Resolution,” CVPR, 2026. paper
- P. Esser, R. Rombach, B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” CVPR, 2021. paper
- X. Wang, L. Xie, C. Dong, Y. Shan, “Real-ESRGAN: Training Real-World Blind Super-Resolution With Pure Synthetic Data,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. source
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- R. Wu, T. Yang, L. Sun, Z. Zhang, S. Li, L. Zhang, “SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. source · supplementary experiments
- H. Liu, C. Li, Q. Wu, Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. source
- Y. Li, K. Zhang, J. Liang, et al., “LSDIR: A Large Scale Dataset for Image Restoration,” CVPR Workshops, 2023. paper
- T. Karras, S. Laine, T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. paper
- I. Loshchilov, F. Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. paper
- P. Esser, S. Kulal, A. Blattmann, et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, 2024. paper
- J. Ho, T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv preprint, 2022. paper
- R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang et al., “NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results,” in Proc. IEEE CVPR Workshops, 2017, pp. 1110–1121. doi · CVF