PiSA-SR: separate pixel and semantic adaptation (CVPR 2025)
Optional · paper guide 7 of 8 for DLCV 09A.2 · One-Step Diffusion Super-Resolution. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- ResShift: a short restoration-conditioned chain (NeurIPS 2023)
- SinSR: distilling a deterministic image-conditioned map (CVPR 2024)
- OSEDiff: LR-latent start and variational score regularization (NeurIPS 2024)
- S3Diff: degradation-guided low-rank adaptation
- TSD-SR: target-aware score regularization (CVPR 2025)
- InvSR: a learned image-conditioned start (CVPR 2025)
- PiSA-SR: separate pixel and semantic adaptation (CVPR 2025)
- AddSR: prediction-refined adversarial restoration distillation
Deep Learning for Computer Vision
Challenge
Pixel reconstruction and generative detail enhancement often compete inside the same adaptation. PiSA-SR separates them into two low-rank update groups, aiming to offer more understandable control over the resulting appearance [8]. The intention is to change how strongly the model restores measured structure and adds semantic/perceptual detail without retraining for every preference.
These are learned control directions, not a physical decomposition of an image into perfectly factual and invented parts. A semantic enhancement can still change letters or object details, and a pixel-oriented model can retain blur. The controls therefore need measured and visual evaluation rather than labels alone.
Method

Figure source: [8], arXiv:2412.03017v2, PDF page 5. Original PDF.
Residual latent map. Encode the enlarged LR input as using a fixed pretrained image autoencoder. The denoising backbone predicts a residual-like correction, giving in the paper’s simplified fixed-timestep formulation. This is an adapted image-to-image residual, not a claim that contains independent Gaussian noise at that timestep.
Pixel adapter. Add a low-rank parameter group to the frozen pretrained denoiser and train it with an L2 reconstruction objective. The resulting parameter set supplies an output focused on reducing degradation and recovering reference-aligned structure. The image encoder and decoder remain fixed in this design.
Semantic adapter. Freeze the learned pixel adapter, add a second low-rank group , and train this group with LPIPS and a classifier-score-distillation objective. The combined parameter set is . The stages are sequential: the semantic phase is not jointly allowed to rewrite the pixel adapter arbitrarily.
Semantic prior signal. A fixed text-conditioned diffusion model supplies conditional and unconditional predictions. Their difference gives a classifier-free-guidance-derived semantic direction [36]. The training objective uses this signal in latent space, together with LPIPS. It does not train a separate generated-distribution score model as in OSEDiff’s VSD regularizer. Consequently, it reduces auxiliary optimization while changing the objective; the two regularizers are not statistically identical.
Default inference. Merge the selected pixel and semantic adapters into one combined model and make one residual prediction. The trained semantic direction supplies the intended enhancement without requiring the training prompt-extraction model online. The fixed image encoder and decoder still execute. The method’s default parameter count includes a large pretrained backbone even though the adapters themselves are small.
Adjustable inference. Evaluate both the pixel-only and combined denoisers, then form
Here weights the pixel-oriented correction and the difference introduced by semantic adaptation. The output latent uses the resulting residual. This configuration needs two denoiser predictions. It is not the same cost as the default combined prediction, and interpolating adapter weights is not algebraically equivalent to this output-difference construction in a nonlinear network.
Training data. The paper uses LSDIR and the first 10,000 FFHQ images with Real-ESRGAN-style degradations [32], [33], [27]. Adapter rank 4 is used in the selected configuration. Training-generated semantic conditions do not imply that inference needs a captioner; the design deliberately distills those semantic preferences into the learned update.
Results
On RealSR, the default setting reports PSNR 25.50, LPIPS 0.2672, CLIPIQA 0.6702 and MUSIQ 70.15. In the control sweep with pixel strength fixed at 1, semantic strength 0.2 gives PSNR 26.95 and MUSIQ 53.64; semantic strength 1.2 gives 24.59 and 71.60. This illustrates a practical preference trade, not independent verification of added detail.
The timing comparison uses ×4 SR from 128 × 128 input on a single A100 80 GB GPU. It reports 0.09 seconds for the default configuration and 0.13 for adjustable inference, with one and two diffusion predictions respectively. A quality-control interface should disclose that distinction rather than advertise every setting as the default one-step latency.
Ablations
The dual-adapter study compares a pixel-only trained variant, semantic-only variant and the two-stage design. Their RealSR PSNR values are 27.28, 24.13 and 25.50; LPIPS values are 0.3090, 0.2803 and 0.2672. MUSIQ is 49.02, 70.69 and 70.15. The combined design improves this reference perceptual distance over the single-objective variants while retaining a stronger pixel score than semantic-only training. These are separately trained ablations, not all identical to zeroing a coefficient in one final model.
The rank study changes one adapter’s capacity while holding the other fixed. Increasing semantic rank from 4 to 32 changes PSNR from 25.50 to 25.40 and LPIPS from 0.2672 to 0.2713, with similar MUSIQ. More rank is not automatically beneficial. The control sweep also shows that the best LPIPS or NR value can occur at an intermediate strength rather than an extreme.
Conclusion
Separating training objectives into two adapters provides useful enhancement directions. A combined default is fast; output-difference control offers flexibility at additional inference cost.
Problems
The control names do not guarantee that semantics and pixels are perfectly disentangled. Severe degradation may need different adaptation, and one fixed pixel adapter may not cover all damage. Strong detail settings can change reference content. Adjustable inference costs more than the default and still inherits the pretrained prior’s biases. Evaluate each selected operating point rather than presenting one score as a certificate for the whole control range.
References
- R. Wu, L. Sun, Z. Ma, L. Zhang, “One-Step Effective Diffusion Network for Real-World Image Super-Resolution,” NeurIPS, 2024. source
- Z. Yue, J. Wang, C. C. Loy, “ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting,” NeurIPS, 2023. source
- Y. Wang, W. Yang, X. Chen, et al., “SinSR: Diffusion-Based Image Super-Resolution in a Single Step,” CVPR, 2024. source · supplementary ablations
- A. Zhang, Z. Yue, R. Pei, W. Ren, X. Cao, “Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors,” arXiv preprint, 2024. paper
- Y. Tai, R. Xie, C. Zhao, K. Zhang, Z. Zhang, J. Zhou, J. Yang, “AddSR: Accelerating diffusion-based blind super-resolution with adversarial diffusion distillation,” Pattern Recognition, 2026. published version · manuscript supplying the discussed tables
- L. Dong, Q. Fan, Y. Guo, Z. Wang, Q. Zhang, J. Chen, Y. Luo, C. Zou, “TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution,” CVPR, 2025. paper
- Z. Yue, K. Liao, C. C. Loy, “Arbitrary-steps Image Super-resolution via Diffusion Inversion,” CVPR, 2025. paper
- L. Sun, R. Wu, Z. Ma, S. Liu, Q. Yi, L. Zhang, “Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach,” CVPR, 2025. paper
- A. Liu, Y. Liu, J. Gu, Y. Qiao, C. Dong, “Blind Image Super-Resolution: A Survey and Beyond,” IEEE TPAMI (early access), 2022. source
- B. B. Moser, A. S. Shanbhag, F. Raue, S. Frolov, S. Palacio and A. Dengel, “Diffusion Models, Image Super-Resolution, and Everything: A Survey,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 7, pp. 11793–11813, 2025. doi · arXiv
- E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. source
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, J. Zhu, “ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation,” NeurIPS, 2023. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- J. Cai, H. Zeng, H. Yong, Z. Cao, L. Zhang, “Toward Real-World Single Image Super-Resolution: A New Benchmark and a New Model,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. source
- P. Wei, Z. Xie, H. Lu, Z. Zhan, Q. Ye, W. Zuo, L. Lin, “Component Divide-and-Conquer for Real-World Image Super-Resolution,” ECCV, 2020. source
- E. Agustsson and R. Timofte, “NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study,” in Proc. IEEE CVPR Workshops, 2017, pp. 1122–1131. doi · CVF
- Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. doi
- K. Ding, K. Ma, S. Wang, E. P. Simoncelli, “Image Quality Assessment: Unifying Structure and Texture Similarity,” IEEE TPAMI, 2022. source
- A. Mittal, R. Soundararajan and A. C. Bovik, “Making a “Completely Blind” Image Quality Analyzer,” IEEE Signal Processing Letters, vol. 20, no. 3, pp. 209–212, 2013. doi
- J. Ke, Q. Wang, Y. Wang, P. Milanfar, F. Yang, “MUSIQ: Multi-scale Image Quality Transformer,” ICCV, 2021. source
- S. Yang, T. Wu, S. Shi, S. Lao, Y. Gong, M. Cao, J. Wang, Y. Yang, “MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment,” CVPR Workshops, 2022. source
- J. Wang, K. C. K. Chan, C. C. Loy, “Exploring CLIP for Assessing the Look and Feel of Images,” AAAI, 2023. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- Z. Wu, Y. Dong, X. Wei, “TUDSR: Twice Upsampling-Diffusion for Higher Super-Resolution,” CVPR, 2026. paper
- P. Esser, R. Rombach, B. Ommer, “Taming Transformers for High-Resolution Image Synthesis,” CVPR, 2021. paper
- X. Wang, L. Xie, C. Dong, Y. Shan, “Real-ESRGAN: Training Real-World Blind Super-Resolution With Pure Synthetic Data,” Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. source
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- R. Wu, T. Yang, L. Sun, Z. Zhang, S. Li, L. Zhang, “SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. source · supplementary experiments
- H. Liu, C. Li, Q. Wu, Y. J. Lee, “Visual Instruction Tuning,” NeurIPS, 2023. source
- Y. Li, K. Zhang, J. Liang, et al., “LSDIR: A Large Scale Dataset for Image Restoration,” CVPR Workshops, 2023. paper
- T. Karras, S. Laine, T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” CVPR, 2019. paper
- I. Loshchilov, F. Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. paper
- P. Esser, S. Kulal, A. Blattmann, et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis,” ICML, 2024. paper
- J. Ho, T. Salimans, “Classifier-Free Diffusion Guidance,” arXiv preprint, 2022. paper
- R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, L. Zhang et al., “NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results,” in Proc. IEEE CVPR Workshops, 2017, pp. 1110–1121. doi · CVF