Full search page →

Latent Consistency Models: endpoints in a guided latent flow (2023)

Optional · paper guide 4 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Instantaneous velocity versus interval velocity
  2. Three meanings of teacher-free

Paper guides · 7

  1. Progressive Distillation: two teacher moves in one student move (ICLR 2022)
  2. Consistency Models: the same endpoint along a path (ICML 2023)
  3. InstaFlow: reflow, then endpoint distillation (ICLR 2024)
  4. Latent Consistency Models: endpoints in a guided latent flow (2023)
  5. Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
  6. DMD: matching a teacher-defined distribution (CVPR 2024)
  7. DMD2: tracking the student and using real data (NeurIPS 2024)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

Pixel-space consistency models demonstrate fast generation, but high-resolution text-to-image synthesis commonly uses a pretrained latent diffusion model. LCM adapts consistency distillation to that setting and incorporates classifier-free guidance into the student [10]. The objective is not simply to run an unchanged latent denoiser fewer times; it is to predict the endpoint of its guided probability flow.

Guidance is a central difficulty because conventional inference evaluates conditional and unconditional predictions separately. Compressing only unguided dynamics while retaining those extra calls would leave part of the inference cost untouched. The student therefore learns guidance-dependent behavior directly.

Method

Compare solver and skipping-step choices at matched displayed sampling steps; faster convergence in these plots is evidence under the selected guidance/training setting.
Original paper figure 3 — Compare solver and skipping-step choices at matched displayed sampling steps; faster convergence in these plots is evidence under the selected guidance/training setting.

Figure source: [10], arXiv:2310.04378v1, PDF page 8. Original PDF.

Latent representation. A pretrained encoder–decoder pair maps images into latents and back [11]. The consistency model acts on noisy image latents ztz_t, not RGB pixels. The output is a predicted clean latent of the same shape, decoded once for one-step generation. The representation reduces spatial computation but retains the autoencoder’s reconstruction limitations.

Endpoint parameterization. For an epsilon-prediction teacher, the consistency function takes the form

fθ(z,c,t)=cskip(t)z+cout(t)z−σtϵ^θ(z,c,t)αt.f_\theta(z,c,t)=c_{\mathrm{skip}}(t)z+c_{\mathrm{out}}(t)\frac{z-\sigma_t\widehat\epsilon_\theta(z,c,t)}{\alpha_t}.

Here zz is the noisy latent, cc the prompt condition, tt time, and αt,σt\alpha_t,\sigma_t the forward-process coefficients. The skip/output functions enforce the clean boundary. The paper also discusses adapting the parameterization to other prediction targets. Conversion conventions must match the checkpoint; replacing an epsilon predictor with a velocity predictor without conversion changes the meaning of the endpoint estimate.

Guidance-conditioned flow. The teacher combines conditional and unconditional noise predictions as (1+ω)ϵ(z,c,t)−ωϵ(z,∅,t)(1+\omega)\epsilon(z,c,t)-\omega\epsilon(z,\varnothing,t) in the paper’s convention. The student receives ω\omega through an additional guidance embedding and learns the guided endpoint. Its distillation target is therefore a family of maps indexed by prompt and guidance strength, not one unguided endpoint reused at all strengths.

Consistency pairs. A teacher solver advances the noisy latent to a cleaner time. The online student’s endpoint prediction is compared with a stopped-gradient moving-average target at that cleaner time. The paper uses a latent distance rather than requiring repeated pixel decoding through an image perceptual network for each target comparison. Teacher solvers are used to make training targets; the one-step student does not execute that solver at deployment.

Skipping intervals. Ordinary neighboring times can be too close to provide a useful large-jump learning signal. LCM pairs states separated by a configurable number of grid indices. With an efficient solver, these skipped targets improve convergence. A skip interval describes training-pair separation, not the number of student inference calls.

One- and few-step inference. A noise latent and prompt/guidance embeddings enter the endpoint model. One call gives a clean-latent estimate; more calls can use lower-noise re-noising and repeated prediction. The schedule and boundary coefficients belong to the trained pipeline. A standard diffusion scheduler is not automatically the right LCM scheduler.

Results

Table 1 evaluates the 512 × 512 setting on the paper’s LAION-Aesthetic-6+ protocol [31]. At guidance value 8, LCM gives FID 35.36, 13.31, 11.10 and 11.84 at one, two, four and eight steps. The corresponding CLIP scores, in the paper’s scaling, are 24.14, 27.83, 28.69 and 28.84. Four steps therefore improve FID substantially over one, and eight do not improve it further in that table.

At 768 × 768 on its other evaluation subset, Table 2 reports FID 34.22, 16.32, 13.53 and 14.97. These are not directly comparable to the 512-resolution rows because the model, evaluation subset and resolution differ. They also should not be placed beside a COCO FID without identifying the changed reference distribution.

Ablations

The solver/skip study compares DDIM, DPM-Solver and DPM-Solver++ constructions with different skipped intervals [5], [6]. The plotted experiments show that choosing a useful skipped interval improves convergence and final quality at the same training budget. The selected DDIM configuration uses a skip of 20 grid indices. This is not a twenty-step inference requirement.

The guidance sweep shows changing alignment and sample quality as guidance changes. No single value is universally best across metrics or resolutions. The one/two/four/eight-step tables themselves reveal a nonmonotonic budget effect: a consistency model is flexible, but additional calls do not guarantee improvement for every scheduler and metric. The paper reports plots for some training sweeps rather than exact tabulated component gains; those gains should not be invented.

Conclusion

Consistency distillation can operate in a pretrained guided latent flow. Guidance conditioning and skipped teacher intervals make the learning target fit the intended few-step use, while retaining a single decoder at output.

Problems

The student inherits latent compression, teacher quality and the guidance range used for training. One step remains a weaker operating point than several steps in these experiments. The method still needs teacher-based target construction and a compatible sampler. Its original experiments concern stated checkpoints and evaluation subsets; later adapters and deployment pipelines need their own evaluation. Later adversarial or distribution-matching methods change the objective and may use different data, so cross-paper score differences are not an isolated test of consistency.

References

  1. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
  2. Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
  3. T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
  4. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  5. J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
  6. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
  7. T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
  8. Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
  9. Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
  10. S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
  11. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  12. X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
  13. X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
  16. T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
  17. T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
  18. M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
  19. T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
  20. K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
  21. Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
  22. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
  23. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
  24. D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
  25. A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
  26. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  27. T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
  28. L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
  29. Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
  30. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  31. C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper