Consistency Models: the same endpoint along a path (ICML 2023)
Optional · paper guide 2 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
On this page
- Progressive Distillation: two teacher moves in one student move (ICLR 2022)
- Consistency Models: the same endpoint along a path (ICML 2023)
- InstaFlow: reflow, then endpoint distillation (ICLR 2024)
- Latent Consistency Models: endpoints in a guided latent flow (2023)
- Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
- DMD: matching a teacher-defined distribution (CVPR 2024)
- DMD2: tracking the student and using real data (NeurIPS 2024)
Deep Learning for Computer Vision
Challenge
A diffusion sampler walks along a probability-flow trajectory to a near-clean endpoint. Consistency Models ask whether one network can predict that endpoint from any point on the path [8]. This makes one-step generation a direct prediction task while allowing multiple predictions when a larger budget is available.
The challenge is not only speed. Two outputs can agree because they are both meaningless constants. A useful consistency model must be anchored at a clean boundary and trained on states that correspond to the same underlying trajectory. The paper presents both distillation from a pretrained score model and direct training from data, which should be distinguished when discussing teacher requirements.
Method

Figure source: [8], arXiv:2303.01469v2, PDF page 3. Original PDF.
Consistency function. For a probability-flow path , define , where is a small near-data boundary. The network output has the same shape as the image. A skip-connected parameterization
uses scalar coefficients satisfying and . Here is the trainable network. It therefore returns the input exactly at the boundary. This boundary identity constrains the endpoint interpretation instead of leaving agreement unconstrained.
Consistency distillation. Sample a real image, a noise level and Gaussian noise to obtain a noisy state. A pretrained score model and a numerical ODE step produce an estimate of the adjacent, cleaner state on the same path. The online model predicts an endpoint from the noisy state; a target model predicts an endpoint from the cleaner state. Minimize their distance, stopping gradients through the target. Target weights track an exponential moving average of online weights in the original distillation algorithm.
With the noisier state and its teacher-produced neighbor, a discrete objective is . Here is a distance, a weight, target parameters, and stops gradients. The solver, grid and metric affect the quality of the supervision even though they are not used by a one-step deployment.
Direct consistency training. Replace the pretrained-score step with data-derived adjacent noise-level pairs constructed using a shared Gaussian noise realization. The theoretical connection is an asymptotic training argument; a finite pair is not literally a perfect ODE integration. The number of noise intervals and target updates are scheduled during training. This removes a separately pretrained teacher but does not remove self-targets or the need for a carefully designed loss.
Sampling. Start from the high-noise distribution and evaluate the endpoint function once. For additional steps, add noise to the current prediction at a lower noise level and evaluate again. Intermediate re-noising lets the model refine a candidate at a different uncertainty level. Repeating the same input at the same time without the sampling construction is not equivalent.
Metric. The original work compares pixel distances with LPIPS [30], a learned perceptual distance. LPIPS changes what errors are emphasized; it is not a likelihood. A perceptual consistency objective can improve appearance while leaving distribution or conditional errors that another metric detects.
Results
Table 1 reports CIFAR-10 FID 3.55 for one-step consistency distillation and 2.93 for two-step distillation. Its progressive-distillation comparison is 8.34 and 5.58 in that paper’s setup. Table 2 reports ImageNet 64 × 64 FID 6.20 for one step and 4.70 for two. Direct consistency training has different results: on that ImageNet task, 13.0 for one step and 11.1 for two. Distillation and direct training are therefore separate measured operating points, not interchangeable descriptions of one checkpoint.
These results support the endpoint formulation and a useful inference-budget trade-off. The distinction between direct training and distillation is therefore essential to interpreting the tables. They do not imply that a one-step endpoint predictor reproduces every statistic of its teacher, nor that a small-image benchmark establishes high-resolution text-conditioning performance.
Ablations
The metric and solver study finds LPIPS more effective than the compared pixel losses for consistency distillation. At a fixed grid size, the second-order Heun construction outperforms Euler in its distillation targets; the selected configuration uses 18 grid points. These choices concern training-target construction, not an 18-step deployment. The direct-training studies show benefits from increasing the interval count during training and adjusting the moving-average schedule. The paper primarily plots these sweeps; assigning one universal numerical gain to “use Heun” across tasks would exceed the evidence.
Conclusion
Learning an endpoint function with a clean-boundary identity provides a principled route to one- and few-step generation. Distillation supplies teacher trajectories; direct training constructs its supervision from data and nearby noise levels.
Problems
Consistency errors can accumulate across time levels, self-targets can be inaccurate, and a finite discretization can differ from the ideal continuous relation. Original direct training trails distillation on several benchmarks. Later improved consistency training revises weighting and target choices [9]. A model’s useful sampler and noise schedule remain part of its definition; endpoint agreement alone is insufficient evidence of quality or diversity.
References
- Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
- Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
- T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
- X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
- J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
- C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
- T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
- Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
- Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
- S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
- R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
- X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
- X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
- A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
- A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
- T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
- T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
- M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
- T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
- K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
- Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
- M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
- A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
- D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
- A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
- J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
- T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
- L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
- Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
- R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
- C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper