Full search page →

Consistency Models: the same endpoint along a path (ICML 2023)

Optional · paper guide 2 of 7 for DLCV 12B · One-Step Generation. Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

On this page

Supplementary notes · 2

  1. Instantaneous velocity versus interval velocity
  2. Three meanings of teacher-free

Paper guides · 7

  1. Progressive Distillation: two teacher moves in one student move (ICLR 2022)
  2. Consistency Models: the same endpoint along a path (ICML 2023)
  3. InstaFlow: reflow, then endpoint distillation (ICLR 2024)
  4. Latent Consistency Models: endpoints in a guided latent flow (2023)
  5. Adversarial Diffusion Distillation: a teacher and a discriminator (ECCV 2024)
  6. DMD: matching a teacher-defined distribution (CVPR 2024)
  7. DMD2: tracking the student and using real data (NeurIPS 2024)

Deep Learning for Computer Vision

  1. 02 · Deep Image Classification
View the whole course tree →

Challenge

A diffusion sampler walks along a probability-flow trajectory to a near-clean endpoint. Consistency Models ask whether one network can predict that endpoint from any point on the path [8]. This makes one-step generation a direct prediction task while allowing multiple predictions when a larger budget is available.

The challenge is not only speed. Two outputs can agree because they are both meaningless constants. A useful consistency model must be anchored at a clean boundary and trained on states that correspond to the same underlying trajectory. The paper presents both distillation from a pretrained score model and direct training from data, which should be distinguished when discussing teacher requirements.

Method

Points on the same probability-flow trajectory map toward a common endpoint; that consistency is distinct from forcing endpoints of unrelated trajectories to agree.
Original paper figure 2 — Points on the same probability-flow trajectory map toward a common endpoint; that consistency is distinct from forcing endpoints of unrelated trajectories to agree.

Figure source: [8], arXiv:2303.01469v2, PDF page 3. Original PDF.

Consistency function. For a probability-flow path {xt}\{x_t\}, define f(xt,t)=xεf(x_t,t)=x_\varepsilon, where ε>0\varepsilon>0 is a small near-data boundary. The network output has the same shape as the image. A skip-connected parameterization

fθ(x,t)=cskip(t)x+cout(t)Fθ(x,t)f_\theta(x,t)=c_{\mathrm{skip}}(t)x+c_{\mathrm{out}}(t)F_\theta(x,t)

uses scalar coefficients satisfying cskip(ε)=1c_{\mathrm{skip}}(\varepsilon)=1 and cout(ε)=0c_{\mathrm{out}}(\varepsilon)=0. Here FθF_\theta is the trainable network. It therefore returns the input exactly at the boundary. This boundary identity constrains the endpoint interpretation instead of leaving agreement unconstrained.

Consistency distillation. Sample a real image, a noise level and Gaussian noise to obtain a noisy state. A pretrained score model and a numerical ODE step produce an estimate of the adjacent, cleaner state on the same path. The online model predicts an endpoint from the noisy state; a target model predicts an endpoint from the cleaner state. Minimize their distance, stopping gradients through the target. Target weights track an exponential moving average of online weights in the original distillation algorithm.

With xtn+1x_{t_{n+1}} the noisier state and x^tn\widehat x_{t_n} its teacher-produced neighbor, a discrete objective is E[λ(tn)d(fθ(xtn+1,tn+1),sg⁡fθ−(x^tn,tn))]\mathbb E[\lambda(t_n)d(f_\theta(x_{t_{n+1}},t_{n+1}),\operatorname{sg}f_{\theta^-}(\widehat x_{t_n},t_n))]. Here dd is a distance, λ\lambda a weight, θ−\theta^- target parameters, and sg⁡\operatorname{sg} stops gradients. The solver, grid and metric affect the quality of the supervision even though they are not used by a one-step deployment.

Direct consistency training. Replace the pretrained-score step with data-derived adjacent noise-level pairs constructed using a shared Gaussian noise realization. The theoretical connection is an asymptotic training argument; a finite pair is not literally a perfect ODE integration. The number of noise intervals and target updates are scheduled during training. This removes a separately pretrained teacher but does not remove self-targets or the need for a carefully designed loss.

Sampling. Start from the high-noise distribution and evaluate the endpoint function once. For additional steps, add noise to the current prediction at a lower noise level and evaluate again. Intermediate re-noising lets the model refine a candidate at a different uncertainty level. Repeating the same input at the same time without the sampling construction is not equivalent.

Metric. The original work compares pixel distances with LPIPS [30], a learned perceptual distance. LPIPS changes what errors are emphasized; it is not a likelihood. A perceptual consistency objective can improve appearance while leaving distribution or conditional errors that another metric detects.

Results

Table 1 reports CIFAR-10 FID 3.55 for one-step consistency distillation and 2.93 for two-step distillation. Its progressive-distillation comparison is 8.34 and 5.58 in that paper’s setup. Table 2 reports ImageNet 64 × 64 FID 6.20 for one step and 4.70 for two. Direct consistency training has different results: on that ImageNet task, 13.0 for one step and 11.1 for two. Distillation and direct training are therefore separate measured operating points, not interchangeable descriptions of one checkpoint.

These results support the endpoint formulation and a useful inference-budget trade-off. The distinction between direct training and distillation is therefore essential to interpreting the tables. They do not imply that a one-step endpoint predictor reproduces every statistic of its teacher, nor that a small-image benchmark establishes high-resolution text-conditioning performance.

Ablations

The metric and solver study finds LPIPS more effective than the compared pixel losses for consistency distillation. At a fixed grid size, the second-order Heun construction outperforms Euler in its distillation targets; the selected configuration uses 18 grid points. These choices concern training-target construction, not an 18-step deployment. The direct-training studies show benefits from increasing the interval count during training and adjusting the moving-average schedule. The paper primarily plots these sweeps; assigning one universal numerical gain to “use Heun” across tasks would exceed the evidence.

Conclusion

Learning an endpoint function with a clean-boundary identity provides a principled route to one- and few-step generation. Distillation supplies teacher trajectories; direct training constructs its supervision from data and nearby noise levels.

Problems

Consistency errors can accumulate across time levels, self-targets can be inaccurate, and a finite discretization can differ from the ideal continuous relation. Original direct training trails distillation on several benchmarks. Later improved consistency training revises weighting and target choices [9]. A model’s useful sampler and noise schedule remain part of its definition; endpoint agreement alone is insufficient evidence of quality or diversity.

References

  1. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” ICLR, 2021. paper
  2. Y. Lipman, M. Havasi, P. Holderrieth, et al., “Flow Matching Guide and Code,” arXiv tutorial, 2024. paper
  3. T. Karras, M. Aittala, T. Aila, S. Laine, “Elucidating the Design Space of Diffusion-Based Generative Models,” NeurIPS, 2022. source
  4. X. Fan, Z. Wu, H. Wu, L. Cao, “A Survey on Pre-Trained Diffusion Model Distillations,” arXiv preprint, version 2, 2025. paper
  5. J. Song, C. Meng, S. Ermon, “Denoising Diffusion Implicit Models,” ICLR, 2021. paper
  6. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, “DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps,” NeurIPS, 2022. paper
  7. T. Salimans, J. Ho, “Progressive Distillation for Fast Sampling of Diffusion Models,” ICLR, 2022. paper
  8. Y. Song, P. Dhariwal, M. Chen, I. Sutskever, “Consistency Models,” ICML, 2023. paper
  9. Y. Song, P. Dhariwal, “Improved Techniques for Training Consistency Models,” ICLR, 2024. paper
  10. S. Luo, Y. Tan, L. Huang, J. Li, H. Zhao, “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference,” arXiv preprint, 2023. paper
  11. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models,” CVPR, 2022. source
  12. X. Liu, C. Gong, Q. Liu, “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” ICLR, 2023. paper
  13. X. Liu, X. Zhang, J. Ma, J. Peng, Q. Liu, “InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation,” ICLR, 2024. paper
  14. A. Sauer, D. Lorenz, A. Blattmann, R. Rombach, “Adversarial Diffusion Distillation,” ECCV, 2024. paper
  15. A. Sauer, F. Boesel, T. Dockhorn, A. Blattmann, P. Esser, R. Rombach, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation,” SIGGRAPH Asia Conference Papers, 2024. paper · published version
  16. T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, T. Park, “One-step Diffusion with Distribution Matching Distillation,” CVPR, 2024. paper
  17. T. H. Nguyen, A. Tran, “SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation,” CVPR, 2024. paper
  18. M. Zhou, H. Zheng, Z. Wang, M. Yin, H. Huang, “Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation,” ICML, 2024. paper
  19. T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, “Improved Distribution Matching Distillation for Fast Image Synthesis,” NeurIPS, 2024. paper
  20. K. Frans, D. Hafner, S. Levine, P. Abbeel, “One Step Diffusion via Shortcut Models,” ICLR, 2025. paper
  21. Z. Geng, M. Deng, X. Bai, J. Z. Kolter, K. He, “Mean Flows for One-step Generative Modeling,” NeurIPS, 2025. paper
  22. M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium,” NeurIPS, 2017. source
  23. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” ICML, 2021. paper
  24. D. Ghosh, H. Hajishirzi, L. Schmidt, “GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment,” NeurIPS Datasets and Benchmarks, 2023. paper
  25. A. Krizhevsky, V. Nair and G. Hinton, “CIFAR-10 and CIFAR-100 datasets,” official dataset page; companion technical report: A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” 2009. dataset and report
  26. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” CVPR, 2009. doi
  27. T. Lin, M. Maire, S. Belongie, et al., “Microsoft COCO: Common Objects in Context,” ECCV, 2014. paper
  28. L. Yang, Z. Zhang, Y. Song, et al., “Diffusion Models: A Comprehensive Survey of Methods and Applications,” ACM Computing Surveys, 2023. paper
  29. Z. Yu, J. Yuan, H. Yang, et al., “DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation,” arXiv preprint, 2026. paper
  30. R. Zhang, P. Isola, A. A. Efros, E. Shechtman and O. Wang, “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” in Proc. IEEE/CVF CVPR, 2018, pp. 586–595. doi · arXiv
  31. C. Schuhmann, R. Beaumont, R. Vencu, et al., “LAION-5B: An open large-scale dataset for training next generation image-text models,” NeurIPS Datasets and Benchmarks, 2022. paper