Fine-Tuning can Distort Pretrained Features (Kumar et al.)
Optional · paper guide 2 of 2 for DLCV 01A · Training Deep Vision Models in Practice. Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
Given a good pretrained model, should you train only a new linear head (linear probing) or update all weights (fine-tuning)? Fine-tuning is known to be more accurate on data like the training data. Kumar et al. asked what happens on shifted data [35].

Figure source: [35], arXiv:2202.10054v1, PDF page 2. Original PDF.
Method
When fine-tuning starts from a fixed or random head, the head and the feature extractor are optimized together. The features of the in-distribution training inputs are updated to suit the head, while the features of out-of-distribution inputs change less, so the final head fits the distorted in-distribution features and works poorly on the others. The authors prove this tradeoff for overparameterized two-layer linear networks and show that, in that setting, early stopping does not fix it: fine-tuning never passes through parameters that do better out of distribution than linear probing. Their remedy is to train the head first (linear probing), then fine-tune everything starting from that head (LP-FT).
Results
On ten distribution-shift benchmarks, fine-tuning was on average 2% more accurate in distribution but 7% less accurate out of distribution than linear probing, and LP-FT beat full fine-tuning by about 1% in distribution and 10% out of distribution [35].
Reading the evidence
Look at two columns for every method: in-distribution (ID) and out-of-distribution (OOD) accuracy. A method that wins ID and loses OOD is fitting features specific to the training distribution. The gaps depend on how good the pretrained features are and how large the shift is: the paper’s claim is that fine-tuning can underperform when the features are good and the shift is large, not that it always does.
Problems
The analysis assumes good pretrained features; with poor features or small shifts, plain fine-tuning can be the better choice. LP-FT adds a stage whose length and learning rate need tuning.
Conclusion
It explains why “train the head first” is a sensible default when a robust model matters, and why the head’s initialization is part of a fine-tuning recipe, not a detail.
References
- T. He, Z. Zhang, H. Zhang, Z. Zhang, J. Xie and M. Li, “Bag of tricks for image classification with convolutional neural networks,” in Proc. IEEE/CVF CVPR, 2019. doi
- A. Karpathy, “A recipe for training neural networks,” blog post, 2019. link
- scikit-learn developers, “load_digits,” scikit-learn documentation, version 1.9.1. link
- M. Buda, A. Maki and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” Neural Networks, vol. 106, 2018. doi
- T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, “Focal loss for dense object detection,” in Proc. IEEE ICCV, 2017. doi
- A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin and A. A. Kalinin, “Albumentations: Fast and flexible image augmentations,” Information, vol. 11, no. 2, 2020. doi
- C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” Journal of Big Data, vol. 6, 2019. doi
- K. He, X. Zhang, S. Ren and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proc. IEEE ICCV, 2015. doi
- P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia and K. He, “Accurate, large minibatch SGD: Training ImageNet in 1 hour,” arXiv:1706.02677 (technical report), 2017. arXiv
- S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. ICML, PMLR 37, 2015. link
- Y. Wu and K. He, “Group normalization,” in Proc. ECCV, 2018. doi
- Y. Bengio, “Practical recommendations for gradient-based training of deep architectures,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
- L. N. Smith, “Cyclical learning rates for training neural networks,” in Proc. IEEE WACV, 2017. doi
- L. N. Smith, “A disciplined approach to neural network hyper-parameters: Part 1 — learning rate, batch size, momentum, and weight decay,” US Naval Research Laboratory Technical Report 5510-026, arXiv:1803.09820, 2018. arXiv
- I. Loshchilov and F. Hutter, “SGDR: Stochastic gradient descent with warm restarts,” in Proc. ICLR, 2017. arXiv
- L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Proc. SPIE 11006, 2019. doi
- PyTorch contributors, “torch.optim.lr_scheduler.OneCycleLR,” PyTorch 2.15 documentation, 2026. link
- K. He, X. Zhang, S. Ren and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE CVPR, 2016. doi
- J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE CVPR, 2009. doi
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015. arXiv
- I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019. arXiv
- PyTorch contributors, “torch.optim.Adam,” PyTorch 2.15 documentation, 2026. link
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, 2014. link
- G. Huang, Y. Sun, Z. Liu, D. Sedra and K. Q. Weinberger, “Deep networks with stochastic depth,” in Proc. ECCV, 2016. doi
- L. Prechelt, “Early stopping — but when?,” in Neural Networks: Tricks of the Trade, 2nd ed., LNCS 7700, Springer, 2012. doi
- V. Godbole, G. E. Dahl, J. Gilmer, C. J. Shallue and Z. Nado, “Deep learning tuning playbook,” Google Research, GitHub repository, 2023. link
- B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” SIAM Journal on Control and Optimization, vol. 30, no. 4, 1992. doi
- P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in Proc. UAI, 2018. arXiv
- D. Morales-Brotons, T. Vogels and H. Hendrikx, “Exponential moving average of weights in deep learning: Dynamics and benefits,” Transactions on Machine Learning Research, 2024. arXiv
- P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, “Mixed precision training,” in Proc. ICLR, 2018. arXiv
- PyTorch contributors, “Automatic mixed precision package — torch.amp,” PyTorch 2.15 documentation, 2026. link
- J. Yosinski, J. Clune, Y. Bengio and H. Lipson, “How transferable are features in deep neural networks?,” in Advances in Neural Information Processing Systems 27, 2014. link
- E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022. arXiv
- J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in Proc. ACL, 2018. doi
- A. Kumar, A. Raghunathan, R. Jones, T. Ma and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” in Proc. ICLR, 2022. arXiv
- C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in Proc. ICLR, 2017. arXiv
- R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, 2020. doi
- PyTorch contributors, “Reproducibility,” PyTorch 2.15 documentation, 2026. link
- Y. Xin, J. Yang, S. Luo, Y. Du, Q. Qin, K. Cen, Y. He, Z. Zhang, B. Fu, X. Yang, G. Zhai, M.-H. Yang and X. Liu, “Parameter-efficient fine-tuning for pre-trained vision models: A survey and benchmark,” arXiv:2402.02242, 2024. arXiv
- G. E. Dahl, F. Schneider, Z. Nado, N. Agarwal, C. S. Sastry, P. Hennig, S. Medapati, R. Eschenhagen, P. Kasimbeg, D. Suo et al., “Benchmarking neural network training algorithms,” arXiv:2306.07179, 2023. arXiv
- P. Kasimbeg, F. Schneider, R. Eschenhagen, J. Bae, C. S. Sastry, M. Saroufim, B. Feng, L. Wright, E. Z. Yang, Z. Nado, S. Medapati, P. Hennig, M. Rabbat and G. E. Dahl, “Accelerating neural network training: An analysis of the AlgoPerf competition,” in Proc. ICLR, 2025. arXiv
- A. Defazio, X. A. Yang, H. Mehta, K. Mishchenko, A. Khaled and A. Cutkosky, “The road less scheduled,” in Advances in Neural Information Processing Systems 37, 2024. link
- S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao et al., “MiniCPM: Unveiling the potential of small language models with scalable training strategies,” in Proc. Conference on Language Modeling (COLM), 2024. arXiv
- A. Hägele, E. Bakouch, A. Kosson, L. Ben Allal, L. Von Werra and M. Jaggi, “Scaling laws and compute-optimal training beyond fixed training durations,” in Advances in Neural Information Processing Systems 37, 2024. link
- M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. Gontijo Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong and L. Schmidt, “Robust fine-tuning of zero-shot models,” in Proc. IEEE/CVF CVPR, 2022. doi
- M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith and L. Schmidt, “Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time,” in Proc. ICML, PMLR 162, 2022. link
- PyTorch contributors, “Automatic mixed precision examples,” PyTorch 2.15 documentation, 2026. link
- PyTorch contributors, “DistributedDataParallel,” PyTorch 2.15.0 source documentation, 2026. Official source.