InstructGPT — Training Language Models to Follow Instructions with Human Feedback
Optional · paper guide 1 of 1 for NLP 09 · Language Modeling with Transformers (Natural Language Generation). Citation numbers refer to the reference list at the bottom of this page.
Navigate: this page and this track: Navigate: this page and this track
Challenge
Predicting likely text does not directly train a model to follow a user’s instruction. Ouyang et al. study how demonstrations and ranked responses can change a pretrained language model’s behavior [25]. Their thesis is that relatively small amounts of human feedback can substantially improve preferences on the evaluated prompt distribution. First understand this chapter’s autoregressive likelihood and supervised fine-tuning. Keep “preferred response,” “factually correct response” and “safe response” as separate evaluation questions.
Method

Supervised fine-tuning (SFT) first fits labelers’ example responses. A reward model then learns to score outputs from pairwise preferences derived from rankings. For one preferred/dispreferred pair, its loss has the form
Here is a prompt, the preferred response, the less-preferred response, and the scalar reward model with parameters . The sigmoid maps the score difference to a preference probability; is the pair’s negative log-likelihood. Training aggregates such pairs rather than requiring a human to assign an absolute numerical quality score.
The policy is initialized from SFT and optimized with proximal policy optimization (PPO) against the learned reward. A penalty discourages moving too far from the SFT policy. PPO-ptx additionally mixes in gradients from the pretraining objective to reduce capability regressions. In Figure 2, the demonstration model, reward model and updated policy have different jobs; the reward model is not the response generator at deployment. A divergence penalty constrains behavioral change, not factual truth.
Results
The paper reports that labelers prefer outputs from a 1.3B-parameter InstructGPT model over those from the 175B GPT-3 baseline on the studied API prompt distribution. Read Figure 1 carefully: its plotted win rates use a common 175B SFT comparison model, rather than displaying every possible pairwise contest. This is preference evidence for that prompt and labeler population, not proof that the smaller model dominates every benchmark. The collected prompts are overwhelmingly English, further limiting the scope of the claim.
Ablations
Compare the SFT, PPO and PPO-ptx variants to separate demonstrations, reward optimization and the extra pretraining mixture. Section 4.2 shows that the mixture mitigates several public-benchmark regressions without eliminating all deficits. Truthfulness and toxicity tests also depend on model size and prompting; the toxicity advantage under a respectful instruction does not simply transfer to unprompted generation. These are reasons to evaluate several behaviors, not collapse them into the reward score.
Conclusion
The reusable distinction is between learning to imitate demonstrations, learning to rank responses, and optimizing a generator against the learned ranker. Trace which data supervises each stage before reading the preference plot. For a different objective that trains representations rather than instruction-following generation, continue to NLP 10.
Problems
Preferences reflect selected labelers and collection procedures. A policy can exploit imperfections in its reward model, and persuasive or agreeable text can still be false. Distribution shift, multilingual prompts and adversarial requests require their own evidence. This historical three-stage recipe is not a complete specification of current deployed assistants, and the paper does not show that human-feedback training solves hallucination, bias or alignment in general.
References
- Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapters 5, 8, 9, 10 and 11, Manning, 2021. Publisher · Training practices · Deployment
- Susan McRoy, Principles of Natural Language Processing, open textbook, Chapters 6 and 9, 2021. Benchmark chapter · Generation chapter
- Ashish Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
- Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI technical report, 2018. Paper
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever, “Language Models are Unsupervised Multitask Learners,” OpenAI technical report, 2019. Paper
- Tom B. Brown et al., “Language Models are Few-Shot Learners,” NeurIPS, 2020. Paper
- Jared Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
- Jordan Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
- Ilya Loshchilov and Frank Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. Paper
- Dan Hendrycks and Kevin Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016, revised 2023. Paper
- Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi, “The Curious Case of Neural Text Degeneration,” ICLR, 2020. Paper
- Angela Fan, Mike Lewis and Yann Dauphin, “Hierarchical Neural Story Generation,” ACL, 2018. Paper
- Jade Goldstein and Jaime Carbonell, “Summarization: (1) Using MMR for Diversity-Based Reranking and (2) Evaluating Summaries,” TIPSTER Phase III Workshop, 1998. Paper
- Sam Wiseman, Stuart M. Shieber and Alexander M. Rush, “Challenges in Data-to-Document Generation,” EMNLP, 2017. Paper
- Ilya Sutskever, Oriol Vinyals and Quoc V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
- Chin-Yew Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” Text Summarization Branches Out, 2004. Paper
- Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS Datasets and Benchmarks Track, 2023. Paper
- Peiyi Wang et al., “Large Language Models are not Fair Evaluators,” ACL, 2024. Paper
- Ziwei Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Computing Surveys 55, 2023. Survey
- Joshua Maynez, Shashi Narayan, Bernd Bohnet and Ryan McDonald, “On Faithfulness and Factuality in Abstractive Summarization,” ACL, 2020. Paper
- Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS, 2020. Paper
- OWASP GenAI Security Project contributors, “LLM01: Prompt Injection,” OWASP guidance, accessed 2026. Official guidance
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. Author publication page
- Long Ouyang et al., “Training language models to follow instructions with human feedback,” NeurIPS, 2022. Paper
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon and Chelsea Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” NeurIPS, 2023. Paper
- Wayne Xin Zhao et al., “A Survey of Large Language Models,” Frontiers of Computer Science, 2026. Updated survey
- Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Survey
- Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie and Ji-Rong Wen, “Pre-Trained Language Models for Text Generation: A Survey,” ACM Computing Surveys, 2024. Publisher · Author manuscript
- Patricia Schmidtova et al., “Do My Eyes Deceive Me? A Survey of Human Evaluations of Hallucinations in NLG,” INLG, 2025. Survey
- Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang and Haofen Wang, “Retrieval-Augmented Generation for Large Language Models: A Survey,” arXiv:2312.10997, revised 2024. Survey
- Woosuk Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP, 2023. Conference · Paper
- Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Chapters 6, 8 and 11, Cambridge University Press, 2008. Author-hosted book
- scikit-learn developers, “Text feature extraction,” scikit-learn documentation, accessed 2026. Documentation
- Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen and Wen-tau Yih, “Dense Passage Retrieval for Open-Domain Question Answering,” EMNLP, 2020. Paper
- Kalervo Järvelin and Jaana Kekäläinen, “Cumulated Gain-Based Evaluation of IR Techniques,” ACM Transactions on Information Systems 20(4), 422–446, 2002. Publisher