Full search page →

InstructGPT — Training Language Models to Follow Instructions with Human Feedback

Optional · paper guide 1 of 1 for NLP 09 · Language Modeling with Transformers (Natural Language Generation). Citation numbers refer to the reference list at the bottom of this page.

Navigate: this page and this track: Navigate: this page and this track

Challenge

Predicting likely text does not directly train a model to follow a user’s instruction. Ouyang et al. study how demonstrations and ranked responses can change a pretrained language model’s behavior [25]. Their thesis is that relatively small amounts of human feedback can substantially improve preferences on the evaluated prompt distribution. First understand this chapter’s autoregressive likelihood and supervised fine-tuning. Keep “preferred response,” “factually correct response” and “safe response” as separate evaluation questions.

Method

Original three-stage InstructGPT workflow: demonstrations, reward modeling and policy optimization
Source: Ouyang et al., NeurIPS 2022, Figure 2, PDF p. 4. Left: fit demonstrations. Middle: learn a reward from ranked responses. Right: use that learned reward to update the policy. Human ranking does not occur at every policy-gradient step. [25]

Supervised fine-tuning (SFT) first fits labelers’ example responses. A reward model then learns to score outputs from pairwise preferences derived from rankings. For one preferred/dispreferred pair, its loss has the form

ℓRM=−log⁡σ(rθ(x,yw)−rθ(x,yl)).\ell_{\mathrm{RM}}=-\log\sigma\left(r_\theta(x,y_w)-r_\theta(x,y_l)\right).

Here xx is a prompt, ywy_w the preferred response, yly_l the less-preferred response, and rθr_\theta the scalar reward model with parameters θ\theta. The sigmoid σ\sigma maps the score difference to a preference probability; ℓRM\ell_{\mathrm{RM}} is the pair’s negative log-likelihood. Training aggregates such pairs rather than requiring a human to assign an absolute numerical quality score.

The policy is initialized from SFT and optimized with proximal policy optimization (PPO) against the learned reward. A penalty discourages moving too far from the SFT policy. PPO-ptx additionally mixes in gradients from the pretraining objective to reduce capability regressions. In Figure 2, the demonstration model, reward model and updated policy have different jobs; the reward model is not the response generator at deployment. A divergence penalty constrains behavioral change, not factual truth.

Results

The paper reports that labelers prefer outputs from a 1.3B-parameter InstructGPT model over those from the 175B GPT-3 baseline on the studied API prompt distribution. Read Figure 1 carefully: its plotted win rates use a common 175B SFT comparison model, rather than displaying every possible pairwise contest. This is preference evidence for that prompt and labeler population, not proof that the smaller model dominates every benchmark. The collected prompts are overwhelmingly English, further limiting the scope of the claim.

Ablations

Compare the SFT, PPO and PPO-ptx variants to separate demonstrations, reward optimization and the extra pretraining mixture. Section 4.2 shows that the mixture mitigates several public-benchmark regressions without eliminating all deficits. Truthfulness and toxicity tests also depend on model size and prompting; the toxicity advantage under a respectful instruction does not simply transfer to unprompted generation. These are reasons to evaluate several behaviors, not collapse them into the reward score.

Conclusion

The reusable distinction is between learning to imitate demonstrations, learning to rank responses, and optimizing a generator against the learned ranker. Trace which data supervises each stage before reading the preference plot. For a different objective that trains representations rather than instruction-following generation, continue to NLP 10.

Problems

Preferences reflect selected labelers and collection procedures. A policy can exploit imperfections in its reward model, and persuasive or agreeable text can still be false. Distribution shift, multilingual prompts and adversarial requests require their own evidence. This historical three-stage recipe is not a complete specification of current deployed assistants, and the paper does not show that human-feedback training solves hallucination, bias or alignment in general.

References

  1. Masato Hagiwara, Real-World Natural Language Processing: Practical Applications with Deep Learning, Chapters 5, 8, 9, 10 and 11, Manning, 2021. Publisher · Training practices · Deployment
  2. Susan McRoy, Principles of Natural Language Processing, open textbook, Chapters 6 and 9, 2021. Benchmark chapter · Generation chapter
  3. Ashish Vaswani et al., “Attention Is All You Need,” NeurIPS, 2017. Paper
  4. Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, “Improving Language Understanding by Generative Pre-Training,” OpenAI technical report, 2018. Paper
  5. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever, “Language Models are Unsupervised Multitask Learners,” OpenAI technical report, 2019. Paper
  6. Tom B. Brown et al., “Language Models are Few-Shot Learners,” NeurIPS, 2020. Paper
  7. Jared Kaplan et al., “Scaling Laws for Neural Language Models,” arXiv:2001.08361, 2020. arXiv
  8. Jordan Hoffmann et al., “Training Compute-Optimal Large Language Models,” arXiv:2203.15556, 2022; published as “An empirical analysis of compute-optimal large language model training,” NeurIPS, 2022. arXiv
  9. Ilya Loshchilov and Frank Hutter, “Decoupled Weight Decay Regularization,” ICLR, 2019. Paper
  10. Dan Hendrycks and Kevin Gimpel, “Gaussian Error Linear Units (GELUs),” arXiv:1606.08415, 2016, revised 2023. Paper
  11. Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, “Layer Normalization,” arXiv:1607.06450, 2016. Paper
  12. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi, “The Curious Case of Neural Text Degeneration,” ICLR, 2020. Paper
  13. Angela Fan, Mike Lewis and Yann Dauphin, “Hierarchical Neural Story Generation,” ACL, 2018. Paper
  14. Jade Goldstein and Jaime Carbonell, “Summarization: (1) Using MMR for Diversity-Based Reranking and (2) Evaluating Summaries,” TIPSTER Phase III Workshop, 1998. Paper
  15. Sam Wiseman, Stuart M. Shieber and Alexander M. Rush, “Challenges in Data-to-Document Generation,” EMNLP, 2017. Paper
  16. Ilya Sutskever, Oriol Vinyals and Quoc V. Le, “Sequence to Sequence Learning with Neural Networks,” NeurIPS, 2014. Paper
  17. Chin-Yew Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” Text Summarization Branches Out, 2004. Paper
  18. Lianmin Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS Datasets and Benchmarks Track, 2023. Paper
  19. Peiyi Wang et al., “Large Language Models are not Fair Evaluators,” ACL, 2024. Paper
  20. Ziwei Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Computing Surveys 55, 2023. Survey
  21. Joshua Maynez, Shashi Narayan, Bernd Bohnet and Ryan McDonald, “On Faithfulness and Factuality in Abstractive Summarization,” ACL, 2020. Paper
  22. Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” NeurIPS, 2020. Paper
  23. OWASP GenAI Security Project contributors, “LLM01: Prompt Injection,” OWASP guidance, accessed 2026. Official guidance
  24. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” ICLR, 2022. Author publication page
  25. Long Ouyang et al., “Training language models to follow instructions with human feedback,” NeurIPS, 2022. Paper
  26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon and Chelsea Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” NeurIPS, 2023. Paper
  27. Wayne Xin Zhao et al., “A Survey of Large Language Models,” Frontiers of Computer Science, 2026. Updated survey
  28. Yupeng Chang et al., “A Survey on Evaluation of Large Language Models,” ACM Transactions on Intelligent Systems and Technology, 2024. Survey
  29. Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie and Ji-Rong Wen, “Pre-Trained Language Models for Text Generation: A Survey,” ACM Computing Surveys, 2024. Publisher · Author manuscript
  30. Patricia Schmidtova et al., “Do My Eyes Deceive Me? A Survey of Human Evaluations of Hallucinations in NLG,” INLG, 2025. Survey
  31. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang and Haofen Wang, “Retrieval-Augmented Generation for Large Language Models: A Survey,” arXiv:2312.10997, revised 2024. Survey
  32. Woosuk Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP, 2023. Conference · Paper
  33. Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Chapters 6, 8 and 11, Cambridge University Press, 2008. Author-hosted book
  34. scikit-learn developers, “Text feature extraction,” scikit-learn documentation, accessed 2026. Documentation
  35. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen and Wen-tau Yih, “Dense Passage Retrieval for Open-Domain Question Answering,” EMNLP, 2020. Paper
  36. Kalervo Järvelin and Jaana Kekäläinen, “Cumulated Gain-Based Evaluation of IR Techniques,” ACM Transactions on Information Systems 20(4), 422–446, 2002. Publisher