InstructGPT: Learning what users prefer

The event and the specific claim

On 4 March 2022, a team of twenty researchers led by Long Ouyang at OpenAI released a paper titled "Training language models to follow instructions with human feedback." The InstructGPT paper reported that human labelers, asked to compare responses, preferred the outputs of a 1.3 billion parameter InstructGPT model to those of the 175 billion parameter GPT-3, despite the smaller model having roughly one hundred times fewer parameters. That is the narrow, documented result: on a distribution of prompts drawn from OpenAI's API, a fine-tuning procedure produced answers people found more useful than a far larger base model produced. OpenAI summarized the work for a general audience in a companion note, Aligning language models to follow instructions, and deployed the resulting models as the default behind its API.

The broad historical judgment is a separate claim, and it should be stated carefully. InstructGPT did not invent reinforcement learning from human feedback, and it did not by itself create the conversational assistant. What it did was demonstrate, at the scale of a frontier language model and on real user prompts, that aligning behavior to human preferences could matter more for perceived quality than adding parameters. That demonstration reorganized how the field thought about the last stage of building a language model, the stage now usually called post-training. Keeping the measured result separate from the trajectory it set is the thread this report follows.

The constraint: a next-token predictor is not an intent-follower

The problem InstructGPT addressed had been visible since GPT-3. Brown and colleagues' few-shot learning paper showed that a 175 billion parameter autoregressive model could perform many tasks described in a prompt, without task-specific parameter updates. But the training objective was to predict the next token in web text, and predicting a plausible continuation of an internet document is not the same as doing what a user asks. A model trained that way will continue a prompt in a way that is fluent and on topic yet still ignores the instruction, invents facts, or produces harmful content, because all of those patterns are well represented in its training data.

The InstructGPT authors frame this as a mismatch between the training objective and the user's intent. A language model is optimized to imitate text; a helpful assistant needs to be helpful, honest, and harmless, three goals the authors adopt explicitly. Closing that gap is not a matter of scale alone, because a bigger imitator is still an imitator. The bet the paper makes is that the missing ingredient is a signal about what people actually want, collected from people and fed back into training. That reframing, treating alignment to intent as its own optimization problem rather than a side effect of pretraining, is the conceptual core of the event.

The three-step recipe: demonstrations, a reward model, and PPO

The method has three stages, and each was a known idea assembled into a working pipeline. First, OpenAI hired a team of about forty contractors and had them write high-quality demonstrations of the desired behavior for prompts submitted to the API. These demonstrations were used for supervised fine-tuning of GPT-3, producing a policy that already followed instructions better than the base model. Second, for each of many prompts the fine-tuned model sampled several candidate outputs, and labelers ranked them from best to worst. Those rankings trained a separate reward model to predict which response a human would prefer. Third, that reward model became the objective for reinforcement learning: the policy was optimized to produce responses the reward model scored highly, using Schulman and colleagues' Proximal Policy Optimization, with a penalty that keeps the tuned policy from drifting too far from the supervised starting point.

A few mechanics are worth stating precisely, because they explain why the pipeline is stable rather than magical. The prompt sets are modest: the supervised stage used demonstrations for prompts on the order of thirteen thousand, the reward model trained on comparisons over roughly thirty-three thousand prompts, and the reinforcement-learning stage ran on around thirty-one thousand more. For each prompt the policy sampled several candidate outputs, between four and nine, and labelers ranked the whole set at once, which yields many pairwise comparisons per prompt and makes the labeling efficient. The reward model that learns from those rankings is itself a language model (a 6 billion parameter variant), and the Kullback-Leibler penalty against the supervised policy is what stops the reinforcement-learning stage from collapsing onto degenerate high-reward text. These are the load-bearing details behind the clean three-box diagram.

None of the three steps was novel in isolation. Learning a reward from human comparisons and optimizing a policy against it was demonstrated by Christiano and colleagues on simulated robotics and Atari in 2017. Applying that loop to a language model came from Ziegler and colleagues' work on fine-tuning language models from human preferences in 2019, and it was shown to produce genuinely better text in Stiennon and colleagues' study of summarization from human feedback in 2020, where a tuned model's summaries were preferred to the human-written reference summaries. InstructGPT's contribution was to run this pipeline on general instruction following at the scale of GPT-3, using prompts that real users had typed rather than a single academic task.

What the numbers actually showed

The headline comparison is the preference win of the 1.3 billion parameter InstructGPT over the 175 billion parameter GPT-3, and it should be read as a preference judgment on a specific prompt distribution, not as a statement that the small model is better at everything. The paper is careful to report a cluster of task-specific improvements alongside it. On truthfulness, measured with the TruthfulQA benchmark of Lin, Hilton, and Evans, the 175 billion parameter InstructGPT produced truthful and informative answers roughly twice as often as GPT-3. On hallucination in closed-domain tasks, where the model should not add information, InstructGPT fabricated content about 21 percent of the time against roughly 41 percent for GPT-3. On toxicity, evaluated with prompts that ask for a respectful continuation, InstructGPT generated about 25 percent fewer toxic outputs than GPT-3.

These are the kinds of numbers that make the case concrete, and they are also the kind that invite overreading. A model that is preferred more often, hallucinates less on a defined task, and is less toxic under a respectful instruction is a better assistant on those axes; it is not a more truthful oracle in general, and the paper does not claim otherwise. Human evaluation itself has a distribution: the labelers who rated outputs were, by design, aligned with the labelers who produced the training data, and the paper reports that held-out labelers who did not contribute to training preferred InstructGPT at a similar rate, which is evidence the gains generalize beyond the specific annotators. Reported this way, the evidence supports a strong but bounded conclusion.

The alignment tax and how the paper paid it down

A recurring worry about preference tuning is that optimizing for what labelers like might degrade raw capability. The paper observed exactly this: the reinforcement learning stage caused performance regressions on several standard public NLP benchmarks, an effect the authors call an alignment tax. Their fix was to modify the reinforcement learning objective to also include a fraction of the original pretraining gradient, a variant they label PPO-ptx, which recovered most of the lost benchmark performance while keeping the preference gains. This detail matters because it separates two claims that are easy to conflate: that alignment tuning changes behavior in ways people prefer, and that it necessarily costs general capability. The paper's answer is that the tax is real but can be reduced by mixing the two objectives.

There is a second, quieter number worth noting. The compute and data spent on this alignment stage were tiny compared with pretraining: a labeling operation of roughly forty people and prompt sets on the order of tens of thousands of examples, against the hundreds of billions of tokens used to train GPT-3. The economic asymmetry is part of why the result landed so hard. If a small, cheap final stage can move human preference more than a large increase in model size, then the marginal return on that stage is enormous, and every serious lab has a reason to invest in it.

Instruction tuning without preferences: the competing approach of its era

InstructGPT was not the only 2021 to 2022 answer to "make the model follow instructions." A parallel line used supervised instruction tuning without any reward model or reinforcement learning. Wei and colleagues' FLAN work fine-tuned a large model on a collection of existing NLP datasets reformatted as natural-language instructions, and found this improved zero-shot performance on held-out task types. Sanh and colleagues' T0 took a similar multitask prompted-training route and showed strong zero-shot generalization from a much smaller model. Both are instruction-following systems, and both rely only on supervised learning over curated tasks.

The contrast with InstructGPT is instructive and is the right competing baseline to keep in view. Supervised instruction tuning teaches a model to imitate correct answers on tasks someone has already assembled and labeled; it is bounded by the coverage and quality of that task collection. Preference-based tuning instead learns a model of human judgment and optimizes against it on open-ended prompts, including prompts with no single correct answer, such as "write a poem" or "explain this to a child." One approach scales by adding more labeled tasks; the other scales by collecting more comparisons of open-ended behavior. The two are not mutually exclusive, and later systems combined them, but in 2022 they represented genuinely different theories of what an assistant needs.

The downstream influence: from the API to ChatGPT and beyond

The most visible descendant arrived nine months later. When OpenAI released ChatGPT in November 2022, its own announcement described the system as a sibling model to InstructGPT, trained with the same reinforcement learning from human feedback method and adapted for dialogue. The Introducing ChatGPT post makes the lineage explicit: the conversational product that reached a mass audience was, in method, the InstructGPT recipe applied to chat. This is the clearest case where a documented research result feeds directly into a documented product, and it is why InstructGPT is often treated as the technical prehistory of the assistant era.

The influence was not confined to OpenAI. Anthropic's work on a helpful and harmless assistant, posted weeks after InstructGPT, built the same preference-modeling and reinforcement-learning loop and studied how helpfulness and harmlessness trade off. Anthropic then extended it in Constitutional AI, which replaces much of the human harmlessness labeling with model-generated feedback guided by written principles. Open models adopted the recipe too: Llama 2's chat models from Meta used supervised fine-tuning followed by reinforcement learning from human feedback, documenting the same three-stage shape. By 2023 the method had also begun to simplify: Direct Preference Optimization by Rafailov and colleagues showed that the same preference data could tune a model with a single classification-style loss, removing the separate reward model and the reinforcement-learning loop. That InstructGPT's exact machinery is already being replaced is a sign of how central its goal became, not a sign that it failed.

The honest limits: whose preferences, and preference is not truth

The paper is unusually candid about what its own method does not deliver, and those caveats are the most important part of placing the event. First, the model is aligned to a particular set of humans: about forty contractors following instructions written by a small group of researchers, working mostly in English. The authors say plainly that InstructGPT is aligned to the preferences of its labelers and the researchers who designed the task, not to some abstract "humanity." Whose values get encoded is a design choice, and a different pool of labelers or a different rubric would produce a different model.

Second, and more fundamental, optimizing for human preference is not the same as optimizing for truth or safety. Labelers prefer answers that look confident, complete, and helpful, which can reward a fluent and plausible falsehood over a hedged correct answer. InstructGPT still makes up facts, still produces biased or toxic text when explicitly prompted to, and still makes simple errors. There is also a structural failure mode: because the reward model is an imperfect proxy for human judgment, pushing the policy too hard against it can produce outputs that score well on the proxy while drifting from what people actually want, a form of over-optimization that later work studies directly. Preference tuning changes behavior in measurable, valuable ways; it does not certify a model as honest, unbiased, or aligned with any given user.

How to place InstructGPT in AI history

The defensible reading is that InstructGPT is a pivotal engineering demonstration rather than an origin point. Its predecessors supplied every component: reward learning from comparisons, preference tuning of language models, PPO, and the base model itself. Its descendants, from ChatGPT to Constitutional AI to Direct Preference Optimization, reused the goal while changing the ingredients. What the March 2022 paper contributed was proof, on a frontier model and on genuine user prompts, that a cheap preference-tuning stage could beat scale on the metric that matters to a user, and a clear, reproducible three-step template for doing it.

It also helps to name what InstructGPT is not, because the popular telling tends to inflate it. It is not the first instruction-following model, it is not the first use of human feedback in machine learning, and it is not a product: the paper describes a research pipeline and an evaluation, and the API deployment and the later chat interface were separate engineering efforts. Nor did it settle any of the questions it raised. The choice of whose preferences to encode, the gap between what raters like and what is true, and the tendency of any learned reward to be gamed are all live research problems that InstructGPT surfaced rather than solved. Its importance is precisely that it turned a diffuse aspiration, "make the model do what users want," into a concrete, measured, repeatable procedure that others could copy, critique, and improve.

A useful way to navigate outward from this event is to hold three questions apart. Which earlier idea supplied a necessary part, such as the reward-from-comparisons loop or PPO? Which later system reused the recipe while changing an ingredient, such as ChatGPT changing the format to dialogue or DPO removing the reward model? And which limitation still required new work, such as whose preferences to encode, how to keep preference from diverging from truth, and how to prevent over-optimization? Answered that way, InstructGPT reads not as the moment machines learned to follow instructions, but as the moment the industry adopted a specific, improvable method for teaching them to, and began arguing productively about its costs.

References

• Ouyang et al. (2022). Training language models to follow instructions with human feedback.
• OpenAI (2022). Aligning language models to follow instructions.
• Brown et al. (2020). Language Models are Few-Shot Learners.
• Christiano et al. (2017). Deep reinforcement learning from human preferences.
• Ziegler et al. (2019). Fine-Tuning Language Models from Human Preferences.
• Stiennon et al. (2020). Learning to summarize from human feedback.
• Schulman et al. (2017). Proximal Policy Optimization Algorithms.
• Lin, Hilton and Evans (2021). TruthfulQA: Measuring How Models Mimic Human Falsehoods.
• Wei et al. (2021). Finetuned Language Models Are Zero-Shot Learners.
• Sanh et al. (2021). Multitask Prompted Training Enables Zero-Shot Task Generalization.
• Bai et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.
• Bai et al. (2022). Constitutional AI: Harmlessness from AI Feedback.
• Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models.
• Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model.
• OpenAI (2022). Introducing ChatGPT.