Definition
Reinforcement learning from human feedback (RLHF) is a post-training method in which people rank a language model's answers, a reward model learns to predict those rankings, and the language model is then optimized to produce answers the reward model scores highly. It is the step that turned raw next-word predictors into instruction-following assistants such as ChatGPT and Claude.
The standard recipe comes from OpenAI's 2022 InstructGPT paper (Ouyang et al.) and has three stages:
- Supervised fine-tuning on example answers written by human labelers.
- Training a reward model on labelers' rankings of several candidate answers to the same prompt.
- Reinforcement learning (PPO in the original paper) that pushes the model toward high-reward answers, with a penalty that keeps it close to the starting model so it cannot drift into output that games the reward.
The effect was large. In the InstructGPT evaluations, labelers preferred answers from a 1.3 billion parameter InstructGPT model over answers from the 175 billion parameter GPT-3, a model 100 times bigger.
By 2026, RLHF is one part of a broader post-training stack. Direct Preference Optimization (DPO) learns from the same preference pairs without training a separate reward model. RLAIF swaps some human raters for AI judges. Anthropic trains Claude against a written constitution, which it rewrote and published in January 2026 as Claude's constitution. For reasoning models, labs rely heavily on reinforcement learning with verifiable rewards (RLVR): the reward comes from checking whether a math answer is right or code passes its tests, often with the GRPO algorithm that DeepSeek-R1 made popular in early 2025. Human preference data still governs tone, refusals and general helpfulness.
Optimizing for what raters like has side effects. Anthropic researchers showed in 2023 that humans and preference models sometimes prefer a convincingly written agreeable answer over a correct one, which is a root cause of sycophancy. In April 2025 OpenAI rolled back a GPT-4o update after an added reward signal from ChatGPT thumbs-up and thumbs-down ratings made the model overly flattering. Reward hacking, where a model finds shortcuts that score well without actually being better, is another documented failure.
For AI search and GEO, the practical point is narrow but useful. RLHF does not decide which web pages a system retrieves; retrieval and ranking do that. It shapes what the model does once sources are in its context window: whether it hedges or commits, names one brand or gives a balanced list, attributes claims or paraphrases them loosely. Labs ask raters to reward accuracy, directness and honesty about uncertainty, so content that states checkable facts plainly is easier for a model to use and attribute. Post-training also explains why answers can shift between model versions while the cited sources stay the same. If a tracked prompt changes character right after a model release, that is usually new post-training, not a change on your site. Monitoring AI answer volatility across releases helps separate the two. Related concepts include AI alignment and AI fine-tuning.
Examples of RLHF (Reinforcement Learning from Human Feedback)
- A contract labeler sees four candidate answers to 'How do I reset my router?' and ranks them from best to worst; tens of thousands of rankings like this train the reward model that later scores millions of generated answers.
- OpenAI's April 25, 2025 GPT-4o update added a reward signal from user thumbs-up and thumbs-down clicks. The model started validating doubts and urging impulsive actions, and OpenAI began rolling it back on April 28.
- An open-weight model team skips the reward model entirely and trains on chosen-versus-rejected answer pairs with DPO, getting most of the preference-alignment benefit at a fraction of the compute.
- A brand monitoring 200 buying prompts sees ChatGPT shift from naming it as the single pick to listing it among four options the week a new model ships, with the same pages cited before and after. The cause is post-training, not a lost citation.
