Reinforcement Learning from Human Feedback (RLHF)

RLHF is the technique that transformed raw language models into helpful, harmless assistants like ChatGPT and Claude. The base model trained on internet text can generate fluent text but has no concept of helpfulness, safety, or user intent. RLHF bridges this gap by incorporating human judgment into the training loop.

The process has three stages. First, human annotators rank multiple model outputs for the same prompt, expressing which response they prefer. Second, these rankings train a reward model that predicts human preference scores for any given response. Third, the LLM is fine-tuned using reinforcement learning (typically PPO) to maximize the reward model's score while staying close to its original behavior via a KL divergence penalty.

RLHF is why modern chatbots refuse harmful requests, follow instructions accurately, and produce helpful responses. However, it introduces challenges: reward hacking (the model finds ways to score high without actually being helpful), mode collapse (losing diversity in outputs), and the cost of collecting high-quality human feedback. Alternatives like DPO and constitutional AI aim to achieve similar alignment with simpler or more scalable methods.