“Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback” was submitted to arXiv on July 27, 2023 by Stephen Casper, Xander Davies, and a large group of co-authors from across the field. It is the standard reference for what is wrong, or at least unsolved, in the method that aligns most frontier language models.
The paper organizes the problems along the three stages of the RLHF pipeline. Gathering human feedback is fallible: labelers disagree, can be fooled, have limited time, and may have biases, so the preference data itself is noisy and incomplete. Training a reward model on that data introduces its own gap, because no reward model perfectly captures human values, and any error becomes a target the policy can exploit - the reward-hacking problem. Finally, optimizing the policy against the reward model is itself unstable and can push the model into strange regions where the reward model’s judgments no longer hold.
Importantly, the authors distinguish problems that are tractable - addressable with better engineering - from ones they argue are fundamental to the RLHF approach and will require different methods to overcome. They also call for auditing standards and greater transparency so these systems can be checked from outside.
For a business reader, this paper is a useful corrective to the impression that alignment is a solved problem. RLHF makes models far more usable, but it rests on a chain of imperfect approximations, and understanding those limits matters for anyone deciding how much to trust a model’s helpful, agreeable answers.