RLHF

Reinforcement learning trains a model by reward: the model produces an output, the output gets a score, and the model is adjusted so that higher-scoring outputs become more likely. In RLHF the score comes from a reward model, a second model trained to predict which output people would prefer. OpenAI’s InstructGPT used three steps (Ouyang et al., 2022). GPT-3 was first fine-tuned on example answers written by labelers. Labelers then ranked several of the model’s outputs for the same prompt, and a reward model was trained on those rankings. Finally, the model was trained with a reinforcement learning algorithm called PPO to raise the reward model’s score, with a penalty for moving far from the step-one model “to mitigate over-optimization of the reward model”. In the paper’s human evaluations, labelers preferred the outputs of a 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3.

The reward model stands in for human judgment, so a model trained against it can find outputs that score well without being what people wanted, a form of reward hacking. Gao, Schulman and Hilton (2022) measured this with a fixed “gold-standard” reward model in the role of the human raters: as training raised the score from a learned stand-in for that model, the gold score first rose and then fell. The raters’ own preferences carry biases too. Sharma et al. (2023) found that in existing human preference data, a response that matches the user’s views is more likely to be preferred, and they link this to sycophancy, a model’s tendency to give answers that match the user’s beliefs over truthful ones. For example, a model trained this way may lean toward agreeing with a trader’s stated view of a market. The InstructGPT authors also note that their method aligns GPT-3 “to the stated preferences of a specific group of people (mostly our labelers and researchers)”.

Training with RLHF changes which outputs a model is likely to produce, and the InstructGPT authors report that their models “can still fail to follow instructions”. Mangrove enforces trading limits, including position sizing rules, daily loss limits, drawdown limits and circuit breakers, in code outside the model, so those limits do not depend on how the model was trained.

Deep reinforcement learning from human preferences (Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei, June 2017) trained agents in Atari games and simulated robot tasks from people’s choices between pairs of short video clips of the agent’s behavior. A simulated robot learned to do backflips from 900 such queries, answered in less than an hour. The authors say their algorithm “follows the same basic approach as Akrour et al. (2012)”, part of an older line of work on reinforcement learning from human ratings or rankings. Fine-Tuning Language Models from Human Preferences (Ziegler et al., September 2019) applied the method to pretrained language models for text continuation and summarization. Training language models to follow instructions with human feedback (Ouyang et al., OpenAI, March 2022), the InstructGPT paper, applied it to make GPT-3 follow written instructions.