How Can Internet Feedback Train Reward Models?

TL;DR

Internet ratings can train reward models to predict human preferences at lower cost and across more tasks than carefully managed interactive feedback. A GPT-2 XL evaluator trained on Reddit writing comparisons reached 74.2 percent accuracy on a difficult test set, but its learned preferences reflect Reddit’s users and require balancing, broader representation, and carefully constructed feedback.

Transcript

hey everyone i'm jonathan ward and i've been mentored by john shulman um over the past six months that amongst i've explored the possibility for large-scale reward modeling um so what this means in practice is how do we learn what people want and then build models that are better able to do that so this uh this question really starts with what shou... Read More

Key Insights

• Reward modeling is a method for learning what people want from their ratings or comparisons, then using the learned preferences to improve model behavior on tasks where a precise objective is difficult to specify.
• Formal tasks are easier to evaluate because correct behavior can be specified clearly, while informal tasks such as writing, summarization, and translation involve nuanced judgments that are difficult to capture fully with a handcrafted function.
• Interactive feedback is accurate when researchers and contractors develop a shared understanding of the desired preferences, but collecting it is expensive. Internet feedback may be less costly, more abundant, and available across a wider variety of tasks.
• Task-oriented internet communities provide clearer supervision than generic likes because their feedback evaluates outputs produced for recognizable inputs. Reddit subreddits can supply structured examples whose scores reflect the aggregate preferences of their participating communities.
• The project’s system contains three roles: a generative model that writes a response to a prompt, an evaluative model that chooses between two responses, and an agent that starts from the generator and improves using evaluator feedback.
• The difficult evaluation set controls for possible shortcuts by comparing responses of roughly similar length that were posted at roughly the same time. On unseen comparisons from this filtered set, the trained reward model reached 74.2 percent accuracy.
• Larger models learn preferences faster from the available comparisons, while increasing the number of training samples continues to improve results, especially for smaller models. The largest tested model, GPT-2 XL, eventually saturated near 74.2 percent accuracy.
• Internet feedback can function like pre-training for preferences, while carefully gathered interactive feedback can function like fine-tuning. This combination could provide broad initial capabilities, followed by targeted feedback that better reflects relevant expertise and groups.

Questions & Answers

Q: What is large-scale reward modeling?

Large-scale reward modeling is an approach for learning what people prefer and using that learned signal to improve model outputs. It is intended for informal tasks where correct behavior cannot be expressed easily as a precise function. Instead of defining quality directly, researchers train an evaluator from human ratings or comparisons and use its judgments as feedback for another model.

Q: Why use internet feedback to train reward models?

Internet feedback is potentially less expensive and more abundant than feedback collected through hired contractors. It may also cover many different tasks because ratings and comparisons already exist across online communities. The tradeoff is reduced control: researchers receive the preferences expressed by existing users rather than ensuring that feedback providers share a carefully defined standard or represent every relevant group.

Q: How was Reddit used to learn writing preferences?

The project focused on r/WritingPrompts, a community where users respond to specific prompts with short stories. Each response receives upvotes and downvotes that produce a score, providing an aggregate signal about community preferences. These scores were converted into comparisons between responses, allowing an evaluative model to learn which story the community was more likely to prefer for a given prompt.

Q: What models were included in the reward modeling system?

The system included a generative model, an evaluative model, and an agent model. The generator received a writing prompt and produced a story. The evaluator received the prompt plus two responses and selected the better response. The agent began from the generative model and was trained further using judgments from the evaluator while competing outputs were compared against each other.

Q: How accurate was the Reddit preference model?

The final reward model achieved 74.2 percent accuracy on a difficult test set of comparisons it had not seen during training. The test set was filtered to reduce shortcuts, including preferences based merely on response length or posting time. The result must also be interpreted in light of noisy aggregate preferences collected across 10 years of Reddit data, which may vary over time.

Q: How did model size and training data affect accuracy?

Larger models learned more quickly from the comparison data, suggesting that they could extract more useful meaning from each example. Adding training samples also produced continued gains, although the benefit was most pronounced for smaller models. GPT-2 XL, the largest model trained in the project, learned fastest but eventually saturated at approximately 74.2 percent accuracy on the filtered evaluation set.

Q: How could reward models transfer across different tasks?

A proposed next step is to combine feedback from many subreddits and tasks, then evaluate the resulting reward model on a task it has not encountered before. Such transfer would be useful when a new task lacks Reddit-scale preference data and collecting extensive interactive feedback is too expensive. Internet signals could provide broad preparation before limited task-specific evaluation or training.

Q: What are the risks of learning preferences from Reddit?

Reddit does not represent the entire global population, so a reward model trained on its feedback learns preferences shaped by a skewed group of users and communities. A model intended to represent a broader judgment of story quality would require a more balanced dataset. Carefully gathered feedback could add writing expertise and incorporate people with different influences and backgrounds.

Summary & Key Takeaways

• Reward modeling offers a way to specify desirable behavior in informal tasks where writing an exact objective is difficult. Instead of relying on functions that approximate quality, a model can learn from people comparing or rating outputs. Internet feedback could provide this preference data at greater scale and lower cost than interactive collection.

• The project uses Reddit’s r/WritingPrompts community, where people respond to prompts and receive scores derived from upvotes and downvotes. A generative model writes stories, an evaluative model compares two responses, and an agent begins with the generator before receiving further training from the evaluator’s judgments about which story is better.

• The evaluator achieved 74.2 percent accuracy after confounders involving response length and posting time were removed. Larger models learned faster, while additional samples particularly benefited smaller models. Future work could combine feedback across subreddits, test transfer to unseen tasks, and supplement broad internet data with carefully designed interactive feedback from representative or expert participants.