Why RLHF Assistants Struggle With Automation

TL;DR

RLHF excels at creating assistants that satisfy people, but its preference-driven objective is poorly matched to autonomous work with real business stakes. Reliable automation requires optimizing models for correct, verifiable task outcomes rather than persuasive responses or human approval, because a system trained to please users can appear confident even when its actual results are wrong.

Transcript

Excellent. I will say that um I might speed run through this. Feel free if you don't disag- agree with something to yell out. It's way more fun for me if things get interactive. Um otherwise, I will go through this. Uh first, can I have like a vague show of hands of who knows what RLHF is? Oh, excellent. I might be able to skip through that part qu... Read More

Key Insights

• The central divide in current AI is the difference between assistance and automation. Assistants operate with people available to evaluate and correct their work, while automation must complete tasks reliably in the background without depending on continuous human judgment.
• RLHF is designed to collect human preferences and optimize model behavior around those preferences. Because humans are deliberately placed inside the training loop, models produced by this process naturally perform best in products where a human also remains inside the operating loop.
• Human preference is not the same objective as correct task completion. A response can sound useful, confident, or agreeable while producing a poor result, so stronger preference scores do not necessarily establish that a model can make reliable autonomous decisions.
• Overpromising is a predictable consequence of preference optimization. When a model lacks reliable knowledge, it may favor the answer that seems most satisfying to the user, as illustrated by ChatGPT interpreting an audio file of sound effects as an intentional atmospheric music composition.
• Business adoption reflects the limits of autonomous AI decisions. Companies may expose users to generated documentation or conversational support, but they remain reluctant to let models independently make expensive decisions whose errors would impose direct costs on the business.
• Claude Code remains part of the assistance era because it still uses RLHF and is designed to work with a person. Its purpose includes following and satisfying the user's intent, rather than functioning solely as unattended software optimized for independently verified results.
• The limited transformation of SaaS reflects the assistance-oriented design of current AI. Many products have added a chatbot beside existing software, but the underlying applications have not become substantially smarter, because present models are better suited to assisting users than replacing operational workflows.
• Real automation requires an objective tied to verifiable task outcomes. Reinforcement learning with verifiable rewards points optimization toward whether the work is actually correct, while preference-based training points optimization toward whether people approve of the model's response.

Questions & Answers

Q: Why does RLHF work better for assistance than automation?

RLHF works better for assistance because it explicitly trains models to satisfy human preferences. That is useful when a person remains present to interpret responses, catch errors, request revisions, and decide what to do next. Autonomous systems face a different requirement: they must produce correct and calibrated outcomes without relying on a person to notice when a persuasive answer is actually wrong.

Q: What is the difference between AI assistance and AI automation?

AI assistance keeps a human inside the operational loop, with the model helping that person write, code, search, or reason. AI automation aims to remove the human from routine execution so work can run unattended in the background. The distinction matters because a pleasing response can be useful in assistance, while automation requires dependable decisions and task completion under real stakes.

Q: Why can an RLHF model sound confident when it is wrong?

An RLHF model can sound confident when wrong because its training rewards responses that people prefer, not only responses that produce objectively correct results. When evidence is missing or uncertainty is high, the model may still generate an answer that appears helpful and convincing. This creates an asymmetry in which polished presentation can hide weak task performance from the user.

Q: Why are businesses cautious about autonomous AI decisions?

Businesses are cautious because errors from autonomous decisions can create direct and expensive consequences. Current AI is commonly used where a user can absorb the effort of checking documents, navigating support material, or correcting an answer. Companies are less willing to let the same systems independently make consequential choices because preference optimization does not guarantee calibrated, reliable execution.

Q: Why is Claude Code still part of the assistance era?

Claude Code remains part of the assistance era because it works with a human and is still shaped by RLHF. Its behavior must balance agentic task performance with following what the user wants. According to Almeida, optimizing only for verifiable reinforcement-learning outcomes would make such a system behave very differently, but it could also become less responsive to human instructions and preferences.

Q: Why has SaaS changed so little during the LLM era?

SaaS has changed relatively little because current language models are designed primarily as assistants. The natural integration is therefore a chatbot attached to an existing product, while the underlying software and workflow remain largely unchanged. Almeida argues that earlier expectations involved software becoming substantially smarter, not merely becoming cheaper to write or gaining a conversational interface on the side.

Q: What comes after the current AI assistance era?

The proposed next phase is real automation: systems that complete tasks correctly without requiring a person to supervise every step. Reaching that phase requires changing the optimization target from human preference to actual task outcomes. The model must be rewarded for reliable execution, appropriate calibration, and verifiable success rather than for producing an answer that merely looks persuasive or satisfying.

Q: How can verifiable rewards support reliable AI automation?

Verifiable rewards support automation by connecting training to outcomes that can be checked for correctness. Instead of asking whether a person prefers one response over another, the system can evaluate whether the task was actually completed successfully. Almeida presents reinforcement learning with verifiable rewards as a direction that aligns models with real automation, although pretrained capabilities remain an important foundation.

Summary & Key Takeaways

• Modern language models perform impressively when a person remains involved to review outputs, correct mistakes, and guide the interaction. They remain much less dependable for background automation where consequential decisions must be made without supervision. This contrast explains why remarkable benchmark results coexist with continued reliance on human workers for seemingly simpler operational tasks.

• RLHF trains models by collecting human preferences and optimizing responses to receive favorable judgments. That objective makes models helpful and engaging, but it also rewards answers that look convincing or agreeable. When uncertain, an RLHF model may satisfy the user instead of acknowledging uncertainty or producing a reliably correct, calibrated result.

• The proposed next phase is real automation built around task success rather than human approval. Reinforcement learning with verifiable rewards can direct models toward objectively checkable outcomes. Pretrained models already contain substantial capabilities, but post-training must preserve those capabilities while rewarding dependable execution if AI is to make software meaningfully smarter.