How to Make Reinforcement Learning Fast and Cheap

TL;DR

Asynchronous pipeline RL keeps GPUs busy by dedicating some to sampling and some to training, propagating new weights mid-sample through in-flight updates. This avoids the idle GPUs of synchronous RL, where step time is dictated by the slowest straggler sample. The tradeoff: tolerating more staleness speeds runs but raises importance-ratio variance, which can destabilize learning.

Transcript

Hey everyone, it's great to meet you all. Really great to be here today. My name is Rhythm. This is my co-founder Lyndon. Our third co-founder, Yash, couldn't make it today, but we're all very excited to be here. Um, three of us were previously researchers at OpenAI, and now we're bringing Frontier AI inside of enterprise at applied compute. Today,... Read More

Key Insights

• Reinforcement learning at Applied Compute works by sampling batches of problems (e.g. four math problems), having an open-source model like GPT-OSS or Llama attempt each 100 times, grading the answers, then reinforcing weights on correct reasoning traces and discouraging incorrect ones.
• Synchronous RL is inefficient because sampling and training happen in lock step, so the whole batch waits for the slowest straggler sample, leaving many GPUs idle until that last completion.
• The long-tail problem is measurable: across 40 arithmetic problems with 32 samples each on Qwen 30B, 99% of samples finished in about 40 seconds but the final 1% took another 80 seconds to complete.
• Asynchronous RL breaks the lock-step condition by allowing training to happen while sampling continues, which keeps GPUs from slacking during the sampling tail.
• Pipeline RL dedicates some GPUs to sampling and some to training; sampling workers never stop, completed samples enter a training queue, and after each train step new weights propagate to all sampling workers via an in-flight weight update.
• In-flight weight updates create stale tokens: a single sample can be generated by multiple policy versions (t, t+1, t+2), so when trained on later it mixes tokens from policies several steps behind the current one.
• Tolerating more staleness reduces idle GPUs, but the policy gradient uses an importance ratio to stay unbiased, and that ratio's variance grows with staleness, making learning unstable and risking divergence.
• Applied Compute's business needs RL runs that are fast (delivered in days), cheap enough for sustainable unit costs, and low-variance in duration so training is reliably fast, not just generally fast, for customers.

Questions & Answers

Q: What is reinforcement learning and how does it train a language model?

Reinforcement learning teaches a model to reason by having it attempt problems many times and reinforcing successful attempts. At Applied Compute, a training step picks a batch of problems (for example four math problems), has an open-source model like a GPT-OSS or Llama model attempt each 100 times generating long reasoning trajectories, grades all the answers, biases the weights to reinforce the thinking trace when correct, and discourages that behavior when incorrect. Over many steps the model becomes very good at the task.

Q: Why is synchronous RL inefficient?

In synchronous RL, sampling and training happen in lock step, so if you train on batches of eight samples you must wait for all eight to finish before training begins, then repeat. Because step times are dictated by whatever sample takes the longest, many GPUs sit idle waiting on the last straggler sample. Applied Compute's measurements showed 99% of samples finishing in about 40 seconds but the final 1% taking another 80 seconds, a long tail that leaves GPUs badly underutilized.

Q: What is asynchronous pipeline RL and how does it work?

Asynchronous pipeline RL breaks the requirement that sampling and training happen in lock step by allowing training while sampling continues. It dedicates some GPUs to sampling and some to training. The sampling workers never stop, constantly doing high-batch-size inference; as samples complete they enter a queue, and training workers pull batches to train on. After training a batch, the training workers propagate the new model weights to all sampling workers in what is called an in-flight weight update, which is what differentiates pipeline RL.

Q: What are stale tokens in RL training?

Stale tokens arise because in-flight weight updates can change a sampling worker's weights while it is still generating a sample. As a result, a single sample can be produced by multiple versions of the policy at time steps t, t+1, and t+2. When that sample is later trained on, some tokens came from a policy three steps behind, some from two steps behind, and the last few from a policy one step behind. This mismatch between the sampling policy and the training policy is called staleness.

Q: What is the tradeoff when tolerating more staleness in RL?

Increasing how much staleness you tolerate reduces idle GPUs and produces faster RL runs, because training workers do not have to wait for in-flight samples to complete before propagating weight updates. However, the policy gradient uses an importance ratio to remain unbiased when sampling from a policy at time t but training with a policy k steps later, and the variance of that ratio increases with staleness. Higher variance can make learning unstable and cause divergence, so there is no free lunch.

Q: What does the importance ratio do in the policy gradient?

The importance ratio adjusts for the fact that a sample was generated by a policy at time step t but is being trained on with a policy at time step t plus k, given k staleness. It is what makes the policy gradient unbiased despite this mismatch. The downside is that as staleness increases, the variance of the importance ratio grows, and higher variance makes learning unstable and can cause the training run to diverge, which is one of Applied Compute's primary research problems.

Q: Why does Applied Compute need low-variance training time estimates?

Applied Compute delivers trained models to customers on the order of days, so runs must be fast and cheap enough for sustainable unit costs. Beyond raw speed, the estimates for how long training jobs will take must be very low variance, because the company does not want to be merely generally fast but reliably fast when working with customers. Predictable, low-variance training durations let them plan customer delivery confidently and scale their use-case-specific training motion sustainably.

Q: How does Applied Compute model the RL system to optimize efficiency?

They treat it as a first-principles systems modeling problem, identifying a cast of characters that describe the system and how they fit together. The pieces include a proxy of compute budget (the number of GPUs, which in the asynchronous setting can be allocated flexibly between training and sampling), the training batch size representing the workload (n problems each sampled multiple times, with harder problems sampled more for diversity), and a proxy of sampling throughput based on how inference engines serve requests using model weights, activations, and the KV cache in GPU memory.

Summary & Key Takeaways

• Applied Compute, founded by three former OpenAI RL researchers, helps enterprises build specialized in-house intelligence that delivers quantitative ROI. RL is the mechanism they use to bring out-of-distribution datasets into distribution, teaching models to reason and excel at company-specific tasks, then deploying with a data flywheel that improves the system with continued use.

• The RL mechanism samples a batch of problems, has an open-source model attempt each many times producing long reasoning trajectories, grades the final answers, and biases the model's weights to reinforce correct thinking traces while discouraging incorrect behavior. Synchronous RL does this in lock step, leaving GPUs idle as they wait on the slowest straggler sample to finish.

• Asynchronous pipeline RL separates sampling and training GPUs, updating sampling-worker weights mid-sample so samples contain stale tokens from multiple policy versions. Higher staleness tolerance reduces idle GPUs but increases importance-ratio variance and instability, creating a core research tradeoff that the team models with first-principles systems analysis over GPU count, batch size, and sampling throughput.