Reinforcement Learning 8: What Are the Advanced Topics in Deep RL?

TL;DR

Advanced topics in deep RL include neural function approximation, direct policy optimization, actor-critic methods, trust regions, continuous actions, learned models, experience replay, multi-step returns, and search. The lecture connects these research areas to policy iteration, model-free control, and off-policy learning, while emphasizing efficiency, computational cost, instability, and possible divergence. Read on to see how these ideas fit together and where their tradeoffs arise.

Transcript

let's get started so we discussed a lot of different topics so far item see if I can somehow get this thing out of the way maybe not oh well so what I thought I do today is to go through some of the active research topics right now I won't go into that much breadth there's many more research topics but I thought it would be good to highlight a litt... Read More

Key Insights

• Actions control available evidence: In a bandit problem, selecting an action does more than produce a reward. It also determines which action’s data becomes observable, while information about the unselected actions remains unavailable. Exploration is therefore part of the learning process itself, not a separate preliminary stage before exploitation begins.
• Bandits already support policy gradients: Policy-gradient methods do not require a complex sequential environment to be introduced. The lecture notes that they can already be discussed in the single-state bandit context, alongside greedy, epsilon-greedy, and upper-confidence-bound algorithms. This places direct policy learning within the earliest decision-making framework covered by the course.
• Policy iteration exceeds dynamic programming: Policy evaluation followed by policy improvement is often described using dynamic-programming terminology. The lecture emphasizes that the pattern is substantially broader. It provides the conceptual structure beneath most reinforcement learning algorithms that seek optimal policies, including methods that do not have direct access to a complete model of the environment.
• Monte Carlo targets completed returns: Monte Carlo prediction follows the selected policy until an episode terminates and then calculates the resulting return. That completed return becomes a supervised-style regression target for the value estimate. Policy improvement can remain a separate step, allowing evaluation and improvement to combine again as policy iteration without requiring known transition dynamics.
• Bootstrapping changes update timing: Temporal-difference learning updates one estimate from another existing estimate rather than waiting exclusively for a completed episode return. This bootstrapping principle distinguishes it from the Monte Carlo procedure described in the lecture. It also forms part of the foundation for control variants such as Q-learning, Sarsa, and double Q-learning.
• Off-policy has broader meaning: Learning off-policy does not only mean learning about the greedy policy, even though Q-learning provides that familiar example. The general case occurs whenever the policy being studied differs from the policy that generated the data. Human-generated observations used to examine the consequences of different actions are another example given in the lecture.
• Deep RL depends on approximation: Deep reinforcement learning is presented as the use of deep neural networks for function approximation within reinforcement learning. Neural networks belong to a wider progression that includes tabular, linear, and nonlinear representations. The important shift is not merely using a larger table, but estimating values or policies through a nonlinear approximating system.
• Divergence can appear in toy problems: Instability is not limited to enormous or complicated applications. The lecture notes that very small examples can make reinforcement learning algorithms diverge when function approximation and learning procedures are combined carelessly. These examples reveal fundamental convergence issues, even though carefully configured practical systems can often avoid unreasonable solutions.
• Careful configuration limits instability: The possibility of divergence does not imply that function approximation is unusable. According to the lecture, practical systems often work without moving toward strange solutions when they are set up carefully. This makes system design important because successful use depends on how the learning algorithm and approximation method are combined.
• Least squares trades compute for data: With a linear value function using a feature vector of n components, the lecture states that least-squares methods require n squared computation. A typical temporal-difference update requires n computation. Least-squares prediction can nevertheless be attractive when greater data efficiency is valuable enough to justify the additional computational expense.
• Actor and critic learn together: Actor-critic methods combine a directly learned policy with a learned value function. The actor represents the behavior used to select actions, while the critic evaluates that behavior. Reinforce can be extended using a baseline, bootstrapping, or both, linking direct policy optimization to value-based evaluation within one learning framework.
• Trust regions restrict policy movement: Trust-region methods constrain how far an updated policy moves from the current policy. The lecture associates this restriction with smoother trajectories and better learning. Within direct policy learning and actor-critic approaches, the constraint addresses the difficulty created when an update changes behavior too substantially from one policy version to the next.

Questions & Answers

Q: What are the advanced topics in deep reinforcement learning covered in Reinforcement Learning 8?

The advanced topics include neural function approximation, direct policy optimization, actor-critic methods, trust regions, continuous actions, learned models, experience replay, multi-step returns, and search. They build on earlier material about bandits, Markov decision processes, policy iteration, and model-free control. Together, these methods broaden the available approaches to learning values, policies, and models. The lecture also highlights their costs and risks, including additional computation, instability, possible divergence, and open research questions.

Q: What is policy iteration in reinforcement learning?

Policy iteration alternates policy evaluation with policy improvement. Evaluation estimates how well the current policy behaves, and improvement uses those estimates to select a better policy. Repeating this process provides a general method for obtaining good or optimal policies. Although it is often discussed with dynamic programming, the lecture says the underlying framework supports most reinforcement learning algorithms for finding optimal policies.

Q: How do Monte Carlo and temporal-difference learning differ?

Monte Carlo learning follows a policy until an episode terminates, then calculates the return and uses it as a regression target. Temporal-difference learning instead bootstraps, updating an estimate from another existing estimate. This allows the two approaches to construct value-learning targets differently. The bootstrapping approach also supports control variants discussed in the lecture, including Q-learning, Sarsa, and double Q-learning.

Q: What is the difference between on-policy and off-policy learning?

On-policy learning collects data with the current policy and uses it to estimate that same policy. Off-policy learning studies a policy different from the one that generated the observations. Q-learning is one example because it learns about a greedy policy with respect to current values. The distinction is broader than greedy learning, since data generated by humans can also be used to study what would happen under different actions.

Q: Why can function approximation diverge in reinforcement learning?

Reinforcement learning algorithms can diverge when function approximation and learning procedures are combined or configured carelessly. The lecture notes that this behavior appears even in very small toy examples, demonstrating a fundamental issue rather than only a large-system problem. Deep neural networks add nonlinear function approximation to the reinforcement learning setting. Careful system configuration matters because practical systems can often avoid estimates that move toward unreasonable solutions.

Q: When is least-squares prediction useful?

Least-squares prediction is most notably useful with a linear function approximator. It can exploit the linear structure to learn more efficiently from available data. The tradeoff is computational cost: for a feature vector with n components, the lecture gives n squared computation for least-squares methods and n computation for a typical temporal-difference update. It is therefore relevant when data efficiency justifies more computation.

Q: What does an actor-critic method learn?

An actor-critic method learns a policy and a value function together. The policy is the actor because it selects behavior, while the value function is the critic because it evaluates behavior. Reinforce can be extended with a baseline, bootstrapping, or both to produce this kind of method. The combination connects direct policy optimization with value estimation and can also accommodate continuous actions and trust-region updates.

Q: How do trust-region methods support policy learning?

Trust-region methods limit how much the learned policy can change during an update. This keeps the new policy relatively close to the current one instead of permitting an unrestricted movement. The lecture says that restricting policy movement can produce smoother trajectories and better learning. The method is relevant to direct policy learning and actor-critic systems because large policy changes can substantially alter behavior between updates.

Summary & Key Takeaways

• Defining the lecture’s scope: The lecture begins by positioning its subject as a selective overview of active research in reinforcement learning and deep reinforcement learning. It aims to highlight current activity and identify work that remains interesting, without claiming comprehensive breadth. Multi-agent learning is explicitly omitted because it is both difficult and too large to receive a fair overview in the remaining time. Before addressing advanced topics, the lecture reviews the technical foundations established across the preceding six lectures.

• Revisiting decisions and exploration: The review starts with learning to make decisions through bandit problems, where there is one state but multiple actions. Even in this limited setting, exploration and exploitation must be balanced because each chosen action determines which reward data becomes visible. The discussed approaches include greedy and epsilon-greedy algorithms, policy gradients, and upper-confidence-bound algorithms. The course then extends the decision problem from bandits to sequential settings represented by Markov decision processes.

• Connecting evaluation with improvement: Dynamic programming introduces planning in Markov decision processes, but the broader lesson is policy iteration. Policy evaluation estimates the performance of a current policy, while policy improvement uses those estimates to produce a better policy. Repeating these two operations provides a general route toward good or optimal policies. Although the terminology commonly appears in discussions of dynamic programming, the lecture stresses that this pattern is more general and underpins most reinforcement learning algorithms for finding optimal policies.

• Learning without an environment model: Model-free prediction and control remove the assumption that transition dynamics are available. Monte Carlo learning follows a policy until an episode terminates, calculates the return, and treats that return as a regression target for estimating the policy’s value. Temporal-difference learning instead bootstraps from existing estimates and supports control methods including Q-learning, Sarsa, and double Q-learning. The lecture also separates on-policy data collection from the broader problem of learning about a policy that did not generate the observations.

• Advancing into deep reinforcement learning: Deep RL combines reinforcement learning with deep neural networks used as function approximators. The lecture places this development alongside direct policy optimization, actor-critic methods, trust regions, continuous actions, learned models, experience replay, multi-step returns, and search. These techniques can improve data use and expand the kinds of decision problems being addressed, but they also bring additional computation, instability, possible divergence, and unresolved research questions. Careful system configuration is therefore a central practical concern.