Stanford CS234: Reinforcement Learning | Winter 2019 | Lecture 9 - Policy Gradient II

TL;DR

Policy gradient methods can be improved by incorporating a baseline, using different step sizes, and ensuring monotonic improvement in policy updates.

Transcript

All right. Welcome back everybody. Um, before we get started today, I- does anybody have any questions about logistics, or midterm, or anything like that? We'll be doing a midterm review on Monday, and the midterm will be on Wednesday. Because there's a number of people in the class, we're gonna be spreading everybody across a couple of rooms, and ... Read More

Key Insights

• ❓ Policy gradient methods can benefit from incorporating baselines and using different step sizes.
• ↩️ Interpolation methods, such as n-step returns, can be used to trade off between bias and variance in gradient estimates.

Questions & Answers

Q: What is the purpose of incorporating a baseline in policy gradient methods?

The baseline is used to reduce variance in the gradient estimates. It helps in providing more stable and accurate updates to the policy parameters.

Q: How can step sizes be chosen in policy gradient methods?

Step sizes can be determined using line search or through other optimization methods that try to ensure monotonic improvement in policy updates. The selection of step sizes depends on the specific problem and application.

Q: How can monotonic improvement be achieved in policy gradient methods?

Monotonic improvement can be achieved by carefully choosing step sizes and ensuring that the value of the new policy is greater than or equal to the value of the previous policy. This can help in making progress towards better policy optimization.

Summary & Key Takeaways

• Policy gradient methods involve optimizing a parameterized policy to maximize a value function that depends on the policy.

• The baseline, often the expected return or value function, is subtracted from the advantage function to reduce variance in the gradients.

• Different step sizes and interpolation methods can be used to determine how far to go along the gradient and achieve monotonic improvement.