Naive Actor Critic With Experience Replay | When Great Ideas Go Horribly Wrong

TL;DR

The tutorial explores the addition of experience replay to an actor critic network and discusses its limitations.

Transcript

welcome back everybody in today's tutorial we're gonna do something that seems really smart on the surface but in the end turns out to be really quite dumb we're gonna try to add experience replay it to an actor critic Network and see how it works let's get to it now this project was motivated by my experience doing research in physics so in physic... Read More

Key Insights

• 💡 Combining different ideas in research can lead to significant breakthroughs, but not all ideas will work.
• 🧑‍🏭 Replay buffers in actor critic networks store log probabilities of actions instead of the actions themselves.
• 🥺 Uniform sampling in experience replay for actor critic networks can lead to suboptimal learning.

Questions & Answers

Q: Why is adding experience replay to an actor critic network a challenge?

Adding experience replay to an actor critic network is challenging because it can lead to large jumps in both parameter space and policy space, making it difficult for the network to learn.

Q: What is the purpose of a replay buffer in an actor critic network?

A replay buffer in an actor critic network is used to store memories of past experiences, including the log probabilities of actions taken. It helps to stabilize learning and improve overall performance.

Q: How does sampling memories from the replay buffer work?

Memories are sampled uniformly from the replay buffer, although in more sophisticated approaches, importance sampling is used. Sampling ensures that the agent learns from a diverse set of experiences.

Q: What is the difference between the actor and the critic in an actor critic network?

The actor in an actor critic network is responsible for selecting actions based on the observed states, while the critic evaluates the value of the chosen actions and provides feedback for learning.

Summary & Key Takeaways

• The tutorial discusses the motivation behind adding experience replay to an actor critic network and highlights the process of combining different ideas in research.

• It explains the structure of a replay buffer in actor critic networks and how it differs from value-based methods like deep Q-learning.

• The tutorial covers the implementation of an actor critic network and the importance of handling different aspects of memory in the network.