Reinforcement Learning in the OpenAI Gym (Tutorial) - SARSA

TL;DR

SARSA is an on-policy, model-free temporal-difference algorithm that learns action values by taking an action, observing the reward and next state, choosing the next action, and updating its Q function. The tutorial applies SARSA to OpenAI Gym’s cart-pole problem with a 0.1 learning rate, 0.9 discount factor, epsilon-greedy choices, and continuous states divided into 10 buckets. Read on for the algorithm, parameters, and discretization details.

Transcript

welcome back everybody to machine learning with Phil I am your host dr. Phil when we last touched on the opening idea we did q-learning to teach the cart pool robot how to dance basically how to balance the pole in this video we're gonna take a look at a related algorithm called salsa so they're related in the sense that they're both types of tempo... Read More

Key Insights

• 🅰️ Salsa and Q-learning are both types of temporal difference learning algorithms used in reinforcement learning.
• 🇶🇦 The Salsa algorithm is an on-policy method, while Q-learning is an off-policy method.
• 🥶 Both Salsa and Q-learning are model-free algorithms and do not require knowledge of the complete model of the environment.
• 🇶🇦 The Salsa algorithm uses the Q function, which estimates future rewards, to guide actions and update the Q function based on rewards and state transitions.
• 🥶 The Salsa algorithm can handle uncertainty in state transition probabilities through its model-free approach.
• 👾 Discretizing the continuous state space in the cart-pole problem may limit the effectiveness of the Salsa algorithm.
• 👋 The Salsa algorithm can achieve good results in the cart-pole problem, but there may be variability due to limitations in the discretized state representation.

Questions & Answers

Q: How does SARSA work in OpenAI Gym’s cart-pole problem?

SARSA initializes a learning rate and Q function, selects an initial state and epsilon-greedy action, then interacts with the environment. After receiving a reward and new state, it chooses the next action with the same epsilon-greedy strategy, updates Q, and repeats until the episode ends.

Q: What is the difference between SARSA and Q-learning?

Both are temporal-difference learning algorithms, but SARSA is on-policy while Q-learning is off-policy. The tutorial also identifies both methods as model-free.

Q: Why is SARSA considered model-free?

SARSA can learn without a complete model of the environment. In particular, it does not require all state-transition probabilities, the probabilities of reaching a new state and receiving a reward after taking an action.

Q: What does the Q function represent in SARSA?

The Q function is the agent’s estimate of discounted future rewards from a given state after taking a particular action. SARSA initializes this function and adjusts it using observed rewards, state transitions, and subsequent actions.

Q: What is the role of alpha in the SARSA algorithm?

Alpha is the learning rate, so it controls the rate of learning and how adjustments are made to the Q function. The cart-pole code uses a small alpha value of 0.1.

Q: How are continuous cart-pole states converted into discrete states?

The example uses the documented limits of the cart-pole variables to create linear spaces and divides them into 10 buckets. A helper function digitizes each observation and returns a four-element vector of bucket values.

Q: What gamma value does the cart-pole SARSA example use?

The main program uses a gamma, or discount factor, of about 0.9. The presenter says 1.0 could also be used because the cart-pole state-transition functions are deterministic as far as he is aware.

Q: How does epsilon-greedy action selection begin in the example?

The example starts epsilon at 1.0 and uses an epsilon-greedy strategy to select the initial action from Q. After each transition, it again selects the next action from the updated state using epsilon-greedy selection.

Summary & Key Takeaways

• The video explains the concept of Salsa, an on-policy temporal difference learning algorithm, in contrast to the off-policy algorithm Q-learning.

• Salsa and Q-learning are both model-free algorithms, meaning they can learn without knowing the complete model of the environment.

• The Salsa algorithm is explained using the example of the cart-pole problem, where the objective is to balance a pole on a moving cart.