Markov Decision Processes 1 - Value Iteration | Stanford CS221: AI (Autumn 2019)

TL;DR

Markov decision processes (MDPs) involve making optimal decisions in uncertain environments, considering the probability of transition between states. Policies are evaluated based on expected utilities, which represent the average rewards obtained by following a specific policy.

Transcript

Okay. Let's start guys. Okay. So our plan for today is to catch up. So we're a little behind. So, uh, it's okay. So today, I want to talk about MDPs, Markov decision processes, and my plan is to talk about that for the first hour. And then after that, I want to talk, uh, for 10 minutes about the previous lecture. So remember, like we went over rela... Read More

Key Insights

• 💄 MDPs involve making decisions under uncertainty, considering potential outcomes and transition probabilities.
• 🚙 Policies determine the optimal actions to take in each state, aiming to maximize expected utilities.
• 🚙 Evaluating a policy involves calculating the expected utility of following that policy in each state.

Questions & Answers

Q: What is a policy?

A policy is a function that maps each state to the optimal action to take in that state, based on maximizing expected utilities.

Q: What is the difference between utility and value in MDPs?

Utility refers to the sum of rewards obtained on a specific path, while value is the expected utility achieved by following a policy.

Q: How is the discount factor gamma selected?

The selection of the discount factor is a design choice and depends on the specific problem. It represents the balance between valuing immediate rewards and considering rewards in the future.

Q: Can a policy change the value of the policies?

Yes, the introduction of new actions or changes in the problem can affect the value of policies as they determine the optimal actions in each state.

Summary & Key Takeaways

• MDPs involve making decisions in uncertain environments, considering the potential outcomes of different actions.

• Policies determine the optimal actions to take in each state, with the goal of maximizing expected utilities.

• Evaluating a policy involves calculating the expected utility of following that policy in each state.