L25.7 Steady-State Probabilities and Convergence

TL;DR

The steady-state convergence theorem states that if a Markov chain has a single recurrent class that is not periodic, the probabilities of being in each state will converge to a steady-state limit.

Transcript

So now we can come to the central topic of our lecture, which describes the conditions under which a Markov chain reaches steady state. The question that we are asking, and which we motivated in the previous lecture by looking at an example with a simple, two-state Markov chain is the following-- we are asking whether the probability of being in st... Read More

Key Insights

• 🏛️ The initial conditions in a Markov chain with multiple recurrent classes can influence the long-time probabilities.
• 🏛️ A Markov chain with a periodic recurrent class will have n-step transition probabilities that do not converge.
• 🏛️ The steady-state convergence theorem guarantees that the probabilities in a Markov chain with a single recurrent class and no periodic states will converge.
• ❓ The theorem provides conditions for convergence and system of linear equations to calculate the steady-state probabilities.

Questions & Answers

Q: How does the initial state affect the long-term probability in a Markov chain with multiple recurrent classes?

In a Markov chain with multiple recurrent classes, the initial state does matter. Depending on which recurrent class the initial state belongs to, the long-term probability of being in a certain state can be either 0 or positive. The initial conditions influence the probability when there are multiple recurrent classes.

Q: What happens to the n-step transition probability in a Markov chain with a periodic recurrent class?

In a Markov chain with a periodic recurrent class, the n-step transition probability for a specific state will never converge. It will keep oscillating between 0 and 1. The periodic nature of the recurrent class prevents convergence of the probability.

Q: How can we determine if a single recurrent class in a Markov chain is periodic or not?

A single recurrent class in a Markov chain is not periodic if it contains at least one self-transition. If there is a state in the recurrent class that allows a transition back to itself in a single step, it is considered aperiodic.

Q: What is the intuitive idea behind the steady-state convergence theorem?

The intuition behind the steady-state convergence theorem lies in considering two independent copies of the Markov chain starting at different initial conditions. If these copies meet at a given state at a given time, their future evolutions cannot be distinguished, regardless of their initial conditions. This means that the initial state does not matter when the chain has a single recurrent class and no periodic states.

Summary & Key Takeaways

• The lecture introduces the question of whether the probability of being in a certain state of a Markov chain at a given time converges to a constant, independent of the initial state.

• The lecture discusses examples where the initial conditions do or do not affect the long-term probability of being in a certain state.

• The steady-state convergence theorem is explained, stating that if a Markov chain has a single recurrent class that is not periodic, the probabilities of being in each state will converge to a steady-state limit.