What Is a Markov Chain? States, Matrices, Steady State

A Markov chain is a simple way to model something that changes over time. The system is always in one of a few states. Each step, it can move to another state, or stay where it is. The key idea is memory. A Markov chain has none. The next state depends only on the current state. It does not depend on the path you took to get here. This is called the Markov property.

This page explains states, the transition matrix, and the steady state. We use a small weather example all the way through.

States and the transition matrix

Let us model the weather with two states: Sunny and Rainy. To describe how the weather changes, we need the chance of moving from each state to each other state. We write these chances in a grid called the transition matrix.

Read it row by row. If today is sunny, there is a 90% chance tomorrow is sunny, and a 10% chance it rains. If today is rainy, there is a 50% chance of sun and a 50% chance of more rain. Each row adds up to 1, because something must happen tomorrow.

Taking one step forward

Say today is sunny. What about tomorrow? We just read the Sunny row: 90% sunny, 10% rainy. What about the day after? Now we are not sure of tomorrow, so we blend the chances from both rows. That is one step of the chain. The software repeats this for you, day by day.

The steady state

Run the chain for a long time and something nice happens. The day-to-day chances settle down. They stop depending on today's weather. This long-run balance is called the steady state.

To find it, we look for a split of days that stays the same after one more step. Call the long-run chance of sun S and the long-run chance of rain R. The steady state must satisfy S = 0.9 S + 0.5 R, with S + R = 1. Solving gives:

S = 5/6 = 0.8333 and R = 1/6 = 0.1667

So in the long run, about 83% of days are sunny and about 17% are rainy. This holds no matter what today's weather is.

Absorbing states

Some states are different. Once you enter them, you can never leave. These are called absorbing states. A good example is a gambler who keeps betting until the money runs out. "Broke" is an absorbing state. The game ends there. Absorbing chains are also the basis of health models, where one state is "death." See the gambler's ruin example and the weather forecasting example for full walk-throughs.

Doing this in software

The SpiceLogic Markov Chain Calculator builds these models for you. You enter the states and the transition chances. It finds the steady state, runs the chain forward, and draws the charts. Rational Will includes the same tools inside its decision models.