Modeling a Markov Chain

A Markov Chain is all about a set of States and Transition Probabilities. Say you have some states and you know the probabilities for going from one state to another, then you can model a Markov chain that can forecast the future state and long-term / steady states. Let's say you have 2 weather conditions. "Sunny day" and "Rainy day" with the following transition matrix.


Where the matrix T represents the weather model in which, a Sunny day is 90% likely to be followed by another sunny day, and a rainy day is 50% likely to be followed by another rainy day.

Step by Step modeling example.

Start the Markov Decision Process tool from Rational Will or the standalone application. You will be asked to enter the states. Enter the 2 states "Sunny day" and "Rainy day" as shown below.


If you want to make changes to the States, like Edit the name or delete, you can do that by selecting a state and right-click to see the context menu. You can double click on the state name to turn into the Edit mode as well.


Click the "Proceed" button. In the next screen, you will be asked if there is any action that you can take that can affect the probabilities of the transition. Simply click "No".


Also, click "No" in the next question when you are asked if any action you can take on a Rainy day. Then you will be asked to enter the transition probabilities from Sunny Day state as shown below.

transition-probabilities- 1

Then click the "Proceed" button. Now, you will be asked to enter the transition probabilities from Rainy day state as shown below.

transition-probabilities- 2

Then click the "Proceed" button. You will be shown the following screen. Click the Finish button.


Now, click the "Finish" button. You will see the following view.

Result View

You will see a rich modeling user interface for your Markov chain.



From the carousel, you can see the Steady State probabilities and lots of useful charts. For example, another useful chart is the forecast chart where you can predict the future state after a given number of iteration.


You can click on a legend to show or hide a specific state. For example, clicking the Sunny Day, once, made the chart hidden.


You can find lots of options in the context menu related to the chart, by right-clicking the mouse.


Custom Expression Editor and Chart.

In Markov Chain Charts Carousel, you can obviously see lots of useful charts for the various perspectives of the Markov chain. You can get a forecast of a specific state. But, what about a composite state, like [State A] AND [State B] OR [State C] AND NOT [State D].

Yes, you can create a custom expression like that and see the forecast chart for such a composite state.

In the carousel, select the Custom Expression panel, as shown below. Then, Enter an expression like this.


Then, select the Chart tab to see the forecast of this compound state. Naturally, the result of this expression will be always 1, as those states are mutually exclusive. The chart tab shows that result.


You can always get help about what math functions are available to use in the expression editor. Just click the Help Icon shown in the Editor and the help tip popup will be displayed.


All Charts

Here is the complete set of charts available for you to analyze.


Decision Graph

Finally, from the Ribbon's View section, you can click the "Decision Graph" button to see the graph of the Markov chain.


Last updated on May 26, 2020