Lab2: Markov Chains and Monte Carlo Methods

PART II

Example 1: Matrices in R and Markov Chains

Let's look at my code in: HW6 Problem 1 and 2

First let's just get the TPM matrix, and call it P

Let's make the P^2 and P^3. There are many ways to do this!

You can extract elements of the matrix, if needed! Can you extract the (3,3) element? The 3rd row?, etc.
You can also perform many other matrix operations (see Examples)

Now, back to HW6 and more! Let's make the Q matrix, the T matrix, and the QT matrix. (Hint: These might be helpful for practice to check your answers.)

We also know from lecture that the from QT matrix will can calculate the probability of absorption.

Example 2: Markov Chains and Games

Class Game Example:

The amount of money he has after each game is a Markov chain with a one-step transition probability matrix. There are 5 States where $0 is State 1, $10 is State 2, $20 is State 3, $30 is State 4 and $40 is State 5. This Markov Chain has two absorbing states, $0 is State 1 and $40 is State 5.

Let's look at my code in: game.r

(You can use this code for Homework!)

Using Matrices in R

Return to Class Game Example:

Q1: What is the probability that the gambler quits in six (or fewer bets)?

Ans: Let's make the P matrix and then calculate P^6.
Beginning with $20, the gambler has the following probability of being in
an absorbing state at time 6: 0.616 + 0.274 = 0.899
So, the gambler has an 89% chance of placing six or fewer bets!

Q2: What is the expected number of bets he makes until he quits?

Ans: Using the Q matrix.
A gambler who begins with $20 will bet an average of 3.85 times before either loosing all his money or doubling it!
(A gambler who begins with $10 will bet an average of 2.54 times before he quits,
and a gambler who begins with $30 will bet on average 3.31 times before he quits)

Q3: What is the probability of absorption? What does this mean?

Example 3: The Gambler's Ruin

Here is the WOLFRAM Demonstration Project simulation

The description is below the simulation.

Example 4: Longrun Probability Using Java

To simulate repeated game-playing, until one side or the other is ahead by a sufficient number,
Let's check out longrun

PART III

Example 5: Sales Velocity: A Markov Chain Analysis

Let's look at the datacamp tutorial: here

Let's look at the code in: salesvelocity.r

Example 6: Simulate a Markov Chain

Let's look at the code in: mcsim.r

What does this do?

If you copy and past the code you should have a trans and run

In order to run the run function you will need to:

> run()

Example 7: Cards and Poker

We will need 3 functions to play poker. I have written these functions, so the help files are not in R, yet.

They are not too complicated. The 3 functions are:

cards.r
rperm.r
poker.r

There are 2 ways to get these functions for your very own.

1. Copy and paste the code into the R command window for each function.
2. Use the R source command which sources the code in the files. You will need 3 source commands:

   > source("https://edoras.sdsu.edu/~babailey/stat550/cards.r")
   > source("https://edoras.sdsu.edu/~babailey/stat550/rperm.r")
   > source("https://edoras.sdsu.edu/~babailey/stat550/poker.r")
   

For practice, generate 1 hand of 5 card poker.

IMPORTANT If you want to reproduce your results, you will have to set the seed of the random number generator.
There is an R function set.seed

To use it,

> set.seed(1)
> poker(draw=5, hand=1)

How can you draw the same hand again?

Looping in R

Let's play 10 poker hands:

> rhands <- matrix(NA,10,5)
> for (i in 1:10){ rhands[i,] <- poker(5) }

What's rhands?

Intro to R Part I: For more practice with R

Many tutorials available!

At the bottom of the page is how to get the "An Introduction to R"

Part IV

Generating Random Normal Variables - Old School

Let's check out the Box-Muller Transformation!

Here is a reference: Wolfram's page

Here is a reference for Monte Carlo Methods: Monte Carlo Integration page

Example 8: Example 5.7 from text using Monte Carlo Integration

An AC unit has an exponential lifetime distribution with mean 4 years.
Use an effective annual rate of interest of 5%.
Let Zbar denote the present value random variable for the unit of payment made at the time point of failure.
Find E(Zbar). (the exact asnwer using integration by parts is 0.83670)

Let's use Monte Carlo Integration to solve this problem!

Let's take a look at a Lab from Dr. Hancock at Clark University

What expectation (integral) do we need estimate?

Here is my R code for this example. You can copy and paste the code into the R command window to run it.

Example 9: Let's Bootstrap!

We're going to follow Lab: Nonparametric Bootstrapping Example 1