Chapter 2 –Markov Processes and Markov Chains 1 Definition of a Markov Processes A Markov process

X t , t T is a stochastic process with the property that, given

the value of Xt, the values of Xs for s > t are not influenced by the values of Xu for u < t. In other words, the probability of any particular future behavior of the process, when its current state is known exactly, is not altered by additional knowledge concerning its past behavior. A discrete time Markov chain is a Markov process whose state space is a finite or countable set, and whose time (or stage) index set is T = (0, 1, 2,…). In formal terms, the Markov property is that P X n 1 j | X 0 i0 ,..., X n 1 in 1 , X n i (2-1) P X n 1 j | X n i for all time points n and all states i0, …, in-1, i, j. It is customary to label the state space of the Markov chain by the nonnegative integers {0, 1, 2, …} and to use Xn=i to represent the process being in state i at time (or stage) n. The probability of Xn+1 being in state j given that Xn is in state i is called the one-step transition probability and is denoted by Pijn , n 1 . That is

Pijn , n 1 P X n 1 j | X n i

(2-2)

The notation emphasizes that in general the transition probabilities are functions not only of the initial and final states, but also of the time of transition as well. If the one-step transition probabilities are independent of the time variable n, i.e.,

Pijn , n 1 Pij , we say that the Markov chain has stationary transition probabilities. We will limit our discussions to Markov processes with stationary transition probabilities only. It is customary to arrange these numbers Pij in a matrix, in the infinite square array P00 P01 P02 P03  P10 P11 P12 P13  P P21 P22 P23  P = 20 (2-3)     Pi 0 Pi1 Pi 2 Pi 3     

and to refer to P = Pij

as the Markov matrix or transition probability matrix of

the process. The (i+1)st row of P is the probability distribution of the values of Xn+1 under the condition that Xn=i. If the number of states is finite, then P is a finite square matrix whose order (the number of rows) is equal to the number of states. Since probabilities are non-negative and since the process must make a transition into some state, it follows that Pij 0 

P j 0

ij

1

for i, j 0,1,2

(2-4)

for i 0,1,2

(2-5)

A Markov process is complete defined if its transition probability matrix and initial state X0 (or, more generally, the probability distribution of X0) are specified. P X 0 i0 , X 1 i1 ,, X n in  P X 0 i0 , X 1 i1 ,, X n1 in1 P X n in | X 0 i0 , X 1 i1 ,, X n1 in1 By definition of a Markov process, we have P X n in | X 0 i0 , X 1 i1 ,, X n 1 in 1

P X n in | X n 1 in 1Pin1 ,in

Thus,

By induction,

(2-7)

P X 0 i0 , X 1 i1 ,, X n in  P X 0 i0 , X 1 i1 ,, X n 1 in 1 Pin1 ,in P X 0 i0 , X 1 i1 ,, X n in Pi0 Pi0 ,i1  Pin 1 , in

It is also evident that P X n1 j1 ,, X nm jm | X 0 i0 , X 1 i1 ,, X n in  P X n1 j1 ,, X nm jm | X n in  for all time points n, m and all states i0, …, in, j1, …, jm. Example 1

(2-6)

(2-8)

(2-9)

A Markov chain X0, X1, X2 … has the transition probability matrix 0 1 2 0 0.6 0.3 0.1 P= 1 2

0.3 0.3 0.4 0.4 0.1 0.5

If it is known that the process starts in state X0 = 1, determine the probability P X 0 1, X 1 0, X 2 2. Example 2

A Markov chain X0, X1, X2 … has the transition probability matrix 0 1 2 0 0.7 0.2 0.1 P= 1 2

0 0.6 0.4 0.5 0 0.5

Determine the conditional probabilities P X 2 1, X 3 1, | X 1 0 and P X 1 1, X 2 1, | X 0 0. Example 3 A simplified model for the spread of a disease goes this way: The total population size is N = 5, of which some are diseased and the remainder are healthy. During a single period of time, two people are selected at random from the population and assumed to interact. The selection is such that an encounter between any pair of individuals in the population is just as likely as between any other pair. If one of these persons is diseased and the other is not, then with probability = 0.1 the transmission takes place. Let Xn denote the number of diseased persons in the population at the end of nth period. Specify the transition probability matrix.

0

1

2

3

4

5

0

1

0

0

0

0

0

1

0

0.96

0.4

0

0

0

2

0

0

0.94

0.06

0

0

3

0

0

0

0.94

0.06

0

4

0

0

0

0

0.96

0.04

5

0

0

0

0

0

1

Pijn, n 1 Pij 0 if j i Pijn , n 1 Pij 0 if j i 1 Therefore, Pii Pi ,i 1 1 .

C (i,1)C ( N i,1) C (i,1)C (5 i,1) Pi ,i 1   0.1 C ( N ,2) C (5,2) i (5 i )  10 2 Transition probability matrices of a Markov chain Let Pij(n ) denote the probability that the process goes from state i to state j in n transitions, i.e.

Pij( n ) P X m n j | X m i.

(2-10)

The n-step transition probability matrix is then expressed by P(n) = Pij(n ) . Theorem 2.1 The n-step transition probabilities of a Markov chain satisfy 

Pij( n ) Pik Pkj( n 1)

(2-11)

k 0

1 i j  where Pij( 0)  . 0 i j  Equation (2-11) is equivalent to P(n) = P P(n-1) and therefore, by iteration, we have P(n) = P PP … P = Pn .

(2-12)

In other words, the n-step transition probabilities Pij(n ) are the entries in the matrix Pn, the nth power of P. A more general form of Equation (2-11) which is known as the ChapmanKolmogorov equations is ( n m ) ij

P



Pik( n ) Pkj( m )

(2-13)

k 0

for all n, m 0 , all i, j. If the probability of the process initially being in state j is pj, i.e., the distribution law of X0 is P X 0 jp j , then the probability of the process being in state k at

time n is 

Pk( n ) p j Pjk( n ) P X n k  j 0

Example 4

A Markov chain

X n  on the states 0, 1, 2 has the transition

probability matrix 0

0 1 2 0.1 0.2 0.7

P= 1 2

0.2 0.2 0.6 0.6 0.1 0.3

(a) Compute the two step transition matrix P2. (b) What is P X 3 1, | X 1 0 ? P X 3 1, | X 1 0P01( 2) 0 .2      0 .1 0 .2 0 .7  0.2 0.13  0.1   

( 2) 01

P

0   1 0 0 P  1  = e1 P2 e2  0   2 

0    The i-th element  0  ei  1  0     0  (c) What is P X 3 1 | X 0 0? P X 3 1, | X 0 0P01( 3) 0 .2  0.1 0.2 0.7        0 .1 0 .2 0 .7  0 .2 0 .2 0 .6  0 .2        0 .6 0 .1 0 .3   0.1 

(2-14)

0   1 0 0 P  1  = e1 P3 e2  0   3 

Example 5

X n  on the states 0, 1, 2 has the transition

A Markov chain

probability matrix 0

0 1 2 0.3 0.2 0.5

P= 1 2

0.5 0.1 0.4 0.5 0.2 0.3

and initial distribution p0 = p1 = 0.5. Compute the probabilities P X 2 0 and P X 3 0. [Solution] 2

P X 2 0p j Pj(02 ) 0.5 P00( 2 ) 0.5 P10( 2) j 0 2

P X 3 0p j Pj(03) 0.5P00(3) 0.5 P10(3) 0.416 j 0

0 .3      P X 2 00.5  0 .3 0 .2 0 .5   0 .5    0 .5    0 .3     0.5   0 .5 0 .1 0 .4   0.5 0.42   0 .5   

0.3 0.2 0.5  0.3 0.2 0.5   0.44 0.18 0.38       P =  0.5 0.1 0.4  0.5 0.1 0.4 =  0.40 0.19 0.41     0.5 0.2 0.3  0.5 0.2 0.3  0.40 0.18 0.42      2

0.44 0.18 0.38  0.3 0.2 0.5   0.412 0.182 0.406       P =  0.40 0.19 0.41  0.5 0.1 0.4 =  0.420 0.181 0.399     0.40 0.18 0.42  0.5 0.2 0.3  0.420 0.182 0.398      3

P00( 2 ) 0.440,

P10( 2) 0.400

P00( 3) 0.412,

P10(3) 0.420

.



P

Note that the n-step transition matrix Pn satisfies

( n) ik

1 .

k 0

3 First Step Analysis Consider he Markov chain

X n  whose transition probability matrix is 0 P= 1 2

0 1 1 0

2 0

  , 0 0 1

where 0, 0, 0 and 1 . Two questions arise: (1) In which state, 0 or 2, is the process ultimately trapped? [absorption state] (2) How long, on the average, does it take to reach one of these states? [time of absorption] The time of absorption of the process can be defined as T min{n 0; X n 0 or X n 2} . Also, let u P X T 0 | X 0 1

[The probability of being absorbed in state 0]

v E T | X 0 1

[Average time of absorption]

From the transition probability matrix, it yields u P X T 0 | X 0 1 2

P X T 0 | X 1 k  P X 1 k | X 0 1 k 0

 1  u  0

Thus, u u

(2-15)

  u  . 1  

(2-16)

The absorption time T is always at least 1. If either X 1 0 or X 1 2 , then no further steps are required. If, on the other hand, X 1 1 , then the process is back at its starting point, and, on the average, v E T | X 0 1additional steps are required before absorption occurs. Weighting these contingencies by their respective probabilities, we have

v E T | X 0 1 1  0  v  0 1 v Thus, v 1 v

(2-17)

1 v . 1 

(2-18)

Now, let’ s consider the four state Markov chain whose transition probability matrix is 0 1 2 3 0 1 0 0 0 1 P10 P11 P12 P13 P= 2 P20 P21 P22 P23 3 0 0 0 1 Absorption now occurs in states 0 and 3, and states 1 and 2 are “ transit” . The probability of ultimate absorption in state 0 depends on the transit state in which the process begin. Therefore, we must extend our notation to include the starting state. Let T min{n 0; X n 0 or X n 3} ui P X T 0 | X 0 ifor i 1,2 [The probability of being absorbed in state 0] vi E  T | X 0 i for i 1,2

[Average time of absorption]

u0 1, u3 0, v0 v3 0. Applying the first step analysis, it yields u1 P10 P11u1 P12u2 u2 P20 P21u1 P22u2

(2-19) (2-20)

( u1 ,u2 ) can then be solved simultaneously. As for the mean time to absorption, the first step equations are

v1 1 P11v1 P12v2 v2 1 P21v1 P22v2 Example 6

A Markov chain

probability matrix

(2-21) (2-22)

X n  on the states 0, 1, 2, 3 has the transition

0 1 P= 2 3

0 1 2 3 1 0 0 0 0.4 0.3 0.2 0.1 0.1 0.3 0.3 0.3 0 0 0 1

From the first step analysis, it yields u1 0.4 0.3u1 0.2u 2 u 2 0.1 0.3u1 0.3u 2 0.7u1 0.2u 2 0.4 0.3u1 0.7u 2 0.1 u1 30

43

,

u 2 19

43

.

What is the probability for the process being absorbed in state 0? We need to know the distribution law of X0, i.e., P X 0 jp j . 

P X T 0p j u j j 0

Suppose that in this example p0 0.2, p1 0.3, p2 0.25, p3 0.25. Then, 30 19 P X T 00.2 1 0.3  0.25  0.25 0 43 43 0.519767

For the mean time of absorption, it yields v1 1 0.3v1 0.2v2 v2 1 0.3v1 0.3v2

Now let’ s consider the probability for the process being absorbed in state 3. Applying the first step analysis yields

u1 0.1 0.3u1 0.2u 2 u 2 0.3 0.3u1 0.3u 2 0.7u1 0.2u 2 0.1 0.3u1 0.7u 2 0.3 u1 13

43

,

u 2 24

43

.

The probability for the process being absorbed in state 1 is 13 24 P X T 10.2 0 0.3  0.25  0.25 1 43 43 0.480233

Now, we want to develop a general form for an N+1 state Markov chain. Let

X n  be a Markov chain on the states 0, 1, …, N. Suppose that states 0, 1, …, r1 are transient in that Pij( n )  0 as n  0 for 0 i, j r 1 while states r,…, N are absorbing states ( Pii , 1

r i N ).

The transition probability matrix has the form P=

Q R 0 I

(2-23)

where 0 is an (N–r+1)r matrix all of whose entries are zero, I is an (N–r+1)  (N–r+1) identity matrix, and Qij Pij for 0 , i j r . Started at one of the transient states X0 = i, where 0 i r , such a process will remain in the transient states for some random duration, but ultimately the process will be absorbed in one of the absorbing states i r ,, N . Let the probability of being absorbed in state k ( r k N ) when the initial state is X0 = i ( 0 i r ) be expressed by Uik. r 1

N

j 0

j r j k

U ik Pik PijU jk Pij 0

(2-24)

r 1

U ik Pik PijU jk , 0 i r , r k N . j 0

(2-25)

Let the random absorption time be expressed by T min{n 0; X n r} . Suppose that a rate g(i) is associated with each transient state i and that we wish to determine the mean total rate that is accumulated up to absorption. Let wi be such total rate when the initial state X0 = i, i.e., T  wi E  g Xn | X 0 i   n 0  

(2-26)

If the rate g(i) is defined as 1 0 i r 1  g ( X n i )  , 0 otherwise

(2-27)

T  wi E  T | X 0 i vi  n 0  

(2-28)

then

(the mean time of absorption when the initial state X0 = i.) If, for a specified transient state k, the rate g(i) = gk(i) is defined as 1 if i k  g ( X n i ) g k  X n i  for 0 , i k r , 0 if i k 

(2-29)

then T  Wik E  gk  Xn  | X 0 i   n 0  

(2-30)

is the mean number of visits to state k ( 0 k r ) prior to absorption, when the process starts from state i. Applying the first step analysis, we have r 1

wi g (i ) Pij w j for 0 i r .

(2-31)

j 0

The special case of 1 0 i r 1  g ( X n i )  0 otherwise gives vi E T | X 0 i by r 1

vi 1 Pij v j , for i 0,1, , r 1. j 0

(2-32)

1 if i k  For the case of g ( X n i ) g k  X n i  , we have 0 if i k  r 1

Wik ik PijW jk for 0 i r .

(2-33)

j 0

Example 7

A Markov chain

X n  on the states 0, 1, 2, 3, 4, 5 has the transition

probability matrix 0 1 2 P= 3 4 5

0 1 0 0.9 0 0.5 0 0 0 0 0 0 0 0

2 3 4 5 0 0 0 0.1 0.4 0 0 0.1 0.6 0.2 0.1 0.1 0.4 0.5 0 0.1 0.4 0 0.5 0.1 0 0 0 1.0

Find the mean duration spent in state 2 if the beginning state is 0. [Solution] r 1

W02 02 P0 jW j 2 0 0.9W12 0.1W52 j 0

W02 0.9W12 0.1W52 Similarly, W12 0.5W12 0.4W22 0.1W52 W22 1 0.6W22 0.2W32 0.1W42 0.1W52 W32 0.4W22 0.5W32 0.1W52 W42 0.4W22 0.5W42 0.1W52 W52 1W52 It is apparent that W52 0. Therefore, a simplied expression is w0 0.9 w1 w2 1 0.6w2 0.2 w3 0.1w4 w3 0.4 w2 0.5w3

w4 0.4w2 0.5w4 The unique solution is w0 4.5, w1 5.0, w2 6.25, w3 w4 5.0.

4 Special Markov Chains The Two State Markov Chain Let

P=

0 1

0 1 1 a a , where 0 a, b 1 b 1 b

(2-34)

be the transition matrix of a two state Markov chain. If a 1 b , then the states X 1 , X 2 , are independent identically distributed random variables with P X n 0b and P X n 1a . [Proof]

P=

0 1

0 1 1 a a , where 0 a, b 1 1 a a

Pij P X n1 j | X n ipc 1 a (or a ) , for j 0 (or 1). 1

1

i 0

i 0

p j pi Pij pc pi pc

Therefore, p j pij and it indicates that X n and X n1 are independent. (Why are they identically distributed?) The n-step transition matrix is given by

Pn =

[Proof] Let

n 1 b a  1 a b  a a  b b a b b a a b

(2-35)

b a b a

A=

and B =

a a b b

then,





Pn =  a b  A  1 a b B . 1

n

Also, AP = A and

BP = (1a b)B

It is easily seen P1 = P . Assume the formula is true for n, then

Pn P

   a b  AP  1 a b BP   a b  A  1 a b  B   a b  A  1 a b B P 1

n

1

n

1

n 1

P n 1 The formula holds for n+1 if it holds for n. Therefore, it holds for all n. Note that 1-a-b 1 when 0 a, b 1 , therefore,

b lim P n  a b n b a b

a a b a a b

.

(2-36)

Markov Chains Associated with IID Random Variables Let denote a discrete valued random variable whose values are nonnegative integers and P i  ai for i 0,1,  and



a i 0

i

1 . Let 0 , 1 ,  , n , 

represent independent observations of . We shall now study the kinds of Markov chains whose state spaces are coincide with the independent observations of . (1) Transition probability matrix characterizing iid random processes Consider a random process

X n n ,

n 0,1,  . Since 0 , 1 ,  , n , 

are independent observations of  ,  X n  is therefore an iid random process. The following transition probability matrix characterizes such iid random processes. a0 a P 0 a0 

a1 a1 a1 

a2  a2  a2  

(2-37)

The row vector represents the probability mass function of . (2) Successive Maximum Series Let X n max  0 , 1 ,  , n . It is readily seen that { X n } is a Markov chain and X n1 max  X n , n1 . Therefore, the transition probability matrix of { X n } is given by 0 1 0 a0 a1 1 0 a0 a1 P 2 0 0 3 0 0  

2 a2 a2 a0 a1 a2 0 

0 0 A0 1 0 P 2 0 3 0 

1 a1 A1 0 0 

2 a2 a2 A2 0 

3 a3 a3 a3 a0 a1 a2 a3  3 a3 a3 a3 A3 

   

   

(2-38)

(2-39)

i

where Ai a j for i 0,1,  . j 0

Example 8 Suppose  1,  2 ,  represent successive bids on certain asset that is offered for sale and X n max  1 ,  , n is the maximum that is bid up to stage n. Suppose that the bid that is accepted is the first bid that equals or exceeds a prescribed level M. What is the average time of sale, i.e., the average time that is required to accept the bid? [Solution]

The time of sale is the random variable T min{n 1; X n M } .

E (T ) E (T |  1 M  E (T |  1 M  1 M ) P  1 M ) P  E (T |  1 M  1 P 1 M  1 M ) P  E (T |  1 E  T | 2 M  P 2 M  E  T | 2 M  P 2 M  1 M )  Since future bids  2,  3, … have the same probabilistic properties as in the original problem, it yields

E  T [1 E (T )]P 1 M  1 P 1 M  E (T ) 1 E  T P 1 M 

(2-40)

(3) Successive Partial Sums Series Let n  2  n , n 1,2, , and, by definition, 0 0 . The process 1  X n n is a Markov chain with the following transition probability matrix 0 0 a0 1 0 P 2 0 3 0 

1 a1 a0 0 0 

2 a2 a1 a0 0 

3 a3 a2 a1 a0 

   

(2-41)

Markov Chains of Success Runs Consider the case of conducting repeated Bernoulli trials (each of which admits only two possible outcomes, success S or failure F). Suppose that in each trial, the probability of S is and the probability of F is = 1 –. We define the run length of a success run (or the success run length) as the number of consecutive trials which yield success. That is, we say a success run of length r happened if the outcomes in the preceding r+2 trials, including the present trial as the last, were respectively, F, S, S,…, S, F. Let’ s label the present state of the process by the length of the success run currently under way. The process is Markov since the individual trials are independent of each other, and its transition probability matrix is given by

0 0  1  P= 2  3   

1  0 0 0 

2 0  0 0 

3 0 0  0 

4 0 0 0  

   

(2-42)

We generalize the above success run process for cases for which state i 1 can only be reached from state i and the run length is renewed (set to zero) if a failure occurs. The corresponding probability transition matrix is therefore given by 0 1 2 3 4 0 p0 q0 0 0 0  1 p1 r1 q1 0 0  P = 2 p2 0 r2 q2 0  (2-43) 3 p3 0 0 r3 q3        Note that state 0 can be reached in one transition from any other state. Another example of success run process is the current age in a renewal process. Consider a light bulb whose lifetime, measured in discrete units, is a random variable  , where P i  ai for i 1,2,  and



a i 1

i

1 .

Let each bulb is replaced by a new one when it burns out. Suppose the first bulb lasts until time  1, the second bulb until time  1+ 2 , and the nth bulb until time 1  n , where the individual lifetimes 1 , 2 ,  are independent observations of a random variable . Let Xn be the age of the bulb in service at time n. Such current age process is depicted in the following figure. Note that the failure can only occur at integer times and, by convention, we set Xn = 0 at the time of a failure. Current Age 5 4 3 2 1 0

2

4

6

8

Time, n

The current age is a success runs Markov process for which ak 1 pk   ak 1 ak 2 

(2-44)

where pk is the probability of returning to state 0 at the next stage, given that the current state is state k.

It is equivalent to say that P X n 1 0, X n k  pk P X n 1 0 | X n k   P X n k  P X n 1 0, X n k  P k 1ak 1 



i 1

i 1

P X n k  P k i ak i .

Consider the success runs Markov Chain on N+1 states whose transition matrix is

0 1 2 P=  N 1 N

0 1 2 1 0 0 p1 r1 q1 p2 0 r2    pN 1 0 0 0 0 0

3 0 0 q2  0 0

…   

N 0 0 0 

(2-45)

 q N 1  1

Note that states 0 and N are absorbing states. Let T be the hitting time to states 0 or N, i.e., T min n 0; X n 0 or X n N . It can be shown that uk P X T 0 | X 0 k   qk 1  p q k k

  q N 1    , k 1,, N 1.  p q   N 1    N 1

(2-46)

u0 1, u N 0. The mean hitting time vk E T | X 0 k 

 k , N 1 1 vk   k , k 1   pk qk pk 1 qk 1 pN 1 q N 1

(2-47)

where  qk kj  p q k k

   qk 1   q j 1     , k j.  p q     k 1  p j 1 q j 1    k 1

(2-48)

For a given state j (0< j < N), the mean total visits to state j starting from state i (Wij) is  1 , j i  pi qi    1   qi   q j 1      Wij   , i j p q     p q  i  p j 1 q j 1  j  i  j   0 , i j 

(2-49)

One-Dimensional Random Walks A one-dimensional random walk is a Markov chain whose state space is a finite or infinite subset of integers, in which a particle, if it is in state i, can in a single transition either stay in i or move to one of the neighboring states i –1, i+1. Let the state space be represented by nonnegative integers, the transition probability matrix of a random walk has the form

0 1 2 P=  i

0 r0 q1 0

1 p0 r1 q2 

2 0 p1 r2 0

i–1 i i+1  0 0 0 p2 0   qi ri pi 0  

(2-50)

The one-dimensional random walk can be used to depict the fortune of a player (player A) engaged in a series of contests. Suppose that a player with fortune k plays a game against an infinitely rich adversary (player B) and has probability pk of winning one unit and probability qk = 1 –pk of losing one unit in the next contest, and r0 = 1. The process Xn, where Xn represents his fortune after n contests, is a random walk. Once the state k = 0 is reached, the process remains in that state. The event of reaching state 0 is known as the “ gambler’ s ruin” . If the adversary (player B) also starts with a limited fortune l and player A has an initial fortune k (k + l = N), we then consider the Markov chain process Xn representing player A’ s fortune after n contests. The transition probability matrix is

0 1 2

P=

0 1 q1 0

1 0 r1 q2

 N 0

2 0 p1 r2 

3 0 0 p2

N  0

q N 1 rN 1  0 0

 p N 1 1

(2-51)

Note that when player A’ s fortune reaches 0 (player A is ruined) or N (player B is ruined) it remains in this same state forever. Also, different contests may be adopted at different stages, and therefore the probability of winning (or losing) one unit depends on player’ s fortune. Let’ s consider the games with identical contests, i.e., pk p, qk 1 p q for all k 1 and r0 1 . The transition probability matrix is 0 1 2 3 N 0 1 0 0 0

P=

1 2

q 0 0 q

 N 0

p 0  0 p 0   q 0 p  0 0 1

(2-52)

Let ui U i 0 be the probability of gambler’ s (player A) ruin starting with initial fortune i. The first step analysis yields ui qui 1 pui 1 for i 1,2,  , N 1 .

(2-53)

 N i when p q 0.5   N ui  i N  q p  q p  when p q N  1  q p 

(2-54)

Also, u0 1 and u N 0 . It can be shown that

“ Gambler’ s ruin”is the event that the process reaches state 0 before reaching state N. This event can be stated more formally if we introduce the concept of hitting time T. Let T be the random time that the process first reaches state 0 or N.

T min n 0, X n 0 or X n N . The event XT = 0 is the event of gambler’ s ruin, and the probability of such event starting from the initial state k is given by U k 0 u k P X T 0 | X 0 k  uk qu k 1 pu k 1 for k 1,  , N 1 .

(2-55)

with u0 1 , u N 0 . Let xk uk u k 1 for k 1,  , N 1 and using p q 1 , we have u k qu k 1 pu k 1 uk qu k pu k Therefore, 0 p(uk 1 u k ) q(uk u k 1 ) for k 1,  , N 1 . pxk 1 qxk 0 for k 1,  , N 1 .

(2-56)

px2 qx1 0 px3 qx2 0  px N qx N 1 0 . 2

q x , x q  q q x2  x2   p 1  p  p x1 , …, x N   3        x1 u1 u0 u1 1 ,

x2 u 2 u1 u2 x1 1 , x1 x2 u 2 1 , x3 u3 u 2 u3 x2 x1 1 , x1 x2 x3 u3 1 ,  xk uk u k 1 , x1  xk u k 1 ,  x N u N u N 1 u N 1 , x1  x N u N 1 1 , Therefore, the equation for general k gives

N 1

q  x N 1   p  x. p    1

u k 1 x1  xk

uk 1 x1 (q / p) x1   q p  x1 k 1





uk 1 1  q p   q p  x1 , for k 1,  , N 1 . k 1

(2-57)

Since u N 0 , it yields





0 1 1  q p   q p  x1 N 1

1 x1  . N 1 1  q p   q p

(2-58)

Substituting the (2-52) into (2-51) gives 1  q p   q p u k 1  . N 1 1  q p   q p k 1

1  q p   q p

k 1

(2-59)

if p q 0.5  k  k 1  q p  if p q 1   q p

Hence, N k  1 (k / N )  when p q 0.5  N  uk  k N 1 (q / p) k  q p  q p  1  when p q N N 1  ( q / p )    1  q p 

(2-60)

Similarly, it can be shown that, when p q 0.5 , the mean duration vk E  T | X 0 k  k  N k  , for k 0,1,  , N . The General Random Walks Consider the following general case of a one-dimensional random walk. 0 1 2 3 N 0 1 0 0 0  0 1 q1 r1 p1 0  0 P = 2 0 q2 r2 p2  0       N 0 0 0 0  1

(2-61)

where pk 0, qk 0 for k 1,2,, N 1 . Let T min n 0; X n 0 or X n N  be the random time that the process first reaches state 0 or N, i.e. the hitting time. The probability of gambler’ s ruin ( ui P X T 0 | X 0 i  ), the mean hitting time ( vk E T | X 0 k  ), and the mean total visit to state k from i ( Wik ), can be expressed by

  N 1 ui  i , i 1,, N 1. 1 1  N 1

(2-62)

where

k 

q1q2  qk , k 1,, N 1. p1 p2  pk

(2-63)

1  N 1    vk  1 1  k 1   1  k 1  , k 1,, N 1.  1 1  N 1    (2-64) where

1 1 1 1 k  q q q   q 1 2 2 3 k 1 k 1

  k , k 1,, N 1.  

  1  i 1  k  N 1  1    , i k   1  N 1 qk k 1     Wik    1   k  N 1     1  i 1    , i k k i 1   q     1  N 1   k k 1  

(2-65)

(2-66)

[Proof] (1) uk pk uk 1 rk uk qk uk 1 for k 1,2,, N 1 , and u0 1 , u N 0 . uk pk uk rk uk qk uk (since pk rk qk 1) 0 pk (uk 1 uk ) qk (uk 1 uk ) . Let xk uk uk 1 , k 1,2,, N , and we have q 0 pk xk 1 qk xk , xk 1  k xk . pk k 1 qi  q1 q2 q2 q1 q1q2  qk 1 x2  x1 , x3  x2  x1 , …, xk  x1  x1 .    p1 p2 p2 p1 p1 p2  pk 1 i 1 pi 

Let k 

q1q2  qk , then xk k 1 x1 , for k 1,2,, N . p1 p2  pk

x1 u1 u0 ,

x1 u1 1 .

x2 u2 u1 ,

x1 x2 u2 1

x3 u3 u2 ,

x1 x2 x3 u3 1  x1 x2  xk uk 1 ( k 1,2,, N )  x1 x2  xN u N 1 1 . uk 1 x1 x2  xk 1 x1 1 x1  k 1 x1

Since u N 0 , it yields u N 1 x1 1 x1  N 1 x1 1  1 1  N 1  x1 0 1 x1  1 1  N 1

Therefore, uk 1 x1 1 x1  k 1 x1 1  1 1  k 1  x1

 1 1  k 1   1 1  N 1 

1 

k  N 1  1 1  N 1 

uk  (2)

vk 1 pk vk 1 rk vk qk vk 1 for k 1,2,, N 1 , and v0 0 , vN 0 . vk pk vk rk vk qk vk 0 1 pk (vk 1 vk ) qk (vk vk 1 ) Let xk vk vk 1 , k 1,2,, N , and we have 0 1 pk xk 1 qk xk . q 1 xk 1  k xk  pk pk

q 1 x2  1 x1  p1 p1

q 1 q q1 1 1 qq q 1 x3  2 x2   2  x1     1 2 x1  2    p2 p2 p2 p1 p1  p2 p1 p2 p1 p2 p2 x4 

q1q2 q3 qq q 1 x1  2 3  3  p1 p2 p3 p1 p2 p3 p2 p3 p3



q2  qk 1  q3  qk 1   qk 1  1 q  qk 1   xk  1 x1         p p   p1  pk 1 p1  pk 1  p2  pk 1   k 2 k 1  pk 1

    xk k 1 x1  k 1  k 1  k 1   k 1 q1 1q2 2 q3 k 2 qk 1 Let

1      1 1 1  k 1  k 1  k 1  k 1   k 1       k 1  q1 1q2 2 q3 k 2 qk 1  q  q  q  q 1 2 2 3 k 2 k 1  1 xk k 1 x1 k 1 . x1 v1 v0 v1 ,

x2 v2 v1 v2 x1 , x1 x2 v2 , …, x1 x2  xk vk , for k 1,2,, N . vk x1 x2  xk

x1  1 x1 1   2 x1 2    k 1 x1 k 1   1 1  k 1  x1  1  k 1 

Let k = N, vN  1 1  N 1  x1  1  N 1 0

 1  N 1   1 1  N 1     N 1  vk  1 1  k 1  1  1  k 1   1 1  N 1  x1 

5 Review of the First Step Analysis Consider the Markov chain of N+1 states. Suppose that states 0, 1, …, r1 are

transient in that Pij( n )  0 as n  0 for 0 i, j r 1 while states r,…, N are absorbing states ( Pii 1, r i N ). The transition probability matrix has the form Q R 0 I

P=

(2-67)

where 0 is an (N–r+1)r matrix all of whose entries are zero, I is an (N–r+1)  (N–r+1) identity matrix, and Qij Pij for 0 , i j r .

R QR I

(2-68)

R QR Q 2 R I

(2-69)

(I Q  Q n-1 )R I

(2-70)

Q2 P = 0 2

P3 =

Q3 0

For higher values of n,

Qn P = 0 n

Let Wij(n ) be the mean total visits to state j up to stage n for a Markov chain starting from state i , i.e., n  Wij( n ) E   X l j  | X 0 i   l 0  

1 if X l j  where  X l j  . 0 if X l j  Consider  X l j as a Bernoulli random variable, and we have E  X l j  | X 0 i 

1P X l j | X 0 i  0 P X l j | X 0 i  . Pij(l )

(2-71)

E  X l j  | X 0 i  Pij( l )

(2-72)

Therefore, n  Wij( n ) E   X l j  | X 0 i   l 0   n

E  X l j  | X 0 i  l 0

n

Wij( n ) Pij(l ) for all states i, j.

(2-73)

l 0

Eq. (2-70) indicates that Pij( l ) Qij(l ) when 0 i, j r. Therefore, n

(n) ij

W

Qij(l ) Qij( 0 ) Qij(1)  Qij( n ) , 0 i, j r. l 0

1 if i j  In particular, Qij( 0)  . 0 if i j  In matrix form, we have

W ( n ) I Q Q 2  Q n I Q(I Q Q 2  Q n -1 )

W ( n ) I QW ( n -1) . r 1

r 1

k 0

k 0

Wij( n ) ij QikWkj( n 1) ij PikWkj( n 1) , 0 i, j r.

(2-74) (2-75)

The limit of Wij(n ) represents the expected value of the total visits to state j from the initial state i, i.e.,

Wij lim Wij( n ) E Total visits to j | X 0 i  , 0 i, j r. n 

W I Q Q 2  W I QW . Or,

(2-76)

r 1

Wij ij PikWkj , 0 i, j r.

(2-77)

W - QW (I Q)W I

(2-78)

W (I Q)-1

(2-79)

k 0

The matrix W (I Q)-1 is called the fundamental matrix associated with Q. The random absorption time is expressed by T min{n 0; N X n r} and it can be expressed by r 1 T 1

T  X n j  .

(2-80)

j 0 n 0

Since Wij is the mean number of total visits to state j from the initial state i (both are transient states), it follows that T 1   Wij E   X n j  | X 0 i , 0 i, j r.  n 0  

(2-81)

Let vi E  T | X 0 i be the mean time to absorption starting from state i. It follows that r 1

vi E  T | X 0 i  Wij

(2-82)

j 0

r 1 T 1   W  E  X n j  | X 0 i E  T | X 0 i  vi , 0 i r.    ij  j 0 j 0 n 0   r 1

r 1

r 1

r 1 r 1

j 0

j 0

j 0 k 0

Wij ij PikWkj , 0 i r. r 1

r 1

j 0

k 0

vi Wij 1 Pik vk , 0 i r.

(2-83)

Starting from initial state i, the probability of being absorbed in state k, i.e. the hitting probability of state k, is expressed by U ik P X T k | X 0 i for 0 i r and r k N . 

n

U ik P X T k | X 0 i  lim P X T k | X 0 i  T 1

n 

T 1

n

PX T 1

T

k | X 0 i 

f ik(1) f ik( 2)  fik( n )

P X 1 k | X 0 i  P X 1 ST , X 2 k | X 0 i   P X l ST , l 1,2,  , n 1; X n k | X 0 i 

P time of absorption in state k being less than or equal to n | X 0 i  P the process being absorbed in state k up to stage n | X 0 i  P n T , X T k | X 0 i 

where fik(n) is the probability that, starting from state i, the process reaches state k for the first time in n steps, and ST represents the set of all transient states, i.e. ST={0,1,…, r1}. n

Let U ik( n ) P . Since state k is an absorbing state, once the X T k | X 0 i  T 1

process reaches state k, it will always remain in that state. Therefore, n

. U ik( n ) P X T k | X 0 i  P X n k | X 0 i  T 1

U ik( n ) Pik( n ) for 0 i r and r k N

(2-84)

n

[Remark]

It is worthy to note that the expression of

PX T 1

T

k | X 0 i 

requires careful interpretation. The meaning of X T k for T i is not the same as X i k . Since T is the time of absorption, X T k for T i implies that the process reaches the absorbing state k for the first time at stage i; where X i k implies that the process is in state k at stage i. n

U ik( n ) P X T k | X 0 i  T 1

P X 1 k | X 0 i  P X 2 k | X 0 i   P X n k | X 0 i  Pik(1) Pik( 2)  Pik( n ) n

Pik(l ) l 1

U ik lim U ik( n ) lim Pik( n ) n 

From Eq. (2-70),

n 





U( n ) I Q  Q n -1 R U ( n ) W ( n -1)R

Taking n to infinity, it yields





U lim U( n ) I Q Q 2  R n 

U WR r 1

U ik Wij R jk , 0 i r and r k N . j 0

From Eq. (2-77), r 1 r 1  r 1  U ik Wij R jk  ij PilWlj  R jk j 0 j 0  l 0  r 1 r 1

r 1

r 1

j 0 l 0

l 0

j 0

Rik PilWlj R jk Rik Pil Wlj R jk r 1

U ik Rik PilU lk l 0

r 1

U ik Pik PilU lk , 0 i r and r k N l 0

6 The Long Run Behavior of Markov Chains Suppose that a transition matrix P on a finite number of states labeled 0, 1,…, N, has the property that the matrix Pk has all its elements strictly positive. Such a transition matrix, or the corresponding Markov chain, is called regular. For a regular Markov chain, there exists a limiting probability distribution (0 , 1,  , N ) where lim Pij( n ) j 0 , n 

j 0,1,  , N and

N

 1 . j 0

j

Example 9 The Markov chain with the following transition probability matrix 0 1 0 0.33 0.67 P= 1 0.75 0.25 P2 =

0.6114 0.3886 0.4932 0.5068 , P3 = , 0.4350 0.5650 0.5673 0.4327

P4 =

0.5428 0.4572 0.5220 0.4780 , P5 = , 0.5117 0.4883 0.5350 0.4560

P6 =

0.5307 0.4693 7 0.5271 0.4729 ,P = . 0.5253 0.4747 0.5294 0.4706

From Eq. (2-36), the limiting probabilities are 0.5282 and 0.4718. Example 10 The Markov chain with the following transition probability matrix 0 1 2 0 0.40 0.50 0.10 P= 1 2

0.05 0.70 0.25 0.05 0.50 0.45 0.1900 0.6000 0.2100

2

P = 0.0675 0.6400 0.2925 0.0675 0.6000 0.3325 0.0908 0.6240 0.2825 4

P = 0.0758 0.6256 0.2986 0.0758 0.6240 0.3002 0.0772 0.6250 0.2978 8

P = 0.0769 0.6250 0.2981 0.0769 0.6250 0.2981

Every transition probability matrix on the states 0, 1, …, N that satisfies the following two conditions is regular: (1) For every pair of states i, j there is a path k1, k2, …, kr for which Pik1 Pk1k 2  Pk r j 0 .

(2) There is at least one state i for which Pii > 0. Theorem Let P be a regular transition probability matrix on the states 0, 1, …, N. Then the limiting distribution (0 , 1,  , N ) is the unique nonnegative solution of the equations N

j k Pkj , k 0,1,  , N , k 0

(2-85)

N

 1 . k 0

(2-86)

k

[Proof] Since the Markov chain is regular, we have a limiting distribution,

lim Pij( n ) j 0 ,

n 

j 0,1,  , N and

N

 1 . j 0

j

From Pn Pn -1 P , it yields N

Pij( n ) Pik( n 1) Pkj ,

j 0,  , N .

k 0

Let n  , then Pij( n )  j while Pik( n )1  k . Therefore, N

j k Pkj , k 0,1,  , N . k 0

Proof of the uniqueness of the solution is skipped. Example 11 Determine the limiting distribution for the Markov chain with the following transition probability matrix 0 1 2 0 0.40 0.50 0.10 P= 1 2

0.05 0.70 0.25 0.05 0.50 0.45

(Solution)

0.400 0.051 0.052 0 0.500 0.701 0.502 1 0.100 0.251 0.452 2 0 1 2 1 0 1 / 13 , 1 5 / 8 , 2 31 / 104

The limiting distribution j , j 0,1,  , N , can also be interpreted as the long run mean fraction of time that the random process

X n is in state j.

7 Including History in the State Description There are cases that the phenomenon under investigation is not naturally a Markov process. However, such phenomenon can still be modeled as a Markov process by including part of the past history in the state description. Suppose that the weather on any day depends on the weather conditions for the previous two days. To be exact, we suppose that

(1) if it was sunny today and yesterday, then it will be sunny tomorrow with probability 0.8; (2) if it was sunny today but cloudy yesterday, then it will be sunny tomorrow with probability 0.6; (3) if it was cloudy today but sunny yesterday, then it will be sunny tomorrow with probability 0.4; (4) if it was cloudy for the last two days, then it will be sunny tomorrow with probability 0.1. Such a model can be transformed into a Markov chain provided we say that the state at any time is determined by the weather conditions during both that day and the previous day. We say the process is in (S,S) : if it was sunny for both today and yesterday; (S,C) : if it was sunny yesterday but cloudy today; (C,S) : if it was cloudy yesterday but sunny today; (C,C) : if it was cloudy for both today and yesterday. Then the transition matrix is Today’ s State Yesterday’ s State

(S,S) (S,C) (C,S) (C,C)

(S , S )

0 .8 0 .2

(S , C ) (C , S ) (C , C )

0 .4 0 .6 0 .6 0 .4 0 .1 0 .9

0.80 0.62 0 0.20 0.42 1 0.41 0.13 2 0.61 0.93 3 0 1 2 3 1 0 3 / 11, 1 1 / 11, 2 1 / 11, 3 6 / 11 . 8 Eigenvectors and Calculation of the Limiting Probabilities Example 12 A city is served by two newspapers, UDN and CT. The CT, however, seems to be in trouble. Currently, the CT has only a 38% market share. Furthermore, every year, 10% of its readership switches to UDN while only 7% of the UDN’ s readership switch to CT. Assume that no one subscribes to both papers and the total newspaper readership remains constant. What is the long-term

outlook for CT? [Solution] The readerships (in percentages) one year later 0.90 0.10   38 62  = 38.54 61.46  0.07 0.93   Or 0.90 0.07  38  38.54   = X 1    0.10 0.93 62 61.46     PT  X 0 X 1 The readerships at the end of the second year: 0.90 0.07  38.54  38.99   = X 2     0.10 0.93 61.46  61.01  

PT  X 1 X 2 Repeatedly, we have 39.36 39.67 39.93 40.14     X 3 ,  X 4 ,  X 5 ,  X 6     60.64 60.33 60.07  59.86       It is clear that CT not only is not in trouble; it is actually thriving. Its market share grows year after year! However, it does show that the rate of growth is slowing, and we can expect that eventually the readerships will reach an equilibrium state, i.e. lim(P T ) n  X 0 X and P T  X X . n 

Eigenvector of a square matrix Let A be an nn matrix. A non-zero vector X such that A X X

(2-87)

( A I ) X 0 (2-88) for some scalar is called an eigenvector for A. The scalar is called the eigenvalue. Now, let’ s consider calculation of Ak  B where B is an n1 vector. Suppose the matrix A has n eigenvalues (or n eigenvectors). Let the eigenvectors be X1, X2, …, Xn. These eigenvectors are linear independent and form a basis for an n-dimensional space. Therefore, we can express the vector B by a linear combination of Xi’ s, i.e., x11  x12  x1n        x21  x22  x  X 1   , X 2   ,  X n 2 n           x  x  x  n1  n 2  nn      1  1  2   2   B  X1 X 2  X n   X        n  n     

  1  2    Xn      n   

Ak  B Ak 1 A X1

X2

Ak 1  AX 1

  1  2    AX n       n   

AX 2

Since A  X X , we have Ak  B Ak 1   2 X 2 1 X1

Ak 1  X1

Ak 1  X1

  1  2    n X n       n   

X2

  1 0  Xn     0 

X2

  1     Xn    2      n   

Ak  B Ak 1  X1

Ak 2  X1

X2

X2

0    1  2  0  2           0  n  n     0

  1     Xn   2      n   

  1     Xn  2  2      n   

  X1

X2



  1     Xn  k  2      n   

Ak B  X1

X2

Note that  is a diagonal matrix and  k 1 0 k   0 

  1     Xn  k  2      n   

0 k2  0

 0   0     kn  

(2-89)