Probability and Its Applications Published in association with the Applied Probability Trust

Editors: J. Gani, C.C. Heyde, P. Jagers, T.G. Kurtz

Probability and Its Applications Azencott et al.: Series of Irregular Observations. Forecasting and Model Building. 1986 Bass: Diffusions and Elliptic Operators. 1997 Bass: Probabilistic Techniques in Analysis. 1995 Berglund/Gentz: Noise-Induced Phenomena in Slow-Fast Dynamical Systems: A Sample-Paths Approach. 2006 Biagini/Hu/Øksendal/Zhang: Stochastic Calculus for Fractional Brownian Motion and Applications. 2008 Chen: Eigenvalues, Inequalities and Ergodic Theory. 2005 Costa/Fragoso/Marques: Discrete-Time Markov Jump Linear Systems. 2005 Daley/Vere-Jones: An Introduction to the Theory of Point Processes I: Elementary Theory and Methods. 2nd ed. 2003, corr. 2nd printing 2005 Daley/Vere-Jones: An Introduction to the Theory of Point Processes II: General Theory and Structure. 2nd ed. 2008 de la Peña/Gine: Decoupling: From Dependence to Independence, Randomly Stopped Processes U-Statistics and Processes Martingales and Beyond. 1999 Del Moral: Feynman-Kac Formulae. Genealogical and Interacting Particle Systems with Applications. 2004 Durrett: Probability Models for DNA Sequence Evolution. 2002, 2nd ed. 2008 Galambos/Simonelli: Bonferroni-Type Inequalities with Equations. 1996 Gani (ed.): The Craft of Probabilistic Modelling. A Collection of Personal Accounts. 1986 Gut: Stopped Random Walks. Limit Theorems and Applications. 1987 Guyon: Random Fields on a Network. Modeling, Statistics and Applications. 1995 Kallenberg: Foundations of Modern Probability. 1997, 2nd ed. 2002 Kallenberg: Probabilistic Symmetries and Invariance Principles. 2005 Lai/de la Peña/Shao: Self-Normalized Processes. Limit Theory and Statistical Applications. 2009 Last/Brandt: Marked Point Processes on the Real Line. 1995 Molchanov: Theory of Random Sets. 2005 Nualart: The Malliavin Calculus and Related Topics, 1995, 2nd ed. 2006 Rachev/Rueschendorf: Mass Transportation Problems. Volume I: Theory and Volume II: Applications. 1998 Resnick: Extreme Values, Regular Variation and Point Processes. 1987 Schmidli: Stochastic Control in Insurance. 2008 Schneider/Weil: Stochastic and Integral Geometry. 2008 Serfozo: Basics of Applied Stochastic Processes. 2009 Shedler: Regeneration and Networks of Queues. 1986 Silvestrov: Limit Theorems for Randomly Stopped Stochastic Processes. 2004 Thorisson: Coupling, Stationarity and Regeneration. 2000

Richard Serfozo

Basics of Applied Stochastic Processes

123

Richard Serfozo

Georgia Institute of Technology School of Industrial & Systems Engineering 765 Ferst Drive NW., Atlanta GA 30332-0205 USA [email protected]

Series Editors: Joe Gani Chris Heyde

Thomas G. Kurtz

Centre for Mathematics and its Applications Mathematical Sciences Institute Australian National University Canberra, ACT 0200 Australia [email protected]

Department of Mathematics University of Wisconsin - Madison 480 Lincoln Drive Madison, WI 53706-1388 USA [email protected]

Peter Jagers Mathematical Statistics Chalmers University of Technology and Göteborg (Gothenburg) University 412 96 Göteborg Sweden [email protected]

ISBN: 978-3-540-89331-8 DOI: 10.1007/978-3-540-89332-5

e-ISBN: 978-3-540-89332-5

Probability and Its Applications ISSN print edition: 1431-7028 Library of Congress Control Number: 2008939432 Mathematics Subject Classification (2000): 60-02, 60-J10, 60-J27, 60-K05, 60-J25 c 2009 Springer-Verlag Berlin Heidelberg
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: WMXDesign GmbH , Heidelberg Printed on acid-free paper 9 8 7 6 5 4 3 2 1 springer.com

I dedicate this book to Joan, Kip, Shannyn, Jillian and Max Serfozo

Preface

The phrase “Applied Stochastic Processes“ refers to stochastic processes that are commonly used as mathematical models of random phenomena that evolve over time or space. Since randomness is ubiquitous in our universe, and maybe beyond, the application areas have been very diverse. Here are some examples. Telecommunications: Sizing networks, antenna coverage, traffic control, alternate routing, and voice recognition. Computers: Network design, parallel processing, artificial intelligence, pattern recognition, and performance optimization. Manufacturing: Forecasting, planning, scheduling, facility location, and resource management. Finance: Portfolios, option pricing, pension funds, and forecasting. Insurance: Risk analysis, demographics, investments, and diversification. Internet: Design, control, optimal searching, parallel processing, advertising, and pattern recognition. Call-centers: Forecasting, staffing, alternate routing, and optimal design. Airlines: Scheduling, maintenance, ticket pricing, and overbooking. Supply chains: Network design, inventory control, transhipping, alternate sources, and contracting. Military: Logistics, scheduling, maintenance, targeting, intelligence, purchasing, and war games. Infrastructure: Reliability and maintenance of roads, buildings, bridges, dams, levees, and utilities. Airports: Traffic control, emergencies, security, and runway design. Inventory control: Retail and rental items, blood, oil, water, and food. Security: Computers, homeland, banks, phones, and data files. Medicine: DNA sequencing, diagnoses, epidemics, and vaccines. Energy: Planning, control, sharing, storage, and disasters.

vii

viii

Other major applications have been in academic disciplines (e.g., Statistics, Mathematics, Engineering, Physics, Biology, Social Sciences and Business), and in subjects related to government (e.g., NASA, NIH and NIST). The documentation of stochastic applications is in company and government technical reports, academic conference proceedings, and journals. Journals that publish research on applied stochastic processes include Advances in Applied Probability, Annals of Applied Probability, Journal of Applied Probability, Probability in the Engineering and Informational Sciences, Queueing Systems: Theory and Applications, and Stochastic Processes and their Applications. The focus of this book is on the principal stochastic processes used in applications that are as follows. This list corresponds to the chapter titles. 1. Markov Chains in Discrete Time 2. Renewal and Regenerative Processes 3. Poisson Processes 4. Continuous-time Markov Chains 5. Brownian Motion. The book describes basic properties of these stochastic processes and illustrates how to use the processes to model systems and solve problems. The presentation is at an introductory level for readers familiar with random variables, distribution functions, manipulations with expectations, and elementary real analysis. Knowledge of stochastic processes or measure theory is not required. A review of conditional probabilities is in the first chapter, and additional background material on probability and real analysis is summarized in the appendix. The book has two aims. One aim is to present theorems and examples of applied stochastic processes as in most introductory textbooks. So the book would be suitable for one or two courses on applied stochastic processes. The second aim is to go beyond an introduction and provide a comprehensive description of the processes in the first four chapters mentioned above, and a considerable coverage of Brownian motion (not including stochastic integration). In this regard, the book emphasizes the following. • Careful and complete proofs that illustrate stochastic reasoning and the algebra and calculus of probabilities and expectations. • The use of point processes as a vehicle to represent special transition times in Markov chains, space-time Poisson processes, Brownian/Poisson particle systems, and regeneration times in complex systems. • Techniques for constructing or formulating processes (e.g., clock times, sample-process representations for Poisson processes, marking and transforming of processes, and subordination of processes). • Mathematical tools and techniques for stochastic analysis including Laplace functionals and Palm probabilities for point processes, coupling, L´evy

ix

formulas for functionals of Markov chains, martingales, stopping times, functional central limit theorems, and convergence concepts. • Poisson processes in space as well as time, marked Poisson processes, and Poisson limits of sparse point processes. • Regenerative phenomena (e.g., crude regenerations in key renewal theorem, and regenerate-increment processes as a framework for various strong laws of large numbers and central limit theorems). A major theme of the book, and of applied stochastic processes in general, is the establishment of limiting distributions and averages for quantities of interest. Accordingly, there is an extensive coverage of characterizations of limiting distributions of the principal processes, strong laws of large numbers for evaluating limiting averages for the processes, central limit theorems that describe deviations of the averages, limit theorems for approximating sparse point processes by Poisson processes, and functional central limit theorems for approximating various processes by functions of Brownian motion. Each chapter contains numerous examples and exercises that illustrate applications or extensions of the theorems. Several sections are devoted to stochastic networks, queueing systems, branching populations, reversible processes, Markov chain Monte Carlo models, compound Poisson processes, Gaussian processes, and Brownian bridge. Important topics in applied stochastic processes that the book does not cover include diffusion processes, stationary processes, stochastic integrals and differential equations, interacting particle systems, simulation, Gibbs fields, finance models, large deviations, and stochastic control (Markov decision models). Most of these topics are in more advanced texts, and the rest are broad enough to be subjects of specialized monographs. I will close with a few acknowledgements. First, I am grateful to William Feller for writing his 1950 book Introduction to Probability and its Applications. It opened my eyes to the notion that “One can make sense out of the nonsense of randomness”, which sparked my interest in probability. I cannot give enough thanks to Erhan Cinlar, my Ph.D. advisor, who has been a kind friend as well as a mentor. I am also very appreciative to those who have developed the knowledge of stochastic processes — I made extensive use of their works in writing the book, especially the work of Olav Kallenberg 2004. My loving wife Joan contributed to the clarity of the exposition by advising me to “reach the reader” by adopting a writing style that is not overly terse and easy to read. My colleague Steve Hackman prodded me along similar lines on editorial issues. Careful readings by Brian Fralix, Anton Kleywegt, Evsey Morozov, Christian Rau, and Georgia Tech students were very helpful in catching many typos and errors. I thank all of you for helping me on this project. Richard Serfozo

Contents

1

Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Probabilities of Sample Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Construction of Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Stopping Times and Strong Markov Property . . . . . . . . . . . . . . 1.6 Classification of States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Hitting and Absorbtion Probabilities . . . . . . . . . . . . . . . . . . . . . . 1.8 Branching Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.9 Stationary Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10 Limiting Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.11 Regenerative Property and Cycle Costs . . . . . . . . . . . . . . . . . . . 1.12 Strong Laws of Large Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.13 Examples of Limiting Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.14 Optimal Design of Markovian Systems . . . . . . . . . . . . . . . . . . . . 1.15 Closed Network Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.16 Open Network Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.17 Reversible Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.18 Markov Chain Monte Carlo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.19 Markov Chains on Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.20 Limit Theorems via Coupling . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.21 Criteria for Positive Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . 1.22 Review of Conditional Probabilities . . . . . . . . . . . . . . . . . . . . . . . 1.23 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 2 5 8 10 16 19 26 30 33 40 42 45 50 53 55 59 61 68 71 73 76 81 84

2

Renewal and Regenerative Processes . . . . . . . . . . . . . . . . . . . . . 2.1 Renewal Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Strong Laws of Large Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 The Renewal Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99 99 104 107

xi

xii

Contents

2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20

Future Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Renewal Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Blackwell’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Key Renewal Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Regenerative Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Limiting Distributions for Markov Chains . . . . . . . . . . . . . . . . . Processes with Regenerative Increments . . . . . . . . . . . . . . . . . . . Average Sojourn Times in Regenerative Processes . . . . . . . . . . Batch-Service Queueing System . . . . . . . . . . . . . . . . . . . . . . . . . . Central Limit Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Terminating Renewal Processes . . . . . . . . . . . . . . . . . . . . . . . . . . Stationary Renewal Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . Refined Limit Laws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Proof of the Key Renewal Theorem* . . . . . . . . . . . . . . . . . . . . . . Proof of Blackwell’s Theorem* . . . . . . . . . . . . . . . . . . . . . . . . . . . Stationary-Cycle Processes* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

114 114 116 118 121 126 126 129 132 135 139 144 148 151 153 155 156

3

Poisson Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Poisson Processes on R+ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Characterizations of Classical Poisson Processes . . . . . . . . . . . . 3.3 Location of Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Functions of Point Locations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Poisson Processes on General Spaces . . . . . . . . . . . . . . . . . . . . . . 3.6 Integrals and Laplace Functionals of Poisson Processes . . . . . . 3.7 Poisson Processes as Sample Processes . . . . . . . . . . . . . . . . . . . . 3.8 Deterministic Transformations of Poisson Processes . . . . . . . . . 3.9 Marked and Space-Time Poisson Processes . . . . . . . . . . . . . . . . 3.10 Partitions and Translations of Poisson Processes . . . . . . . . . . . . 3.11 Markov/Poisson Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Poisson Input-Output Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.13 Network of Mt /Gt /∞ Stations . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.14 Cox Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.15 Compound Poisson Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.16 Poisson Law of Rare Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.17 Poisson Convergence Theorems* . . . . . . . . . . . . . . . . . . . . . . . . . . 3.18 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

169 170 173 176 179 181 183 188 190 194 196 201 203 206 211 214 216 218 225

4

Continuous-Time Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Markov Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Transition Probabilities and Transition Rates . . . . . . . . . . . . . . 4.5 Existence of CTMCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Uniformization, Travel Times and Transition Probabilities . . .

241 242 245 247 251 253 255

Contents

4.7 4.8 4.9 4.10 4.11 4.12 4.13 4.14 4.15 4.16 4.17 4.18 4.19 4.20 4.21 4.22

xiii

Stationary and Limiting Distributions . . . . . . . . . . . . . . . . . . . . . Regenerative Property and Cycle Costs . . . . . . . . . . . . . . . . . . . Ergodic Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Expectations of Cost and Utility Functions . . . . . . . . . . . . . . . . Reversibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Modeling of Reversible Phenomena . . . . . . . . . . . . . . . . . . . . . . . Jackson Network Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Multiclass Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Poisson Transition Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Palm Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . PASTA at Poisson Transitions . . . . . . . . . . . . . . . . . . . . . . . . . . . Relating Palm and Ordinary Probabilities . . . . . . . . . . . . . . . . . Stationarity Under Palm Probabilities . . . . . . . . . . . . . . . . . . . . . G/G/1, M/G/1 and G/M/1 Queues . . . . . . . . . . . . . . . . . . . . . . Markov-Renewal Processes* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

258 263 264 269 272 277 282 287 291 299 303 306 310 314 321 323

5

Brownian Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Definition and Strong Markov Property . . . . . . . . . . . . . . . . . . . 5.2 Brownian Motion as a Gaussian Process . . . . . . . . . . . . . . . . . . . 5.3 Maximum Process and Hitting Times . . . . . . . . . . . . . . . . . . . . . 5.4 Special Random Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Optional Stopping of Martingales . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Hitting Times for Brownian Motion with Drift . . . . . . . . . . . . . 5.8 Limiting Averages and Law of the Iterated Logarithm . . . . . . . 5.9 Donsker’s Functional Central Limit Theorem . . . . . . . . . . . . . . . 5.10 Regenerative and Markov FCLTs . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 Peculiarities of Brownian Sample Paths . . . . . . . . . . . . . . . . . . . 5.12 Brownian Bridge Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.13 Geometric Brownian Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.14 Multidimensional Brownian Motion . . . . . . . . . . . . . . . . . . . . . . . 5.15 Brownian/Poisson Particle System . . . . . . . . . . . . . . . . . . . . . . . . 5.16 G/G/1 Queues in Heavy Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . 5.17 Brownian Motion in a Random Environment . . . . . . . . . . . . . . . 5.18 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

341 342 345 349 352 354 358 361 364 368 373 377 379 383 385 387 389 393 394

6

Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Probability Spaces and Random Variables . . . . . . . . . . . . . . . . . 6.2 Table of Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Random Elements and Stochastic Processes . . . . . . . . . . . . . . . . 6.4 Expectations as Integrals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Functions of Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . .

405 405 407 409 410 412

xiv

Contents

6.6 6.7 6.8 6.9

Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conditional Probabilities and Expectations . . . . . . . . . . . . . . . . Existence of Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . . Convergence Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

415 417 419 421

Bibliographical Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437

Chapter 1

Markov Chains

A sequence of random variables X0 , X1 , . . . with values in a countable set S is a Markov chain if at any time n, the future states (or values) Xn+1 , Xn+2 , . . . depend on the history X0 , . . . , Xn only through the present state Xn . Markov chains are fundamental stochastic processes that have many diverse applications. This is because a Markov chain represents any dynamical system whose states satisfy the recursion Xn = f (Xn−1 , Yn ), n ≥ 1, where Y1 , Y2 . . . are independent and identically distributed (i.i.d.) and f is a deterministic function. That is, the new state Xn is simply a function of the last state and an auxiliary random variable. Such system dynamics are typical of those for queue lengths in call centers, stresses on materials, waiting times in production and service facilities, inventories in supply chains, parallel-processing software, water levels in dams, insurance funds, stock prices, etc. This chapter begins by describing the basic structure of a Markov chain and how its single-step transition probabilities determine its evolution. For instance, what is the probability of reaching a certain state, and how long does it take to reach it? The next and main part of the chapter characterizes the stationary or equilibrium distribution of Markov chains. These distributions are the basis of limiting averages of various cost and performance parameters associated with Markov chains. Considerable discussion is devoted to branching phenomena, stochastic networks, and time-reversible chains. Included are examples of Markov chains that represent queueing, production systems, inventory control, reliability, and Monte Carlo simulations. Before getting into the main text, a reader would benefit by a brief review of conditional probabilities in Section 1.22 of this chapter and related material on random variables and distributions in Sections 1–4 in the Appendix. The rest of the Appendix, which provides more background on probability, would be appropriate for later reading.

R. Serfozo, Basics of Applied Stochastic Processes, Probability and its Applications. c Springer-Verlag Berlin Heidelberg 2009 

1

2

1 Markov Chains

1.1 Introduction This section introduces Markov chains and describes a few examples. A discrete-time stochastic process {Xn : n ≥ 0} on a countable set S is a collection of S-valued random variables defined on a probability space (Ω, F, P ). The P is a probability measure on a family of events F (a σ-field) in an event-space Ω.1 The set S is the state space of the process, and the value Xn ∈ S is the state of the process at time n. The n may represent a parameter other than time such as a length or a job number. The finite-dimensional distributions of the process are P {X0 = i0 , . . . , Xn = in },

i0 , . . . , in ∈ S, n ≥ 0.

These probabilities uniquely determine the probabilities of all events of the process. Consequently, two stochastic processes (defined on different probability spaces or the same one) are equal in distribution if their finite-dimensional distributions are equal. Various types of stochastic processes are defined by specifying the dependency among the variables that determine the finitedimensional distributions, or by specifying the manner in which the process evolves over time (the system dynamics). A Markov chain is defined as follows. Definition 1. A stochastic process X = {Xn : n ≥ 0} on a countable set S is a Markov Chain if, for any i, j ∈ S and n ≥ 0, P {Xn+1 = j|X0 , . . . , Xn } = P {Xn+1 = j|Xn }, P {Xn+1 = j|Xn = i} = pij .

(1.1) (1.2)

The pij is the probability that the Markov  chain jumps from state i to state j. These transition probabilities satisfy j∈S pij = 1, i ∈ S, and the matrix P = (pij ) is the transition matrix of the chain. Condition (1.1), called the Markov property, says that, at any time n, the next state Xn+1 is conditionally independent of the past X0 , . . . , Xn−1 given the present state Xn . In other words, the next state is dependent on the past and present only through the present state. The Markov property is an elementary condition that is satisfied by the state of many stochastic phenomena. Consequently, Markov chains, and related continuous-time Markov processes, are natural models or building blocks for applications. Condition (1.2) simply says the transition probabilities do not depend on the time parameter n; the Markov chain is therefore “time-homogeneous”. If the transition probabilities were functions of time, the process Xn would be a non-time-homogeneous Markov chain. Such chains are like time-homogeneous 1

Further details on probability spaces are in the Appendix. We follow the convention of not displaying the space (Ω, F, P ) every time random variables or processes are introduced; it is mentioned only when needed for clarity.

1.1 Introduction

3

chains, but the time dependency introduces added accounting details that we will not address here. See Exercises 12 and 13 for further insights. Since the state space S is countable, we will sometimes label the states by integers, such as S = {0, 1, 2, . . .} (or S = {1, . . . , m}). Under this labeling, the transition matrix has the form ⎡ ⎤ p00 p01 p02 · · · ⎢p10 p11 p12 · · ·⎥ ⎢ ⎥ ⎥ P=⎢ ⎢p20 p21 p22 · · ·⎥ ⎣. . . . . . . . . . . . . . ⎦ .............. We end this section with a few preliminary examples. Example 2. Binomial Markov Chain. A Bernoulli process is a sequence of independent trials in which each trial results in a success or failure with respective probabilities p and q = 1−p. Let Xn denote the number of successes in n trials, for n ≥ 1. By direct reasoning, it follows that Xn has a binomial distribution with parameters n and p:

n k P {Xn = k} = p (1 − p)n−k , 0 ≤ k ≤ n. k Now, suppose at the nth trial that Xn = i. Then at the next trial, Xn+1 will equal i + 1 or i with probabilities p and 1 − p, respectively, regardless of the values of X1 , . . . , Xn−1 . Thus Xn is a Markov chain with transition probabilities pi,i+1 = p, pii = 1 − p and pij = 0 otherwise. This binomial Markov chain is a special case of the following random walk. Example 3. Random Walk. Suppose Y1 , Y2 , . . . are i.i.d. integer-valued random variables, and define X0 = 0 and Xn =

n 

Ym ,

n ≥ 1.

m=1

The process Xn is a random walk on the set of integers S, where Yn is the step size at time n. A random walk represents a quantity that changes over time (e.g., a stock price, an inventory level, or a gambler’s fortune) such that its increments (step sizes) are i.i.d. Since Xn+1 = Xn + Yn+1 , and Yn+1 is independent of X0 , . . . , Xn , it follows that, for any i, j ∈ S and n ≥ 0, P {Xn+1 = j|X0 , . . . , Xn−1 , Xn = i} = P {Xn + Yn+1 = j|Xn = i} = P {Y1 = j − i}. Therefore, the random walk Xn is a Markov chain on the nonnegative integers S with transition probabilities pij = P {Y1 = j − i}.

4

1 Markov Chains

When the step sizes Yn take values 1 or −1 with p = P {Y1 = 1} and q = P {Y1 = −1}, the chain Xn is a simple random walk. Its transition probabilities, for each i, are pi,i+1 = p,

pi,i−1 = q,

pij = 0,

for j = i + 1 or i − 1.

This type of walk restricted to a finite state space is described next. Example 4. Gambler’s Ruin. Consider a Markov chain on S = {0, 1, . . . , m} with transition matrix ⎤ ⎡ 100..... ⎢q 0 p 0 . . . ⎥ ⎥ ⎢ ⎢0 q 0 p 0 . . ⎥ ⎥ ⎢ P=⎢ ⎥ ⎢ . . . . . . . . .⎥ ⎣ . . . 0 q 0 p⎦ . . . . . .01 One can interpret the state of the Markov chain as the fortune of a Gambler who repeatedly plays a game in which the Gambler wins or loses $1 with respective probabilities p and q = 1 − p. If the fortune reaches state 0, the Gambler is ruined since p00 = 1 (state 0 is absorbing — the chain stays there forever). On the other hand, if the fortune reaches m, the Gambler retires with the fortune m since pmm = 1 (m is another absorbing state). A versatile generalization to state-dependent gambles (and other applications as well) is with a transition matrix ⎤ ⎡ r0 p0 0 . . . . . . . . . . . . ⎢q1 r1 p1 0 . . . . . . . .⎥ ⎥ ⎢ ⎢ 0 q2 r2 p2 0 . . . .⎥ ⎥ P=⎢ ⎢ . . . . . . . . . . . . . . . . . .⎥ ⎥ ⎢ ⎣. . . . 0 qm−1 rm−1 pm−1 ⎦ . . . . . . . . . . qm rm In this case, the outcome of the game depends on the Gambler’s fortune. When the fortune is i, the Gambler either wins or loses $1 with respective probabilities pi or qi , or breaks even (the fortune does not change) with probability ri . Another interpretation is that the state of the chain is the location of a random walk with state-dependent steps of size −1, 0, or 1. Markov chains are common models for a variety of systems and phenomena, such as the following, in which the Markov property is “reasonable”. Example 5. Flexible Manufacturing System. Consider a machine that is capable of producing three types of parts. The state of the machine at time period n is denoted by a random variable Xn that takes values in S = {0, 1, 2, 3}, where 0 means the machine is idle and i = 1, 2 or 3 means the machine produces a type i in the time period. Suppose the machine’s production schedule is Markovian in the sense that the next type of part it produces, or a possible

1.2 Probabilities of Sample Paths

5

idle period, depends only on its current state, and the probabilities of these changes do not depend on time. Then Xn is a Markov chain. For instance, its transition matrix might be ⎤ ⎡ 1/5 1/5 1/5 2/5 ⎢1/10 1/2 1/10 3/10⎥ ⎥ P=⎢ ⎣ 1/5 0 1/5 3/5 ⎦ 1/5 0 2/5 2/5 Such probabilities can be estimated as in Exercise 65, provided one can observe the evolution of the system. Otherwise, the probabilities can be determined by subjective reasoning or other techniques. Note that at any period in which the machine produces a type 1 part, in the next period it produces a type 1, 2 or 3 part with respective probabilities 1/2, 1/10, and 3/10. Also, whenever the machine is idle, it remains idle in the next period with probability 1/5. Consequently, the probability the machine is idle for m periods is (4/5)(1/5)m−1, which is a geometric distribution with parameter 4/5 and mean 1.25 periods; see Exercise 6.

1.2 Probabilities of Sample Paths A basic issue in analyzing the structure of a stochastic process is to describe its finite-dimensional distributions. This section shows that these distributions for a Markov chain Xn are simply products of its transition probabilities and the probability distribution of the initial state X0 . Finite-dimensional distributions and more general properties of sample paths2 are conveniently expressed by n-step transition probabilities, which are obtained as the nth product of the transition matrix. Proposition 6. Suppose Xn is a Markov chain on S with transition probabilities pij and initial distribution αi = P {X0 = i}. Then, for any i0 , . . . , in ∈ S and n ≥ 0, P {X0 = i0 , . . . , Xn = in } = αi0 pi0 ,i1 · · · pin−1 ,in . Proof. Proceeding by induction, this statement is obvious for n = 0. Now, assume it is true for some n, and let An = {X0 = i0 , . . . , Xn = in }. Then the statement is true for n + 1, since P (An+1 ) = P (An )P {Xn+1 = xn+1 |An } = αi0 pi0 ,i1 · · · pin−1 ,in pin ,in+1 , 2

A sample path of a stochastic process Xn is a realization of it as a function of time. For instance, if X0 (ω) = i0 , . . . , Xn (ω) = in , then i0 , . . . , in is the sample path associated with the outcome ω.

6

1 Markov Chains

where P {Xn+1 = xn+1 |An } = pin ,in+1 by the Markov property. Proposition 6 says that the probability the Markov chain traverses a path i0 , i1 , . . . , in is just the multiplication pi0 ,i1 · · · pin−1 ,in of the probabilities of these transitions. Therefore, the probability that the Markov chain up to time n has a sample path in a subset P of S n+1 is  P {X0 = i0 }pi0 ,i1 · · · pin−1 ,in . (1.3) P {(X0 , . . . , Xn ) ∈ P} = (i0 ,...,in )∈P

For instance, if the Xn are the monthly profits of a company, then the probability its profits will increase throughout n months is ∞ 

P {X0 ≤ X1 ≤ . . . ≤ Xn |X0 = i0 } =

pi0 ,i1 · · ·

i1 =i0

∞ 

pin−1 ,in .

in =in−1

Also, the profit in the third month has the distribution   pi0 i1 pi1 i2 pi2 j . P {X3 = j|X0 = i0 } = i1 ∈S

i2 ∈S

Many probabilities like these for the Markov chain can be expressed conveniently in terms of the transition matrix P = (pij ) and its nth product Pn , n ≥ 0. By definition, P0 = I (the identity matrix), and Pn = Pn−1 P, for n ≥ 1. Let pnij denote the (i, j)th entry of Pn (so Pn = (pnij )).3 Then by the definition of matrix multiplication,  pi,i1 pi1 ,i2 · · · pin−1 ,j . (1.4) pnij = i1 ,...,in−1 ∈S n−1

Remark 7. n-Step Probabilities. The probability P {Xn = j|X0 = i} is the sum of the probabilities of all paths of the form i, i1 , . . . , in−1 , j, which is the sum in (1.4). Consequently, P {Xn = j|X0 = i} = pnij . This probability can be obtained upon computing Pn . Furthermore, denoting the initial distribution αi = P {X0 = i} as a row vector α = (αi ), we have P {Xn = j} = ( αPn )j , which is the jth value of the row-vector αPn . Interestingly, the multiplication property of matrices Pm+n = Pm Pn , for m, n ≥ 1, yields the Chapman-Kolmogorov equations 3

n Keep in mind that n in pn ij is not the usual multiplication operation, but P “is” the nth product of P.

1.2 Probabilities of Sample Paths

7



pm+n = ij

n pm ik pkj ,

i, j ∈ S.

k∈S

This says that the probability the chain moves from i to j in m + n steps is equal to the probability that it moves from i to any k ∈ S in m steps, and then it moves from k to j in n more steps. The following examples involve n-step probabilities. Example 8. Expected Costs or Utilities. Suppose there is a value fn (i) ∈ R associated with the Markov chain being in state i at time n. The value could be a cost, reward or some utility parameter. Then the mean value at time n, assuming it exists, is  E[fn (Xn )] = P {Xn = j}fn (j) j∈S

=



( αPn )j fn (j) = αPn fn ,

j∈S

where fn = (fn (i)) is a column vector of the values. Furthermore, the mean value up to time n is E[

n 

fm (Xm )] =

m=1

n 

αPm fm .

m=1

Example 9. Taboo Probabilities. There are many probabilities for a Markov chain involving paths that avoid a specified region in the state space. In particular, consider the taboo probability n A pij

= P {X1 , . . . , Xn−1 ∈ A, Xn = j|X0 = i},

that the chain Xn moves from state i to state j in n steps without entering a taboo set A ⊂ S. Then as in (1.3) and Remark 7,  n n pi,i1 pi1 ,i2 · · · pin−1 ,j = qij , i, j ∈ Ac , A pij = i1 ,...,in−1 ∈Ac n where Qn = (qij ) is the nth product of Q = (pij ; i, j ∈ Ac ) (the matrix P c restricted to A = S\A).

Example 10. Maxima of a Markov Chain. Suppose the Markov chain Xn has the state space S = {1, 2, . . .} and consider the maximum process Mn = max Xm , 0≤m≤n

n ≥ 0.

For instance, if Xn is the stress on a part at time n and X0 = i, then the probability the stress does not exceed a level  > i up to time n is

8

1 Markov Chains

P {Mn ≤ |X0 = i} = P {X1 ≤ , . . . , Xn ≤ |X0 = i} =

 

n qij ,

j=1 n where Qn = (qij ) is the nth product of Q = (pij ; i, j ≤ ). The Mn is generally not a Markov chain, but it is when the Xn are i.i.d. (see Exercise 17).

1.3 Construction of Markov Chains This section addresses the following questions. Is there a general framework for constructing or identifying Markov chains? Is there a Markov chain associated with any transition matrix? If so, how is it constructed? How can one simulate a Markov chain? These questions are answered by the first result that shows how to formulate a Markov chain as a function of i.i.d. random variables. Recall that the random walk in Example 3 is “constructed” with i.i.d. random variables. That is, its evolution is represented by the recursive equation Xn = Xn−1 + Yn , n ≥ 1, where X0 = 0, and Yn are i.i.d. random variables. Here is an analogous and more general construction of a Markov chain via a general recursive equation. Proposition 11. Suppose {Xn : n ≥ 0} is a stochastic process on S of the form Xn = f (Xn−1 , Yn ), n ≥ 1, (1.5) where f : S × S  → S and Y1 , Y2 , . . . are i.i.d. random variables with values in a general space S  that are independent of X0 . Then Xn is a Markov chain with transition probabilities pij = P {f (i, Y1 ) = j}. Proof. The result will follow upon showing that, for any i, j and n, P {Xn+1 = j|X0 , . . . , Xn−1 , Xn = i} = P {f (i, Yn+1 ) = j|X0 , . . . , Xn−1 , Xn = i} = P {f (i, Yn+1 ) = j} = pij . The first equality follows since Xn+1 = f (i, Yn+1 ) given Xn = i. The second equality is due to the fact that Yn+1 is independent of (X0 , . . . , Xn ), because this vector, by (1.5), is a function of (X0 , Y1 , . . . , Yn ), which is independent of Yn+1 by assumption. The last equality follows by the definition of pij and fact that Yn+1 and Y1 have the same distribution. Proposition 11 is useful for identifying stochastic processes that are Markov chains. This approach is often easier than verifying the Markov property directly; illustrations are in the next section. We now establish that any Markov chain can be constructed as in Proposition 11. The proof uses the following fact (Exercise 11 gives a similar result

1.3 Construction of Markov Chains

9

for general random variables and discusses how the result is used to generate random samples from a distribution). Remark 12. Uniform Representation of a Random Variable. Let α be a probability measure on S = {0, 1, . . .}, and let U be a random variable that has a uniform distribution on [0, 1]. Define X = h(U ), where h(u) = j

if u ∈ Ij , for some j ∈ S,

(1.6)

j−1 j and Ij = [ k=0 αk , k=0 αk ). Then P {X = j} = αj . This follows since P {h(U ) = j} = P {U ∈ Ij } = αj . Theorem 13. (Construction of Markov Chains) Let pij be Markovian transition probabilities, and let α be a probability measure on S. Label the elements of S such that S = {0, 1, . . .}. Suppose U0 , U1 , . . . are i.i.d. with a uniform distribution on [0, 1]. Assume X0 = h(U0 ), where h is given by (1.6). Define Xn = f (Xn−1 , Un ), n ≥ 1, where, for each i, f (i, u) = j

if u ∈ Iij , for some j ∈ S,

(1.7)

j−1 j and Iij = [ k=0 pik , k=0 pik ). Then {Xn : n ≥ 0} is a Markov chain with initial distribution α and transition probabilities pij . Proof. By Remark 12, X0 has the distribution αj . Furthermore, by Proposition 11, Xn is a Markov chain with transition probabilities P {f (i, U1) = j} = P {U1 ∈ Iij } = pij . We are now ready to establish that there exists a Markov chain associated with any transition matrix. Corollary 14. (Existence of Markov Chains) For any Markovian transition probabilities pij and probability measure α on S, there exists a Markov chain {Xn : n ≥ 0} on S with transition probabilities pij and initial distribution α. Proof. Corollary 6 in the Appendix says that, for any specified countable collection of distributions on R, there exist a probability space and independent random variables on it that have the specified distributions. This justifies the existence of the random variables Un in Theorem 13, which in turn justifies the existence of a Markov chain {Xn : n ≥ 0} on S with transition probabilities pij and initial distribution α. Theorem 13 also yields the following simulation procedure. Remark 15. Simulation of a Markov Chain. One can generate an n-step path i0 , i1 , . . . , in of a Markov chain with transition probabilities pij and initial distribution α as follows. First generate values u0 , u1 , . . . , un from a uniform

10

1 Markov Chains

distribution on [0, 1]. Then set i0 = h(u0 ) and im = f (im−1 , um ), 1 ≤ m ≤ n− 1, where h and f are defined respectively by (1.6) and (1.7). The resulting i0 , i1 , . . . , in is the desired n-step path of the Markov chain.

1.4 Examples This section contains more examples of Markov chains. Some of these are justified by verifying the Markov property, while others are justified by the recursive-equation framework in Proposition 11. Example 16. Machine Deterioration Model. A machine is continuously used to perform a certain job (e.g., a fork-lift truck in a bottling plant, or a metal cutting tool), and Xn denotes its state of deterioration at time n, where the set of states is S = {0, 1, . . . , }. When the machine’s deterioration is in state i < , in the next time period it either remains at that level with probability pii or it increases to a level j > i with probability pij > 0. When the machine is in state , in the next time period it either remains there with probability p or it enters state 0 with probability p0 = 1 − p . Entering state 0 means the machine is replaced with a new one (or is repaired to be like new). These movements are independent of the past history of the machine. Under these assumptions, Xn is a Markov chain with transition matrix ⎡ ⎤ p00 p01 p02 . . . . . . . ⎢ 0 p11 p12 p13 . . . . .⎥ ⎢ ⎥ ⎥ P=⎢ ⎢ 0 0 0 p22 p23 . . ⎥ ⎣. . . . . . . . . . . . . . . .⎦ p0 . . . . . 0 0 p Example 17. (s, S) Inventory Model. A commodity is stocked in a warehouse to satisfy continuing demands, and the demands D1 , D2 , . . . in time periods 1, 2, . . . are i.i.d. nonnegative integer-valued random variables. The inventory is controlled by the following (s, S) inventory-control policy, where s < S are predetermined integers. At the end of period n − 1, the inventory level Xn−1 is observed, and one of the following actions is taken: • Replenish the stock (instantaneously) up to the level S if Xn−1 ≤ s. • Do not replenish the stock if Xn−1 > s. Assume X0 ≤ S for simplicity and that X0 is independent of the Dn . Under this control policy, the inventory level satisfies the recursion if Xn−1 ≤ s S − Dn Xn = Xn−1 − Dn if s < Xn−1 ≤ S, n ≥ 1. Therefore, by Proposition 11, Xn is a Markov chain with

1.4 Examples

11

pij =

P {D1 = S − j} if i ≤ s P {D1 = i − j} if s < i ≤ S.

Example 18. Movement on a Graph. Suppose that an item moves on the directed graph shown below, where pij is the probability the item moves from node i to node j, independent of its past history. The item spends exactly one time period at each node it visits, and so pii = 0 for each i. Then the location of the item Xn at time n is a Markov chain on the set of nodes S = {1, . . . , 7}. For instance, the nodes could represent machines in a manufacturing facility, where a job moves from machine 1 to 7. It spends one time period at each machine it visits, and its random path through the machines, which requires 4 time periods, is determined by its type or other factors. When a job is finished at machine 7, another job immediately begins at machine 1, and this 4-period manufacturing cycle is repeated indefinitely, with exactly one job in the system at any time. In this setting, the Markov chain Xn records the machine location of a typical job. - 4m 2m Z Z >   Z Z  ZZ Z ~m Z ~ 5m  - 1m 7 Z > 6 >    Z   ZZ ~ 3m  - 6m  

?

Fig. 1.1 Markov Chain on a Graph

Example 19. Success Runs. Suppose Xn is a Markov chain with transition matrix ⎤ ⎡ 1 − p0 p0 0 . . . . . ⎢ 1 − p1 0 p1 0 . . . ⎥ ⎥ ⎢ ⎢ 1 − p2 0 0 p2 0 . . ⎥ ⎥ ⎢ P=⎢ ⎥ ⎢ . . . . . . . . . . . . . . .⎥ ⎣ 1 − pm . . . . 0 0 pm ⎦ ............... This is a model for success runs in a sequence of independent events that result in a success or failure. Namely, Xn denotes the number of successes since the last failure prior to the nth event, and upon having i successes, the next event results in a success with probability pi , or a failure with probability 1 − pi , where 0 < pi < 1. For instance, as a model of accidents in a manufacturing plant, Xn could denote the number of weeks without an accident up to week n. Example 20. Age and Residual Processes for Discrete-Time Renewals. Suppose ξ1 , ξ2 , . . . are i.i.d. positive integer-valued random variables with ri =

12

1 Markov Chains

n P {ξ1 = i}. The times Tn = m=1 ξm form a discrete-time renewal process, where Tn is the time of the nth renewal (general renewal processes are studied in the next chapter). Consider the process Xn = n − Tk−1

if n ∈ [Tk−1 , Tk ).

This is the time since the last renewal prior to or at time n, and Xn is called the age process for the renewals. Clearly, Xn is a success-runs Markov chain as above, where i  pi0 = 1 − pi = ri+1 /(1 − rk ) k=1

is the probability of a renewal in the next time period conditioned that the renewal time was greater than i. A related process is Xn = Tk − n

if n ∈ [Tk−1 , Tk ),

which is the time to the next renewal after time n. The Xn is the residual process for the renewals. It is a Markov chain on S = {0, 1, . . .} that decreases by one unit in each time period until it reaches 0, and then it jumps to state j with probability rj+1 . Thus, its transition probabilities are pi,i−1 = 1, i ≥ 1;

p0j = rj+1 , j ≥ 0;

pij = 0 otherwise.

As an example, suppose that jobs are processed one at a time and the processing times are ξ1 , ξ2 , . . . Then Xn denotes the age of the job being processed at time n; age meaning the length of time the job has been in process counting time n. Here 1 − pi is the probability that a job of age i is finished in the next period. The other random variable Xn denotes the time needed to finish the job being processed at time n. A variety of queueing and inventory models for systems such as computer systems may be more realistic in discrete time rather than continuous time. The next examples are abstract models for such systems. They have the extraordinary property that their system state is a tractable function of i.i.d. random variables based on the fundamental recursion in Proposition 11. Example 21. Discrete-Time M/M/1 Queueing System. Consider a singleserver processing system in which items arrive according to a Bernoulli process as in Example 2, where p is the probability of an arrival (a success) at any discrete time. The service times of the items are i.i.d. and independent of the arrival process, and each service time has a geometric distribution q(1−q)n−1 , n ≥ 1, with parameter q and mean 1/q, where q is the probability of a service completion in any discrete time. Let Xn denote the number of items in the system at time n. Under these assumptions, the Xn satisfy the recursion Xn = (Xn−1 + Vn − Un )+ , where

1.4 Examples

13

Vn is the number of (potential) arrivals and Un is the number of (potential) service completions at time n. Under these assumptions, Un and Vn take values in {0, 1} and (Un , Vn ), n ≥ 1, are i.i.d. We also assume they are independent of X0 . Then by Proposition 11, Xn is a Markov chain on the nonnegative integers S with transition probabilities pij = P {(i + V1 − U1 )+ = j}, which are, for i ≥ 1, p01 = p,

p00 = 1 − p,

pi,i+1 = p(1 − q),

pi,i−1 = q(1 − p),

pi,i = pq + (1 − p)(1 − q).

That is, at each time when at least one item is in service, there may be an arrival and no service completion with probability p(1 − q), or a service completion and no arrival with probability q(1−p), or no change in the system with probability pq + (1 − p)(1 − q) (an arrival and a departure occur, or neither occurs). The Xn is called a discrete-time M/M/1 queueing process4 (analogous queueing processes in continuous time are studied in the next chapters). One can also view Xn as a random walk, or the size of a population in which a single birth or death occurs with the preceding probabilities. One can model other variations of this queueing system (e.g., with limited waiting space for items, or with batch arrivals) by the input-output model in the next example. Hereafter, we will use the following standard shorthand notation. Definition 22. Random variables X and Y are equal in distribution, denoted d by X = Y , if they have the same distribution. The same notation applies if X and Y are random vectors or more general random elements. We will now describe a general Markov chain for analyzing queueing systems, including the preceding M/M/1 system. Example 23. Input-Output Process as a Reflected Random Walk. Consider an input-output system in which the quantity of items Xn in the system (possibly negative) at the end of time period n is in a set S of all integers in a fixed interval [a, b]. The a and b are integers that may be infinite. In each period n, the system has a “potential” input (or increase) Vn and “potential” output (or decrease) Un . The pairs (Un , Vn ), n ≥ 1, are i.i.d. non-negative integer-valued vectors, and are independent of X0 . Part or all of an arriving quantity is rejected (or disregarded) to the extent that its admittance would force the system state to exceed b. Similarly, outputs are disregarded to the extent that they would move the system state below a. Inputs and outputs occurring at the same time cancel each other. 4

In M/M/1, the M stands for memoryless or Markovian when referring to the memoryless geometric distribution (Exercise 7) of inter-arrival times and service times, and 1 refers to the number of servers.

14

1 Markov Chains

Then the quantity in the system at time n satisfies the recursion5 Xn = a ∨ [b ∧ (Xn−1 + Vn − Un )],

n ≥ 1.

(1.8)

By Proposition 11, Xn is a Markov chain on S with transition probabilities pij = P {a ∨ [b ∧ (i + V1 − U1 )] = j}.

(1.9)

This Markov chain has the extraordinary property that its recursive equation (1.8) has a closed form solution for Xn . To see this, consider the random walk or netput process n  Zn = (Vm − Um ). m=1

Note that X n = X0 + Zn −

n  (Xm−1 + Vm − Um − b)+ m=1

 −(a − (Xm−1 + Vm − Um ))+ . This says Xn is the netput Zn compensated by the quantities in the sum that are disregarded to keep the chain in its state space S. In other words, Xn is the random walk Zn “reflected” at the boundaries a and b. One can show by induction, or direct substitution (Exercise 19) that the Xn satisfying (1.8) has the form

 Xn = (X0 + Zn ) ∧ [b + min (Zn − Zm )] (1.10) 1≤m≤n

  max (a + Zn − Zm ) ∧ [b + min (Zn − Z )] . 1≤m≤n

m+1≤≤n

Because Vn − Un are i.i.d., the distribution of the increments of Zn in the preceding expression has the simplification d

(Zn − Z1 , Zn − Z2 , . . . , Zn − Zn−1 ) = (Zn−1 , Zn−2 , . . . , Z1 ). Consequently, Xn in (1.10) has the simpler form (in distribution)

 d Xn = (X0 + Zn ) ∧ (b + min Zm ) 0≤m≤n−1

  (a + Zm ) ∧ (b + max min Z ) . 0≤m≤n−1

(1.11)

0≤≤n−m−1

Here Z0 = 0. This formula describes the distribution of Xn as a function of the distribution of the netput process Zn (the basic “system data”). 5

x ∨ y = max{x, y} and x ∧ y = min{x, y}; we also use x+ = 0 ∨ x.

1.4 Examples

15

In particular, ⎧   ⎨ (X0 + Zn ) ∨ a + max0≤m≤n−1 Zm , d   Xn = ⎩ (X0 + Zn ) ∧ b + min0≤m≤n−1 Zm ,

if b = ∞, if a = −∞.

(1.12)

For instance, when a = 0, b = ∞ and X0 = 0, d

Xn = max Zm . 0≤m≤n

(1.13)

Note that the M/M/1 queue in Example 21 has this nice representation. The preceding input-output model applies to a variety of contexts. For instance, the input-output variables Vn and Un need not be inputs or outputs in the usual sense. Here is an example. Example 24. Waiting Times in a G/G/1 Queue.6 Suppose that items arrive to a processing system at integer-valued times 0 < T1 < T2 < . . . such that the inter-arrival times Un = Tn − Tn−1 are i.i.d., where T0 = 0. The arrival at time Tn has an integer-valued service time Vn , and the Vn are i.i.d. and independent of the arrival times. The service discipline is first-come-firstserved with no preemptions. Of paramount interest are the times that items wait in the queue before receiving service. Let Wn denote the time in queue of the item that arrives at time Tn . Then Dn = Tn + Wn + Vn is the departure time of the nth item. We define Wn by induction. For simplicity, assume the system is empty at time 0. Then clearly W1 = 0 and, assuming W1 , . . . , Wn−1 are defined, 0 if Dn−1 < Tn Wn = Dn−1 − Tn otherwise. Substituting Dn−1 = Tn−1 +Wn−1 +Vn−1 in this expression yields the Lindley recursion n ≥ 1. Wn = (Wn−1 + Vn−1 − Un )+ , Note that Wn is a reflected random walk as in Example 23, where the potential input and output quantities are now the service times and inter-arrival times (here Vn−1 can be replaced by Vn under the assumptions). Therefore, d as in (1.13), Wn is a Markov chain with Wn = max0≤m≤n Zm . The framework in Example 23 also covers the variation in which the waiting times Wn are restricted to not exceed a finite level b. For instance, a system controller may ensure that if an item’s waiting time reaches b, then the item is served by an auxiliary server (possibly at a higher cost, but outside of the model). 6

The G/G/1 stands for i.i.d. inter-arrival times with a general distribution, i.i.d. service times with a general distribution, and processing by 1 server.

16

1 Markov Chains

1.5 Stopping Times and Strong Markov Property We will now begin a detailed study of the evolution of Markov chains. This section describes the strong Markov property, which is a generalization of the one-step look-ahead Markov property (1.1). It is used for evaluating conditional probabilities conditioned at certain “random times” called stopping times. One important consequence is a regenerative property that the times between entrances of a Markov chain to a fixed state are i.i.d. We start with preliminaries on stopping times. Definition 25. A random variable τ that takes values in {0, 1, . . . , ∞} is a stopping time for a process {Xn : n ≥ 0} if, for any finite n, the event {τ = n} is a function of the history X0 , . . . , Xn up to time n. Exercise 23 gives additional characterizations. Stopping times are also called optional times, or Markov times when Xn is a Markov chain. Important examples are hitting times. A hitting time of a subset A ⊂ S by a process Xn is defined by τ = min{n ≥ 1 : Xn ∈ A}. This is infinite, by convention, when no such n exists. The τ is sometimes called a first entrance or return time7 to A. It is a stopping time since {τ = n} = {X1 , . . . , Xn−1 ∈ / A, Xn ∈ A}. Constants, of course, are stopping times. There are many ad hoc examples of stopping times such as τ = min{n ≥ 2 : Xn = Xn−1 = Xn−2 }, the first time the chain remains in the same state for 3 periods. On the other hand, many random times τ for which {τ = n} involves information about the future Xn+1 , Xn+2 , . . . are not stopping times. An example is the last exit time from a set A defined by τ = sup{n ≥ 1 : Xn ∈ A}, which is infinite when the set is empty. The Markov property (1.1) says that, at any time n, knowing the present state Xn , the next state Xn+1 is conditionally independent of the past X0 , . . . , Xn−1 . We will now establish the strong Markov property that at any finite stopping time (or any deterministic time), the future of the process is conditionally independent of the past given the present state, and the distribution of the future is equal to that of the original chain. 7

A variation of τ is τ  = min{n ≥ 0 : Xn ∈ A}. Clearly τ  = τ if X0 ∈ / A and τ  = 0 if X0 ∈ A.

1.5 Stopping Times and Strong Markov Property

17

Theorem 26. (Strong Markov Property) Suppose that τ is a finite-valued stopping time for a Markov chain Xn on S. Then, for any i ∈ S and i1 , i2 , . . . , j1 , . . . , jm ∈ S and m ≥ 1,   P {Xτ +1 = j1 , . . . , Xτ +m = jm X0 = i0 , . . . , Xτ −1 = iτ −1 , Xτ = i} = P {X1 = j1 , . . . , Xm = jm |X0 = i}.

(1.14)

Proof. For simplicity, let us write (1.14) as P (Aτ |Bτ ) = P (A0 |X0 = i),

(1.15)

where An = {Xn+1 = j1 , . . . , Xn+m = jm } and Bn = {X0 = i0 , . . . , Xn−1 = in−1 , Xn = i}. To prove (1.15), first note that by conditioning on τ , P (Aτ |Bτ ) =

∞ 

P (Aτ |Bτ , τ = n)P {τ = n|Bτ }.

(1.16)

n=0

Next, observe that8 P (Aτ |Bτ , τ = n) =

P (An , Bn , τ = n) . P (Bn , τ = n)

Since τ is a stopping time, {τ = n} is determined by X0 , . . . , Xn , and so Bn = Bn ∩ {τ = n} when P (Bn , τ = n) > 0. Using this fact in the preceding display, we have P (An , Bn ) P (Aτ |Bτ , τ = n) = . P (Bn ) Furthermore, expressing the last two probabilities as multiplications of transition probabilities as in Proposition 6 and canceling terms, we obtain P (An , Bn ) = pi,j1 · · · pjm−1 ,jm = P (A0 |X0 = i). P (Bn ) Using the preceding two displays in (1.16) proves (1.15). The Strong Markov Property stated in (1.14) for only m steps in the future also applies to the entire future of the chain as follows. Remark 27. Probabilities of the Infinite Future. Property (1.14) is equivalent to the following: For any i ∈ S and B ∈ S ∞ , 8

In expressions like these, commas are often used instead of set intersection; e.g. P (Aτ , Bn , τ = n) = P (Aτ ∩ Bτ ∩ {τ = n})

18

1 Markov Chains

P {(Xτ +1 , Xτ +2 , . . .) ∈ B|X0 , . . . , Xτ −1 , Xτ = i} = P {(X1 , X2 , . . .) ∈ B|X0 = i}.

(1.17)

This equivalence follows since the distribution (or conditional distribution) of an infinite sequence (Y1 , Y2 , . . .) of random variables is determined by the distributions of its finite parts (Y1 , . . . , Ym ) for m ≥ 1. Another equivalent statement is that, for any bounded function f : S ∞ → R+ , E[f (Xτ +1 , Xτ +2 , . . .)|X0 , . . . , Xτ −1 , Xτ = i] = E[f (X1 , X2 , . . .)|X0 = i].

(1.18)

The equivalence of (1.17) and (1.18) follows by basics of equality in distribution; see Exercise 18 ((a) and (b) in this exercise have the same form as (1.17) and (1.18)). The strong Markov property says, loosely speaking, that a Markov chain regenerates, or starts anew, at a stopping time. For instance, if τ is the hitting time of a state i and it is finite, then since Xτ = i, P {Xτ +m ∈ A} = E[P {Xτ +m ∈ A|X0 , . . . , Xτ }] = P {Xm ∈ A|X0 = i}. Let us extend this idea to see what happens at successive entry times of a state. Suppose state i is such that the Markov chain Xn enters i infinitely often. For simplicity, assume X0 = i. The times 0 = τ0 < τ1 < τ2 < · · · at which the chain enters (or hits) i are defined recursively by τn = min{m > τn−1 : Xm = i},

n ≥ 1.

(1.19)

These are stopping times of Xn since9 {τn > } = {

 

1(Xm = i) < n}.

m=1

The next result is a special case of the more general regenerative property in Proposition 67 below. Proposition 28. (Inter-arrival Times at a State) Under the preceding assumptions, the times ξn = τn − τn−1 , n ≥ 1, between entrances to state i are i.i.d. Proof. We will show by induction that ξ1 , . . . , ξn are i.i.d. for n ≥ 1. The statement is obviously true for n = 1. Next, assume it is true for some n. To prove ξ1 , . . . , ξn+1 are i.i.d., it suffices by Exercise 20 to show that, for any m and n, 9 We frequently use the indicator function 1(·) which is 1 or 0 according as the “statement” (·) is true or false.

1.6 Classification of States

19

P {ξn+1 = m|ξ1 , . . . , ξn } = P {ξ1 = m|X0 = i}. But this follows by the strong Markov property at τn where Xτn = i, and the fact that (ξ1 , . . . , ξn ) is a function of X1 , . . . , Xτn . Example 29. Busy Period. Consider the M/M/1 queueing chain Xn in Example 21, where p is the probability of an arrival at any discrete time, and q is the probability of a service completion in any discrete time. Suppose p < q, which ensures that the system empties out infinitely often (see Exercise 50). Let 0 ≤ τ1 < τ2 < · · · denote the times at which the queue becomes empty (i.e., Xn hits 0). Then by Proposition 28, the durations ξn = τn −τn−1 , n ≥ 2, between the successive empty times are i.i.d. Let us see what else we can glean from this property. We can write ξn = γn + βn , where γn is the time until the next arrival after time τn−1 when the system becomes empty, and βn is the length of the busy period starting at time τn−1 + γn . By the strong Markov property at τn−1 , where Xτn−1 = 0, P {γn > m} = E[P {γn > m|X0 , . . . , Xτn−1 }] = P {γ1 > m|X0 = 0} = (1 − p)m . This is simply the geometric probability of a typical arrival time. Furthermore, it follows by the the more general regenerative property in Proposition 67 below that the durations of the busy periods βn are i.i.d. for n ≥ 2. Although there are no known formulas for the distributions of these times, Exercise 51 shows that E[β1 |X(0) = 0] = q(1 − p)/(q − p) − 1/p.

1.6 Classification of States Depending on its transition probabilities, a Markov chain may visit some states infinitely often and visit other states only a finite number of times over the infinite time horizon. Also, if a state is visited infinitely often, the mean time between visits may be infinite or finite. These properties are the basis of a classification of states of a Markov chain, which we now present. Throughout this section, Xn will denote a Markov chain on S with transition probabilities pij . The form of the distribution of X0 is not important for many results, and so we will often use conditional probabilities and expectations given X0 = i, and express them as Pi (A) = P {A|X0 = i},

Ei [Z] = E[Z|X0 = i}.

For instance, Pi {Xn = j} = pnij . We begin by studying the hitting times τj = min{n ≥ 1 : Xn = j},

j ∈ S.

20

1 Markov Chains

Consider the probability fijn = Pi {τj = n},

n ≥ 1,

that the chain starting at i enters j for the “first time” at the nth step. These probabilities are expressible in terms of the pij by the following recursive equations. The proof is a classic use of a “first-step analysis” that involves conditioning on X1 and using the Markov property. Proposition 30. For i, j ∈ S, fij1 = pij and fijn =



n−1 pik fkj ,

n ≥ 2.

(1.20)

k=j

Proof. This expression follows, since conditioning on X1 and using the Markov property,   n−1 Pi {τj = n|X1 = k}Pi {X1 = k} = pik fkj . fijn = k=j

k=j

Another important quantity for the chain is the probability that beginning at i it ever hits j, which is fij = Pi {τj < ∞} =

∞ 

fijn .

n=1

Note that summing (1.20) over all n yields the linear equations  pik fkj , i, j ∈ S. fij = pij +

(1.21)

k=j

Further properties of the passage or hitting probabilities fij are in Section 1.7. We are now ready to start classifying states of the Markov chain Xn . Definition 31. A state i is recurrent if fii = 1 (the chain returns to i with probability one), and i is transient if it is not recurrent. A recurrent state i is positive recurrent if Ei [τi ] < ∞, and it is null recurrent if Ei [τi ] = ∞. The recurrent or transient nature of a state j depends on the number of visits the Markov chain Xn makes to that state, which we denote by Nj =

∞ 

1(Xn = j).

n=0

These quantities, which may be infinite, are related to the successive times 0 < τ1 (j) < τ2 (j) < . . . at which the chain enters j (recall (1.19)), where τ1 (j) = τj . Namely,

1.6 Classification of States

21

Pi {Nj ≥ n} = Pi {τn (j) < ∞},

n ≥ 1.

(1.22)

By the definition of Nj and Ei [1(Xn = j)] = pnij , we have Ei [Nj ] =

∞ 

pnij .

(1.23)

n=0

The distribution of Nj in terms of the fij is as follows. Proposition 32. For i, j ∈ S, Pi {Nj > n} = fij (fjj )n ,

n ≥ 0,

(1.24)

and Pi {Nj = 0} = 1 − fij . Hence Pi {Nj = ∞} =

0 if fjj < 1 fij if fjj = 1.

Furthermore, Ei [Nj ] = 0 if fij = 0; and otherwise, fij /(1 − fjj ) if fjj < 1 Ei [Nj ] = ∞ if fjj = 1.

(1.25)

In particular, if i is transient, then Pi {Ni = n} = (1 − fii )fiin−1 , which is a geometric distribution with mean 1/(1 − fii ). Proof. Proceeding by induction, (1.24) is true for n = 0 by the definition of fij . Next, assume (1.24) is true for some n − 1. Using {Nj > n} ⊆ {Nj ≥ n} and (1.22), Pi {Nj > n} = Pi {Nj ≥ n, Nj > n} = Pi {Nj ≥ n}Pi {Nj > n|τn (j) < ∞}.

(1.26)

From the strong Markov property at τn (j) and Xτn (j) = j, it follows that Pi {Nj > n|τn (j) < ∞} = fjj . Applying this and the induction hypothesis to the last line in (1.26) yields (1.24) for n, which completes the induction. Next, note that from (1.24), we have Pi {Nj = ∞} = lim Pi {Nj > n} = fij 1(fjj = 1). n→∞

Finally (1.25) follows from (1.24) and the formula in Exercise 5 for means. Here are characterizations for a state to be recurrent or transient.

22

1 Markov Chains

Corollary 33. State i is recurrent ⇐⇒ Pi {Ni = ∞} = 1 ⇐⇒ Ei [Ni ] =

∞ 

pnii = ∞.

n=0

Equivalently, State i is transient ⇐⇒ Pi {Ni < ∞} = 1 ⇐⇒ Ei [Ni ] =

∞ 

pnii < ∞.

n=0

Proof. In the first assertion, the forward implications follow by Proposition 32 and (1.23). Also, Ei [Ni ] = ∞ implies i is recurrent by (1.25). Corollary 34. If j is transient, then limn→∞ pnij = 0, for each i. Proof. From the definition of Nj , Proposition 32 and Corollary 33, ∞ 

pnij = Ei [Nj ] = fij Ej [Nj ] < ∞.

n=0

Since this sum is finite, pnij → 0 for each i. To continue classifying the evolution of the Markov chain Xn on S, we will use the following terminology. State j is accessible from state i, denoted by i → j, if pnij > 0 for some n ≥ 1 (i.e., fij > 0). States i and j communicate with each other, denoted by i ↔ j, if i → j and j → i. This communication relation ↔ is an equivalence relation; see Exercise 24. Consequently, there exists a partition of S into disjoint equivalence classes, which we call communication classes. A set of states C in S is said to be closed if no state outside of C is / C). If C = {i} (a accessible from any state in C (pij = 0 for any i ∈ C, j ∈ singleton set) is closed, then i is an absorbing state (i.e., pii = 1). A closed set may contain several communication classes. Note that if C1 and C2 are closed, then so is C1 ∩ C2 , which is generally not empty unless the Ci are non-identical communication classes. The C is an irreducible set if i ↔ j for any i, j ∈ C. The communication classes are therefore irreducible. In addition, we say that the set C is recurrent if all of its states are recurrent. Similarly, C is transient (or positive recurrent or null recurrent, etc.) if all its states are of that type. A communication class need not be closed if it is transient, but the class is closed when it is recurrent. Proposition 35. A recurrent communication class is closed. Proof. Suppose C is a communication class that is not closed. Then there exist i ∈ C and j ∈ C such that pij > 0; and j → i. Since a return to i is not possible if j is entered, 1 − fii ≥ pij > 0. But this contradicts fii = 1, which holds because i is recurrent. Thus C is closed.

1.6 Classification of States

23

Another concept for classifying Markov chains is a subtle property concerning the times between visits to a state. In Example 18, the times between visits to each state are multiples of 4 (they are periodic with period 4). In general, the period di of a state i is the greatest common divisor of all n that satisfy pnii > 0. In other words, di is the largest integer such that pnii > 0 if and only if n is a multiple of di . State i is aperiodic if di = 1, and otherwise it is periodic. For instance, in Examples 5 and 16, each state is aperiodic; and the random walk in Example 3 is periodic with period 3 if each step size is a multiple of 3. Exercise 25 shows that within a communication class, each state has the same period. Therefore, if pii > 0 for any i in a communication class, then all the states in the class are aperiodic. Example 36. Consider a Markov chain on S = {1, 2, . . . , 7} with transition probabilities ⎤ ⎡ .5 .4 .1 0 0 0 0 ⎢ 0 .8 0 0 0 .1 .1⎥ ⎥ ⎢ ⎢0 0 0 0 0 0 1⎥ ⎥ ⎢ ⎥ P=⎢ ⎢.5 0 0 0 .5 0 0 ⎥ ⎢.3 0 0 0 .7 0 0 ⎥ ⎥ ⎢ ⎣ 0 .3 .7 0 0 0 0 ⎦ 0 0 0 0 0 .8 .2 From the transition graph in Figure 1.2, one can see that C = {2, 3, 6, 7} and S are the only two closed sets. Also, C is a communication class and T = {1, 4, 5} are communication classes. From results below, it follows that C is a class of positive recurrent states and T is a class of transient states. Furthermore, the states in these classes are aperiodic. .5

.8 .4

1

.2 .1

2

7 .1

.5

.3

4

.1

5 .5

.3

.1

3 .7

.8

6 .7

Fig. 1.2 Transition Graph for Example 36

We now establish the major result that all of the states in an irreducible set are of the same type, and they have the same period. Theorem 37. Suppose a set of states C is irreducible. Then C is either recurrent or transient. If C is recurrent, then it is either positive recurrent or null recurrent. In addition, each state in C has the same period.

24

1 Markov Chains

The proof will use the following characterization of null recurrence, which is proved in Section 1.20 by a coupling argument. Theorem 38. For an irreducible Markov chain on S with transition probabilities pij , a recurrent state i is null-recurrent if and only if lim pnii = 0.

n→∞

In this case, limn→∞ pnji = 0, for j ∈ S. Proof of Theorem 37. To prove C is either recurrent or transient, it suffices to prove the following statements: (a) If some i ∈ C is recurrent and j ∈ C, then j is recurrent. (b) If some i ∈ C is transient and j ∈ C, then j is transient. Clearly (b) follows from (a). Indeed, if i ∈ C is transient, then no state in C can be recurrent by (a), and hence all states in C must be transient. To prove (a), suppose i ∈ C is recurrent, and choose j ∈ C. Since i ↔ j, n m+n+ there exist m and n such that a = pm = Pm P Pn , ji pij > 0. Using P  n  pm+n+ ≥ pm ji pii pij = apii . jj

(1.27)

∞ ∞ Therefore, =0 pjj ≥ a =0 pii . By Corollary 33, the last sum is infinite since i is recurrent, and hence the first sum is infinite, proving that j is recurrent. Next, consider the case in which C is recurrent. To prove C is positive recurrent or null recurrent, it suffices to prove the following statements. (c) If some i ∈ C is positive recurrent and j ∈ C, then j is positive recurrent. (d) If some i ∈ C is null recurrent and j ∈ C, then j is null recurrent. Arguing as above, it follows that (c) implies (d). It remains to prove (c). We will use the negation of the assertion in Theorem 38, which is that i is positive recurrent

⇐⇒

lim supn→∞ pnii > 0.

(1.28)

Now, suppose i ∈ C is positive recurrent and j ∈ C. From (1.27), which is also valid here, and (1.28), we have lim sup pnjj ≥ a lim sup pnii > 0. n→∞

n→∞

Thus, the limit superior term for pnjj is positive, and so j is positive recurrent by (1.28). The proof that each state in C has the same period is Exercise 25. An irreducible Markov chain on a finite state space is automatically positive recurrent by the following result. Corollary 39. If a closed communication class is finite, then it is positive recurrent.

1.6 Classification of States

25

Proof. By Theorem 37, a closed communication class C is either transient or positive recurrent or null recurrent. Therefore, it suffices to show that C cannot be transient or null recurrent. Suppose i ∈ C is transient. Then each j ∈ C is transient by statement (b) in the proof of Theorem 37, and so pnij → 0 by Corollary 34. Therefore, since C is finite, we would have the contradiction  1 = Pi {Xn ∈ C} = pnij → 0. j∈C

Thus C is not transient. A similar argument shows that C is not null recurrent; here one uses pnij → 0 from Theorem 38. We end this section by describing a canonical decomposition of the Markov chain Xn on S. The starting point is the fact that one can partition S into disjoint communication classes; the classes are irreducible, but not necessarily closed. Let C1 , C2 , . . . denote the finite or infinite sequence of communication classes that are recurrent, and hence closed by Proposition 35. Then set T = S\ ∪k Ck . Clearly, T is transient because it consists of communication classes that are not recurrent. Of course, T is not necessarily closed, and it may be empty, or it may be equal to S. The transition matrix of Xn , with its rows arranged so the sets of states C1 , C2 , . . . , T appear in that order, has the form ⎡ ⎤ P1 0 0 · · · 0 ⎢ 0 P2 0 · · · 0 ⎥ ⎢ ⎥ ⎢ 0 0 P3 · · · 0 ⎥ ⎢ ⎥ (1.29) P=⎢ ⎥ ⎢...............⎥ ⎣...............⎦ Q1 Q2 Q2 · · · Q where Pk = (pij : i, j ∈ Ck ),

Qk = (pij : i ∈ T, j ∈ Ck ),

Q = (pij : i, j ∈ T ).

In summary, these observations establish that the Markov chain Xn on S has the following structure. Theorem 40. (Decomposition Property) The state space has the unique representation S = T ∪ C1 ∪ C2 ∪ . . ., where T is the set of transient states, and C1 , C2 , . . . are closed, irreducible recurrent sets. The transition matrix has the form (1.29). This decomposition tells us that if the Markov chain starts in a recurrent set Ck , then it moves within that set forever under the transition probabilities in Pk . On the other hand, if the chain starts in the transient set T , then it moves within T and either enters one of the recurrent sets and remains in that set thereafter, or it may remain in T forever, provided T is infinite (a finite T cannot be closed by Corollary 39).

26

1 Markov Chains

Example 41. Consider a Markov chain on S = {1, 2, . . . , 9} whose transition matrix has the following form, where means a positive probability. ⎤ ⎡ 000

000 ⎢ 0 0 0 0 0⎥ ⎥ ⎢ ⎢0 0 0 0 0 0 0 0⎥ ⎥ ⎢ ⎢ 0 0 0 0 0 0⎥ ⎥ ⎢ ⎥ P=⎢ ⎢0 0 0 0 0 0 0⎥ ⎢0 0 0 0 0 0 0 0⎥ ⎥ ⎢ ⎢0 0 0 0 0 0 0 ⎥ ⎥ ⎢ ⎣0 0 0 0 0 0 0⎦ 00000 00

Tracing out the transition graph of this chain, one can see that its closed irreducible sets are C1 = {3}, C2 = {5, 8} and C3 = {6, 7, 9}; and its set of transient states is T = {1, 2, 4}. Now, based on these sets and Theorem 40, reordering the states in the order 3, 5, 8, 6, 7, 9, 1, 2, 4 yields the more informative transition matrix: ⎡ ⎤

00000000 ⎢0

000000⎥ ⎢ ⎥ ⎢0

000000⎥ ⎢ ⎥ ⎢000

0000⎥ ⎢ ⎥ ⎥ P=⎢ ⎢000 0 000⎥ ⎢000 0 000⎥ ⎢ ⎥ ⎢0 0 0000 ⎥ ⎢ ⎥ ⎣ 00 00

0⎦

000000

We end the classification of Markov chains with a few more terms. The Markov chain Xn is irreducible if its state space S is irreducible. In that case, by Theorems 37 and 40, all states of the chain are either positive recurrent, null recurrent or transient, and all states have the same period. The Markov chain is called ergodic if it is irreducible, and its states are positive recurrent and aperiodic. The limiting behavior of ergodic Markov chains is the main topic for the rest of this chapter. Before getting into this, further insight on the movement of a Markov chain between subsets of its state space are given in the next section followed by a section on branching processes.

1.7 Hitting and Absorbtion Probabilities Basic performance measures for a Markov chain are as follows. • The probability that a chain will ever hit a specified set of states (e.g., a desired stock price).

1.7 Hitting and Absorbtion Probabilities

27

• The probability that a chain, beginning in transient states, will be absorbed in a certain recurrent class of states. • The mean time for a chain to hit a set, or to be absorbed in recurrent states. This section characterizes these and related quantities. Throughout this section, Xn will be a Markov chain on S with transition probabilities pij . Let S0 and S1 be disjoint (nonempty) subsets of the state space S. Define τ = min{n ≥ 1 : X0 , X1 , . . . , Xn−1 ∈ S0 , Xn ∈ S1 }. This is infinite when no such n exists. The τ is the time at which the chain exits S0 for the first time and enters S1 , or the time at which the chain first hits S1 without passing through the set (S0 ∪ S1 )c (which may be empty). Clearly τ is a stopping time of the chain Xn . We will first consider the hitting or entrance probabilities γi = Pi {τ < ∞},

i ∈ S0 .

Clearly γi is the probability the chain starting at i ∈ S0 eventually enters or hits S1 without passing through (S0 ∪ S1 )c (or simply the probability the chain hits S1 from S0 ). To avoid degenerate cases, assume the chain does not stay in S0 forever: Pi {Xn ∈ S0 , n ≥ 0} = 0,

i ∈ S0 .

(1.30)

This general formulation of hitting-probabilities covers a variety of events, including the following. (a) A chain hits a single state or set B (S0 = B c , S1 = B). (b) A chain beginning in its set of transient states T is absorbed in a recurrent class C (S0 = T , S1 = C). (c) A chain hits a set B without passing through a set A (S0 = (A ∪ B)c , S1 = B). (d) A chain beginning in its set of transient states T is absorbed in a recurrent class C at a state k ∈ C (S0 = T , S1 = {k}). The next result characterizes the hitting probabilities γ = (γi : i ∈ S0 ) in terms of r = (ri : i ∈ S0 ) and Q = (qij : i, j ∈ S0 ), where  ri = pij , qij = pij , i, j ∈ S0 . j∈S1

Theorem 42. The probabilities γi of hitting S1 from S0 satisfy  pij γj , i ∈ S0 . γi = ri + j∈S0

(1.31)

28

1 Markov Chains

This in matrix notation is γ = r + Qγ. Moreover, γ = is the smallest solution to the equation y = r + Qy,

∞ n=0

y ≥ 0.

Qn r, and this

(1.32)

If S0 is finite, then γ = (I − Q)−1 r and this is the unique solution to (1.32). Proof. Assertion (1.31) follows since, by conditioning on X1 ,  γi = Pi {γ < ∞|X1 = j}pij j∈S

=



[1(j ∈ S1 ) + γj 1(j ∈ S0 )]pij .

j∈S

Next, by the definitions of rj and qij and taboo-probability reasoning as in Example 9, γi =

∞ 

Pi {τ = n}

n=1

=

∞  

Pi {X1 , . . . , Xn−2 ∈ S0 , Xn−1 = j}P {Xn ∈ S1 |Xn−1 = j}

n=1 j∈S0

=

∞  

n−1 qij rj .

n=1 j∈S0

 n Thus, in matrix notation γ = ∞ n=0 Q r. Now, suppose y is any nonnegative solution to y = r + Qy. Then by induction n  y= Qm r + Qn+1 y, n ≥ 0. m=0

Consequently, y ≥ lim

n→∞

n  m=0

Qm r =

∞ 

Qm r = γ.

m=0

Thus γ is the smallest solution to y = r + Qy. Next, suppose S0 is finite. To show γ is the unique solution to y = r + Qy, let y be any solution and set x = y − γ. Clearly x = Qx and by induction x = Qn x. Now Qn → 0 since by (1.30), n qij ≤ Pi {Xm ∈ S0 , m ≤ n − 1} → 0.

Therefore, x = Qn x → 0, which proves that γ is the unique solution to y = r + Qy. From γ = r + Qγ, it follows that γ = (I − Q)−1 r, provided the inverse (I − Q)−1 of I − Q exists. But this exists by a basic property of linear

1.7 Hitting and Absorbtion Probabilities

29

algebra, since the preceding paragraph showed that the only solution of the finite linear equations (I − Q)x = 0 is x = 0. We now consider expected hitting times and related random quantities. Fix a subset B of S and let τB = min{n ≥ 1 : Xn ∈ B}. For each i ∈ B c , assume Pi {τB < ∞} = 1. For f : S → R+ , define τ B −1

vi = Ei [

f (Xn )],

i ∈ Bc.

n=0

The f (j) would typically be a cost or utility for the chain visiting state j. Then vi would be the expected utility up to the time the chain enters B. An important example is the expected time vi = Ei [τB ] to hit B (here f (j) ≡ 1). As another example, vi is the expected number of visits Xn makes to a fixed state k ∈ B c before it enters B, when f (j) = 1(j = k). From the proof of Theorem 42, it is clear that the following result is true. Theorem 43. The assertions and proof of Theorem 42 hold with  γi = vi , and ri = pij f (j), i ∈ B c , j∈B

where it is assumed that each ri < ∞. Example 44. Gambler’s Ruin Model. Consider the random walk Xn on S = {0, 1, . . . , m} as in Example 4, where Xn is the fortune of a gambler at the nth play of a game. At each play, the gambler wins or looses one dollar with respective probabilities p and q = 1 − p. Clearly 0 and m are absorbing states and states 1, . . . , m − 1 are transient states. Let γi be the probability that the gambler goes broke (enters state 0) when starting with X0 = i dollars. Theorem 42 tells us that the probabilities γi are the unique solution to the difference equation γi = qγi−1 + pγi+1 ,

1 ≤ i ≤ m − 1,

with boundary conditions γ0 = 1 and γm = 0. We will solve this by a method analogous to that used for differential equations. Consider a solution of the form γi = ri , where r is a parameter to be determined. Substituting this in the preceding equation and dividing by ri−1 , we obtain pr2 − r + q = 0, which has roots r1 = 1, r2 = q/p. In case r1 = r2 (i.e. p = 1/2), the general solution is γi = a1 r1i + a2 r2i , where the coefficients are given by the boundary conditions 1 = γ0 = a1 + a2 , Therefore the solution is

0 = γm = a1 + a2 (p/q)m .

30

1 Markov Chains

γi =

(q/p)i − (q/p)m , 1 − (q/p)m

if p = 1/2.

Furthermore, letting q/p → 1 in this expression and using L’Hospital’s rule we obtain the solution γi = 1 − i/m, if p = q = 1/2. In addition, the probability that starting at i the gambler reaches m before being ruined is 1 − vi , since the fortune eventually is absorbed in 0 or m. The expected time the game lasts before the gambler’s fortune reaches 0 or m is the subject of Exercise 45.

1.8 Branching Processes This section describes a classical Markov chain model for describing the size of a population in which each member of the population independently produces offspring. The main issue is under what conditions does the population explode to infinity or become extinct in a finite time. The principal results are a characterization of the probability of extinction and a procedure for computing it. Consider a population of identical items that evolves in discrete time as follows. Each item in the population lives for a single time period and at the end of its one-period life it produces k items with probability pk , k ≥ 0, independently of the other items. This off-spring probability measure has a finite mean μ, and p0 ∈ (0, 1). Consequently, whenever the population size is i, the probability that the population dies out in the next time period is pi0 . A major quantity of interest is the probability that the population eventually becomes extinct. Let Xn denote the population size at time n. For simplicity, assume X0 = 1 (the case X0 > 1 is covered in Exercise 61). This process satisfies the recursive formula Xn+1 = 0 when Xn = 0, and otherwise, d

Xn+1 = ξn1 + ξn2 · · · + ξnXn ,

n ≥ 0,

(1.33)

where ξn1 , ξn2 , . . . are i.i.d. random variables with probability measure p that are independent of Xn . The interpretation is that the ith item present at time n produces ξni items at time n + 1. From this representation it follows by Proposition 11 that Xn is a Markov chain. It is clear that the chain is aperiodic, 0 is an absorbing state, and the other states are transient. Because of the representation (1.33), the analysis in this section is based on properties of sums of i.i.d. random variables, not involving knowledge of Markov chains. Some indication of the growth of Xn is given by its mean. Conditioning on Xn−1 and using (1.33), we have

1.8 Branching Processes

31

 E[Xn ] = E E[Xn |Xn−1 ] = μE[Xn−1 ]. Then iterating this backward and using X0 = 1 yields E[Xn ] = μn .

(1.34)

From this one can see how the mean population evolves as a function of the mean production μ of each item. In particular, E[Xn ] converges to 0, 1 or ∞ according as μ is < 1, = 1, or > 1. We will now discuss the possibility of the population becoming extinct in terms of zn = P {Xn = 0}, n ≥ 1, which is the probability of extinction before or at time n. Clearly zn is increasing, since Xn = 0 implies Xn+1 = 0. This and zn ≤ 1 ensure the existence of the limit z = lim zn , n→∞

which is the probability the population eventually becomes extinct. To characterize the extinction probabilities zn and z, we will use φ(s) =

∞ 

p k sk ,

0 ≤ s ≤ 1,

k=0

which is the generating function of the single-item production. Here are some preliminary insights. Proposition 45. The time-dependent extinction probabilities zn are given by z1 = p0 and zn = φ(zn−1 ), n ≥ 2. (1.35) In addition, the extinction probability z ∈ (p0 , 1] is such that z = φ(z); so z is a fixed point of φ. Proof. Since X0 = 1, we have, upon conditioning on X1 , zn =

∞ 

P {Xn = 0|X1 = k}pk ,

n ≥ 1.

k=1

When X1 = k, each of these k items generates a separate subpopulation, and so the population at time n is the union of k i.i.d. subpopulations that k have evolved for n − 1 time units. Therefore, P {Xn = 0|X1 = k} = zn−1 . Substituting this in the preceding display proves (1.35). Furthermore, since the generating function φ(s) is continuous and zn → z, then letting n → ∞ in (1.35) yields z = φ(z). The preceding result says the extinction probability z is a fixed point of φ and z is obtainable by the recursion (1.35). Here is a characterization of z.

32

1 Markov Chains

Theorem 46. If μ ≤ 1, then z = 1. If μ > 1, then z is the unique s in (0, 1) that satisfies s = φ(s). Proof. If μ < 1, then by (1.34), P {Xn ≥ 1} ≤ E[Xn ] = μn → 0. Therefore z = limn→∞ (1 − P {Xn ≥ 1}) = 1. If μ > 1, then Lemma 47 below establishes that φ has a unique fixed point in (0, 1), which is necessarily the extinction probability z by Proposition 45. Finally, if μ = 1, the only case of interest is p0 + p1 < 1, since p0 > 0. Then another application of Lemma 47 yields z = 1. Lemma 47. In the context of Theorem 46, if μ > 1 or if μ = 1 and p0 + p1 < 1, then the generating function φ(s) is strictly convex. Moreover, φ(s) has a unique fixed point z in (p0 , 1) if μ > 1, and z = 1 if μ = 1. Proof. Under the assumptions, φ is strictly convex since 

φ (s) =

∞ 

k(k − 1)pk sk−1 > 0,

s ∈ [0, 1].

k=2

Also, the graph φ (1) = μ ≥ 1. s = φ(s). That Figure 1.3 show

of φ(s) goes through the point (1, 1) and its slope there is Therefore, if μ > 1, there is exactly one s < 1 for which is, φ has a unique fixed point z in (p0 , 1). The graphs in the two possibilities

z < 1 if φ (1) = μ > 1, 1

and z = 1 if φ (1) = μ = 1. 1 φ(s)

p0

φ(s) p0 0

1

0

1

Fig. 1.3 Graph of φ(s)

For the interesting case in which μ > 1, the extinction probability z is the unique solution of the equation s = φ(s), for s ∈ (0, 1) (which is a polynomial equation when the quantity an item can produce is bounded). The solution can sometimes be obtained when φ(s) has a nice form such as in Exercise 62. As another example, consider the elementary case in which each item can produce at most 2 items. Then the extinction probability z is the solution of the quadratic equation s = p0 + p1 s + p2 s2 . Thus z = min{p0 /p2 , 1}. More

1.9 Stationary Distributions

33

generally, when each item can produce at most m items, then z is the solution of a polynomial equation of order m. Here is an easy way to find bounds on z. Remark 48. Bounds. In case μ > 1, suppose 0 < a ≤ b < 1 are such that φ(a) ≥ a and φ(b) ≤ b. Then a ≤ z ≤ b by Figure 1.3. For practical applications, one can compute the extinction probability by the recursion (1.35) as follows. Remark 49. Computation of Extinction Probability. When μ > 1, the following procedure yields a value z ∗ that is within ε of z. Let z0 = 0 and compute z1 , z2 , . . . by zn = φ(zn−1 ). Stop at z ∗ = min{zn : φ(zn + ε) ≤ zn + ε}. Then zn ↑ z ∗ and Remark 48 ensure that 0 ≤ z − z ∗ ≤ ε.

1.9 Stationary Distributions We now turn to the main theme of characterizing the long run or limiting behavior of ergodic Markov chains in terms of their stationary distributions. This section introduces the notion of a stationary distribution for a Markov chain and shows that a Markov chain is stationary if its initial distribution is a stationary distribution. The rest of the section is devoted to establishing the existence of stationary distributions for positive recurrent Markov chains. Included are general formulas for stationary distributions and examples showing how to evaluate such distributions. Definition 50. A probability measure π on S is a stationary distribution for the Markov chain Xn (or for P) if  πi = πj pji , i ∈ S. (1.36) j

This equation in matrix notation is π = πP, where π = (πi : i ∈ S) is a row vector. In general, any measure η with i ηi ≤ ∞ that satisfies η = ηP is an invariant measure for P. A Markov chain may have an infinite number of stationary distributions or invariant measures; see Exercise 30. We begin by relating stationary distributions to stationary processes. Definition 51. A stochastic process {Xn : n ≥ 0} on a general state space is stationary if, for any n ≥ 0,

34

1 Markov Chains d

(Xn , . . . , Xn+k ) = (X0 , . . . , Xk ),

k ≥ 1.

(1.37)

That is, the finite-dimensional distributions of Xn remain the same if the time is shifted by any amount n. A stationary process is sometimes said to be a process that is in equilibrium or in steady state. If a process Xn is stationary, then necessarily the distribution of each Xn d does not depend on n (i.e., Xn = X0 , n ≥ 1). This simpler condition is also sufficient for a Markov chain to be stationary as we now justify. Also, a Markov chain is stationary if and only it its initial state has a stationary distribution. Proposition 52. The following statements are equivalent for the Markov chain Xn . (a) Xn is stationary. d

(b) Xn = X0 , n ≥ 1. (c) The distribution of X0 is a stationary distribution. Proof. Because the finite-dimensional probabilities of the Markov chain are simply products of transition probabilities, the stationarity criterion (1.37) is equivalent to P {Xn = i0 }

k 

pim−1 ,im = P {X0 = i0 }

m=1

k 

pim−1 ,im ,

i0 , . . . , ik ∈ S.

m=1

In light of this, (a) is equivalent to (b). Next, note that (b) is equivalent to  P {X0 = i} = P {X0 = j}pnji ,

n ≥ 1.

j

Then (b) implies (c), since the last equality for n = 1 is statement (c). Conversely, if (c) holds, then one can show by induction that the preceding equality holds for each n ≥ 1, and hence (c) implies (b). We are now ready to present the major result that any positive recurrent Markov chain has a stationary distribution. This is based on the following preliminary result that an irreducible, recurrent Markov chain has a unique invariant measure up to a multiple by a constant. Consider the measure η on S defined as follows. For any fixed i ∈ S, let τi = min{n ≥ 1 : Xn = i}, and define ηi = 1 and τ i −1

ηj = Ei [

1(Xn = j)],

j ∈ S\{i}.

(1.38)

n=0

The ηj (a function of i) is the expected number of visits Xn makes to state j in between visits to i. The measure η is positive if each ηj is positive. Another

1.9 Stationary Distributions

35

way of writing the preceding expression is ηj =

∞ 

Pi {Xn = j, τi > n},

j ∈ S\{i}.

(1.39)

n=0

This follows by taking expectations of the identity10 τ i −1

1(Xn = j) =

n=0

∞ 

1(Xn = j, τi > n).

n=0

The proof of the following result is at the end of this section. Theorem 53. (Invariant Measures) If the Markov chain Xn is irreducible and recurrent, then η defined by (1.38) is a positive invariant measure for the chain. This invariant measure is unique up to multiplication by a constant. In some instances, the invariant measure (1.38) for an irreducible recurrent Markov chain is finite and hence can be normalized to be a stationary distribution. This is true, of course, when the state space is finite. It is also true for infinite state spaces when the chain has the added condition of being positive recurrent. Here is a more general result that an irreducible chain has a positive stationary distribution if and only if it is positive recurrent. Theorem 54. (Stationary Distributions) An irreducible Markov chain Xn has a positive stationary distribution if and only if all of its states are positive recurrent. In that case, the stationary distribution is unique and has the following form: For any fixed i ∈ S,

  τi −1 Ei 1(X = j) n n=0 πj = , j ∈ S, (1.40) μi where τi = min{n ≥ 1 : Xn = i} and μi = Ei [τi ]. Another expression for this distribution is j ∈ S. (1.41) πj = 1/μj , Proof. Suppose the irreducible chain Xn has a positive stationary distribution π. We first show that the chain is not transient. Suppose to the contrary that it is transient. Since π = πP, we have π = πPn , for n ≥ 1. Also, by Corollary 34, pnij → 0 as n → ∞ for each j. Therefore, by the dominated convergence theorem for sums,11  πj = πi pnij → 0. i

But this contradicts πj > 0. Thus the chain is not transient. 10 11

Identities like

 τ −1

n=0

Yn =

∞

n=0

Yn 1(τ > n) are convenient for computing expectations.

As an example, for a probability measure   p on S and bounded fn : S → R such that fn (i) → f (i), it follows that i fn (i)pi → i f (i)pi .

36

1 Markov Chains

Now, since the chain is irreducible but not transient, it must be recurrent by Theorem 37. To prove the chain is positive recurrent, it suffices to show that Ei [τi ] is finite for some i. Let η be defined as in (1.38); it is the unique positive invariant measure for the chain by Theorem 53. Because the chain is assumed to have a stationary distribution π,this distribution  is a multiple of η, and hence it must have the form πj = ηj / k ηk , where k ηk is necessarily finite. Now by Exercise 5, an interchange of sums, and expression (1.39) for ηj , we have Ei [τi ] =

∞ 

Pi {τi > n} =

n=0

=



∞  

Pi {Xn = k, τi > n}

n=0 k

ηk < ∞.

(1.42)

k

Thus i is positive recurrent, and hence the chain is positive recurrent. Nowconsider the converse and assume the chain is positive recurrent. Then k ηk = μi is finite as we saw in (1.42). Hence πj = ηj /μi , which is (1.40), is the unique positive stationary distribution for the chain. Finally, note that ηi = 1 and (1.38) imply πi = 1/μi , and this is true for any fixed i. Thus, it follows that (1.41) is an alternative expression for the distribution in (1.40). The preceding theorem yields the following criterion for ergodicity that is very useful for applications. Corollary 55. An irreducible aperiodic Markov chain is ergodic if and only if it has a stationary distribution. In this case, the stationary distribution is positive and has the form shown in Theorem 54. Proof. If a Markov chain is ergodic, it has a positive stationary distribution by Theorem 54. Conversely, if an irreducible aperiodic Markov chain has a stationary distribution, then this stationary distribution is positive by Theorem 53, and hence is ergodic by Theorem 54. Although (1.40) and (1.41) are closed-form expressions for this distribution, they are typically not used to obtain numerical values for π. They come in handy, however, for the analysis or modeling of ergodic Markov chains (e.g., Proposition 69 below). For an irreducible, aperiodic Markov chain, a common approach for determining whether or not the chain is ergodic is as follows. First, find (if possible) a positive invariant measure η that satisfies the equations η = ηP . This is typically done algebraically, or by a computer package or by verifying by substitution that a candidate distribution satisfies the equations. Then find conditions under which η is finite or infinite. When η is finite, it can be normalized to be the stationary distribution of the chain, and hence the

1.9 Stationary Distributions

37

chain is ergodic. On the other hand, when η is infinite, the chain is either null-recurrent or transient. Here are two illustrations. Example 56. Consider the flexible manufacturing system in Example 5 in which Xn is the state of a machine, which can be idle (state 0) or producing a type i part (i = 1 or 2 or 3). Suppose Xn is a Markov chain with transition matrix ⎡ ⎤ .1 .2 .2 .5 ⎢.3 .4 0 .3⎥ ⎥ P=⎢ ⎣.4 0 .4 .2⎦ .3 0 .2 .5 This chain is clearly ergodic since it is irreducible, finite and aperiodic. The equations π = πP are π0 = .1π0 + .3π1 + .4π2 + .3π3 π1 = .2π0 + .4π1 π2 = .2π0 + .4π2 + .2π3 π3 = .5π0 + .3π1 + .2π2 + .5π3

That is, −9π0 + 3π1 + 4π2 + 3π3 = 0 =0 2π0 − 6π1 − 6π2 + 2π3 = 0 2π0 5π0 + 3π1 + 2π2 − 5π3 = 0

and solve the last set of equations for the other variables, and We will fix π0  then use 1 = i πi to find π0 . From the second and third equations in the last display, π1 = π0 /3, and π2 = π0 /3 + π3 /3. Substituting these in the first equation yields π3 = 20π0 /13, and  then π2 = 11π0 /13. The fourth equation is not needed. Finally, using 1 = i πi , we obtain the stationary distribution π=

 39 13 33 60  , , , . 145 145 145 145

This distribution tells us, for instance, that the machine works on a type 3 part about 41% of the time, and it is idle more than 25% of the time. Example 57. Machine Deterioration: Stationary Distribution. Consider Example 16 in which the Markov chain Xn denotes the state of deterioration of a machine over time with transition matrix

38

1 Markov Chains

⎡ p00 p01 p02 ⎢ 0 p11 p12 ⎢ P=⎢ ⎢ 0 0 p22 ⎣. . . . . . . . p0 . . . . .

⎤

. . . .. . . p13 . . . . .⎥ ⎥ p23 p24 . . ⎥ ⎥ . . . . . . . .⎦ 0 0 p

Assume the pij are positive. Then the chain is ergodic since it is irreducible and aperiodic and finite. The system of equations π = πP is π0 = π0 p00 + π p0 i  πi = πj pji , 1 ≤ i ≤ . j=0

Because of the structure of these equations, they have a tractable algebraic solution, which one can see by considering them one at a time in the order i = 0, 1, . . . Namely, the solution to the last  equations, for a fixed π0 , is πi = π0 ci , where the ci are given recursively by c0 = 1 and ci =

i−1 1  cj pji , 1 − pii j=1

1 ≤ i ≤ .

  Then from 1 = i=0 πi , it follows that π0 = ( i=0 ci )−1 . Thus, the stationary distribution of the chain is ci πi = 

j=0 cj

,

0 ≤ i ≤ .

The remainder of this section is devoted to the proof of Theorem 53, which is restated here. Theorem 58. (Invariant Measures) If the Markov chain Xn is irreducible and recurrent, then η defined by (1.38) is a positive invariant measure for the chain. This invariant measure is unique up to multiplication by a constant. Proof. The first task is to verify η = ηP. From (1.39), ηj =

∞ 

Pi {Xn = j, τi > n},

n=0

where i is a fixed reference state. For a more convenient expression, let Q = n (qk ), where qk = pk 1( = i), and denote its nth product by Qn = (qk ), where qk = 1(k = ). Expressing τi in terms of the Xm , and using products of probabilities as in Section 1.2, we have for n ≥ 1 and j = i, n Pi {Xn = j, τi > n} = Pi {Xm = i, 1 ≤ m ≤ n − 1, Xn = j} = qij .

1.9 Stationary Distributions

39

This is the taboo probability of moving from i to j in n steps while avoiding i. Thus, in matrix notation, the vector η (including ηi = 1), has the form η=

∞ 

ei Qn ,

(1.43)

n=0

where ei is the m-dimensional row vector with 1 in position i and 0 elsewhere. Using a little algebra on this sum and noting that Q is P with zeros in column i, we have ∞  η = ei + ei Qn−1 Q = ei + ηQ = ηP. n=1

Thus, η is an invariant measure. To prove each ηj is finite, suppose to the contrary that ηj = ∞ for some j = i. Since i ↔ j, there is an m such that pm ji > 0. Also, η = ηP implies η = ηPm by induction. Then we obtain the contradiction  m ηk pm 1 = ηi = ki ≥ ηj pji = ∞. k

Therefore, each ηj is finite. Furthermore, each ηj is positive because there is an  such that pij > 0, and so using ηi = 1, ηj =



ηk pkj ≥ ηi pij = pij > 0.

k

To prove η is unique up to multiplication by a constant, let γ be another positive invariant measure. Assume γi = 1, where i is the fixed reference state in the definition of η. There is no loss in generality in this assumption since any such measure can be normalized by dividing by γi . Let ζ = γ − η. The proof will be complete upon showing that ζ = 0. We first show that ζ ≥ 0. Note that γ = γP can be written as γ = ei + γQ.

(1.44)

Then by induction, γ=

n 

ei Qm + γQn+1 ,

n ≥ 0.

m=0

Since the last term is nonnegative, γ≥

n  m=0

Thus ζ = γ − η ≥ 0.

ei Qm →

∞  m=0

ei Qm = η.

40

1 Markov Chains

Next, subtracting η = ηP from γ = γP yields ζ = ζP. This implies ζ = ζPn , n ≥ 1. Using this and ζi = 1 − 1 = 0, we have  ζj pnji , n ≥ 1. (1.45) 0 = ζi = j=i n

By the irreducibility of the chain, for each j, there is an nj such that pjij > 0. Then ζ ≥ 0 and (1.45) imply that ζj = 0 for each j.

1.10 Limiting Distributions We have seen that a positive recurrent Markov chain has a unique stationary distribution π that satisfies the balance equations π = πP. This section shows that this stationary distribution is also the limiting distribution when the Markov chain is ergodic. Suppose Xn is a Markov chain on S with transition probabilities pij . A probability measure π is the limiting distribution of the chain if lim P {Xn = i} = πi ,

n→∞

i ∈ S.

Note that π does not depend on the distribution of X0 . Exercise 31 shows that a limiting distribution for a Markov chain is always a stationary distribution — no additional assumptions on the chain are needed. On the other hand, there are non-ergodic chains with stationary distributions that are not limiting distributions. Here is the major result concerning limiting distributions for ergodic Markov chains. Theorem 59. If a Markov chain is ergodic, then its stationary distribution is its limiting distribution, which is positive. Proof. The assertion follows by the coupling described in Theorems 110 and 111 below. Theorem 54 ensures that π is positive. An alternative approach is to prove the assertion by applying the discrete-time version of the renewal theorem; see the proof of Theorem 51 in Chapter 2. The next result, which includes Corollary 55, says that the existence of a limiting distribution is another criterion for ergodicity. Corollary 60. For an irreducible, aperiodic Markov chain, the following statements are equivalent. (a) The chain is ergodic. (b) The chain has a stationary distribution. (c) The chain has a limiting distribution. When these statements hold, the stationary and limiting distributions are the same, and it is positive.

1.10 Limiting Distributions

41

Proof. The equivalence of (a) and (b) follows by Corollary 55. Next, (a) ⇒ (c) by Theorem 59, and (c) ⇒ (b) by Exercise 31. The equality and positiveness of the stationary and limiting distributions follow by Theorem 59 and Corollary 55. The rest of this section covers properties of limiting distributions. We will often use the notion of convergence in distribution of random variables defined as follows (later, we use this convergence for more general random elements). In particular, if a Markov chain Xn has a limiting distribution π, then Xn converges in distribution to a random variable X with distribution π. Definition 61. Random variables Xn with values in S converge in distribud tion to a random variable X, denoted by Xn → X, if lim P {Xn = i} = P {X = i},

n→∞

i ∈ S.

d

The Appendix points out that Xn → X is equivalent to E[f (Xn )] → E[f (X)], for any bounded f : S → R.

(1.46)

This characterization of convergence in distribution yields the following result for Markov chains. Remark 62. Limits of Expectations. The Markov chain Xn has a limiting distribution π if and only if, for any bounded function f : S → R,  f (i)πi . lim E[f (Xn )] = n→∞

i

The last limit statement also justifies the limiting average n−1

n  m=1

E[f (Xm )] →



f (i)πi .

i

n This follows by the property that n−1 m=1 am → a if an → a. More general averages are in Remark 78 and Exercises 34 and 36. A useful property of ergodic Markov chains is that many functions of it also have limiting distributions. This is because of the following elementary but important result, which says that the distant future of an ergodic Markov chain behaves as if the chain were stationary. Proposition 63. (Asymptotic Stationarity). An ergodic Markov chain Xn is asymptotically stationary in the sense that d ¯0, X ¯ 1 , . . .), (Xn , Xn+1 , . . .) → (X

as n → ∞,

(1.47)

where X¯n is a stationary version of Xn (the chain X¯n is stationary and it has the same transition probabilities as Xn ).

42

1 Markov Chains

¯ 0 = i} and Xn and X¯n have the same Proof. Since P {Xn = i} → P {X transition probabilities, it follows that, for B ⊂ S ∞ ,  ¯ 1, X ¯ 2 , . . .) ∈ B} P {Xn = i}Pi {(i, X P {(Xn , Xn+1 , . . .) ∈ B} = i

→



¯ 0 = i}Pi {(i, X ¯ 1, X ¯ 2 , . . .) ∈ B} P {X

i

¯0, X ¯ 1 , . . .) ∈ B}. = P {(X ¯ 1, X ¯ 2 , . . .) ∈ B}. The convergence is of the form (1.46) for f (i) = Pi {(i, X This proves (1.47). Remark 64. By properties of convergence in distribution, each of the following statements is equivalent to (1.47). For any bounded f : S ∞ → R, ¯0, X ¯ 1 , . . .)], E[f (Xn , Xn+1 , . . .)] → E[f (X d

¯0, X ¯ 1 , . . .). f (Xn , Xn+1 , . . .) → f (X Example 65. For an ergodic Markov chain Xn on a countable set S of real numbers, we know by Proposition 63 and Remark 64 that d ¯ ¯ Xn+1 − Xn → X 1 − X0 , d

¯0, X ¯ 1 }. max{Xn , Xn+1 } → max{X In other words, lim P {Xn+1 − Xn = k} =

n→∞

lim P {max{Xn , Xn+1 } ≤ k} =

n→∞



πi pi,i+k ,

i



πi pij .

i≤k j≤k

1.11 Regenerative Property and Cycle Costs The strong Markov property says, loosely speaking, that a Markov chain regenerates, or starts anew, at a stopping time. This section describes a related result that if a Markov chain enters a fixed state infinitely often, then it regenerates each time it enters the state, and its sample-path segments between entrances to the state are i.i.d. This fundamental “regenerative” property is the key to obtaining strong laws of large numbers for ergodic Markov chains that we present in the next section, and for obtaining related central limit theorems that we present in Chapter 2. We also present a formula for the

1.11 Regenerative Property and Cycle Costs

43

mean of certain costs or utilities in a regenerative cycle. Basics of more general regenerative processes are covered in Chapter 2. We begin by identifying a regenerative property that any ergodic Markov chain has. Suppose a Markov chain Xn on S has a state i that it enters infinitely often (this would be true for any state when the chain is irreducible and recurrent). For this fixed state i, let 0 = τ0 < τ1 < τ2 < · · · denote the times at which the chain hits i. Recall that the times τn − τn−1 between the entrances to i are i.i.d. by Proposition 28. This is part of the more general regenerative property described as follows. The nth segment (or cycle) of the Markov chain in the time interval [τn−1 , τn ) is defined by   ζn = τn − τn−1 , {Xm : m ∈ [τn−1 , τn )} , n ≥ 1. The segment ζn takes values in {(m, j) : m ≥ 1, j ∈ {i} × S m−1 }, and it contains all the information about the sample path of the chain in the time interval [τn−1 , τn ). Definition 66. The Markov chain Xn is regenerative over the times τn if the segments ζn , for n ≥ 1, are i.i.d. In particular, τn − τn−1 are i.i.d. More general regenerative processes are studied in the next chapter. Proposition 67. (Regenerative Property) If the Markov chain Xn starting at X0 = i enters state i infinitely often, then the chain is regenerative over the times τn at which the chain enters i. Proof. This follows by the proof of Proposition 28 with ζn in place of ξn . Example 68. Consider the (s, S) inventory model in Example 17, where Xn is the inventory level in period n. Assuming the demands are positive, it follows that this Markov chain Xn enters state S, where the inventory is replenished, infinitely often. Then this chain is regenerative over the replenishment times τ1 < τ2 < . . . . Consequently, the times between replenishment τn − τn−1 are i.i.d. Also, since the evolutions of the inventory in these cycles are i.i.d., there are many performance parameters that are i.i.d. such as: τn −1 • The amounts of time Yn = m=τ 1(Xm > i) in the cycles during n−1 which the inventory exceeds i. τn −1 1(Xm+1 < Xm ) in the cycles • The numbers of demands Nn = m=τ n−1 that are satisfied. The regenerative property plays an important role in determining average costs or performance values for a Markov chain. n A variety of costs or utility functions for a Markov chain are of the form m=0 Vm , where Vn are values associated with the chain at time n. Then next result is a formula for the mean of such a cost function in a regenerative cycle. The formula is due to the form of a stationary distribution expressed as (1.40). We use the formula in the next sections to describe limiting values in strong laws of large numbers.

44

1 Markov Chains

Proposition 69. (Cycle Costs) Let Xn be an irreducible positive recurrent Markov chain with stationary distribution π. Suppose Vn , n ≥ 1, are realvalued random variables associated with the chain such that Ei [Vn |X0 , . . . , Xn ] = aXn ,

n ≥ 0,

where aj are constants. Then, for the hitting time τi of a fixed state i, Ei

i −1

τ

  Vn = πi−1 aj πj ,

n=0

(1.48)

j

provided the last sum is absolutely convergent.12 Proof. Noting that {τi > n} is a function of X0 , . . . , Xn , and using the pullthrough formula for conditional probabilities, the left-hand side of (1.48) is Ei

∞



∞ 

    Vn 1(τi > n) = Ei Ei 1(τi > n)Vn X0 , . . . , Xn

n=0

= Ei

∞





n=0

1(τi > n)aXn =



n=0

=

πi−1



aj Ei

∞



 1(Xn = j, τi > n)

n=0

j

aj πj .

j

The last equality is due to Theorem 54 and (1.39), and the equality  before it uses Fubini’s theorem in the appendix and the assumption that j |aj |πj is finite. A typical example of (1.48), for f : S 2 → R, is Ei

i −1

τ

    f (Xn , Xn+1 ) = πi−1 f (j, k)pjk πj .

n=0

j

(1.49)

k

In this case, E[f (Xn , Xn+1 )|X0 , . . . , Xn ] =



f (Xn , k)pXn k .

k

Here is an illustration of the use of (1.49). Example 70. Let Xn denote the M/M/1 queueing chain as in Example 21. Assume that p < q, which says the probability of an arrival is less than the probability of a service completion in a time period. Then by Exercise 50, the stationary distribution is πi = (1 − ρ)ρi , i ≥ 0, where ρ = p(1 − q)/[q(1 − p)]. 12

A sum



j cj

is absolutely convergent if

 j

|cj | is finite.

1.12 Strong Laws of Large Numbers

45

The ρ is the traffic intensity. Let us consider the number of customers N that are served in a busy period (between visits to state 0). Since N is also the number of arrivals in a busy period, assuming X0 = 0, N=

τ 0 −1

1(Xn+1 = Xn + 1).

n=0

From p01 = p, pj,j+1 = p(1 − q), j ≥ 1, and (1.49), E0 [N ] = π0−1



πj pj,j+1 = p +

j

∞ 

ρj p(1 − q).

j=1

Thus, we have the result E0 [N ] = p(1 − qρ)/(1 − ρ).

1.12 Strong Laws of Large Numbers Many properties or performance measures of a stochastic process are expressed as limiting averages, e.g., the average amount of time the process spends in a state, or the average cost of running the process. Such an average is usually expressed as a strong law of large numbers (SLLN), where the limit is a function of the limiting distribution of the process. In this section, we present several strong laws of large numbers for ergodic Markov chains. We begin with a motivating example. Example 71. Multi-Type Job Processing. A system processes several types of jobs, where S denotes the set of job types. Suppose that a sequence of jobs it processes (labeled by job type) is an ergodic Markov chain Xn on S with stationary distribution π (here the label n refers to the order in a sequence and is not a time parameter in the usual sense). The revenue received from each type-i job is f (i), where f : S → R+ . Our interest is in the average revenue per job processed over the infinite time horizon. First, n note that the average revenue per job from the first n jobs is n−1 m=1 f (Xm ). Then by Theorem 74 below, the average revenue per job (over the infinite horizon) is the limiting average13 lim n

n→∞

−1

n  m=1

f (Xm ) =



f (i)πi

a.s.

i

In applications like this, rewards or utility values are sometimes “random” functions of the process. For instance, in this job-processing setting, suppose the processing times of a type-i job are i.i.d. random variables with mean 13

a.s. stands for almost surely, which means with probability one. For instance, Yn → Y a.s. means P {limn→∞ Yn = Y } = 1, and Y > Z a.s. means P {Y > Z} = 1.

46

1 Markov Chains

ai , independent of other jobs and the order in which they are processed. Let V1 , V2 , . . . denote the processing times of the respective jobs X1 , X2 , . . . Then by Theorem 75 below, the average processing time per job is n 

lim n−1

n→∞

Vm =



m=1

ai πi

a.s.

i

As we will see shortly, the average cost and processing time in the preceding example are examples of a SLLN. Such laws are applications of the following one, which is proved in standard texts on probability theory. Theorem 72. (Classical SLLN) If Y1 , Y2 , . . . are i.i.d. random variables with a mean that may be infinite, then lim n−1

n→∞

n 

Ym = E[Y1 ]

a.s.

m=1

We will now describe analogous results for Markov chains. More general SLLNs for regenerative processes are in Chapter 2. For this discussion, suppose Xn is an ergodic Markov chain on S with transition probabilities pij and stationary distribution π. We first consider limiting averages associated with hitting times of a state. Fix a state i, and let 0 < τ1 < τ1 < · · · denote the successive times at which the chain hits or enters state i. Define Ni (n) =

n 

1(Xm = i),

m=1

which is the number of times the chain hits i in the first n time periods. Proposition 73. The average time between visits to state i is lim n−1 τn = 1/πi

n→∞

a.s.

(1.50)

a.s.

(1.51)

The average number of visits to state i is lim n−1 Ni (n) = πi

n→∞

Proof. We can write n

  n−1 τn = n−1 τ1 + (1 − 1/n) (n − 1)−1 (τm − τm−1 ) . m=2

Exercise 33 justifies n−1 τ1 → 0 a.s. Also, by Proposition 28, the times τm − τm−1 between entrances to state i are i.i.d. for m ≥ 2, and their mean is μi = 1/πi by Theorem 54. Then the classical SLLN ensures that the bracketed

1.12 Strong Laws of Large Numbers

47

term in the preceding expression converges to 1/πi a.s. Combining these observations14 proves (1.50). In addition, (1.51) follows from (1.50) by Theorem 10 in Chapter 2. We are now ready to present two results that establish limiting averages like those in Example 71. The first result is a SLLN, also called an ergodic theorem, for ergodic Markov chains. Theorem 74. For the ergodic Markov chain Xn with stationary distribution π and any f : S → R, lim n−1

n→∞

n 

f (Xm ) =



m=1

f (i)πi

a.s.,

i

provided the sum is absolutely convergent. This SLLN justifies the approximation n  m=1

f (Xm ) ≈ n



f (i)πi ,

for large n.

i

n In addition, the distribution of the sum  m=1 f (Xm ) can be approximated by a normal distribution with mean n i f (i)πi as shown by the central limit theorem for Markov chains in Example 68 in Chapter 2. Theorem 74 is a special case of the next SLLN for “random” functions of ergodic Markov chains. Here we use the following conditional probability terminology. Random variables Y1 , Y2 , . . . are conditionally independent given the Markov chain X = {Xn } if, for any y1 , . . . , ym and m, P {Y1 ≤ y1 , . . . , Ym ≤ ym |X} =

m 

P {Yk ≤ yk |X}.

k=1

If in addition, P {Yk ≤ y|X} = P {Yk ≤ y|Xk }, independent of k, then we say Y1 , Y2 , . . . are conditionally independent given {Xn } with distributions P {Y1 ≤ y|X1 }. For instance, in Example 71 the processing times V1 , V2 , . . . of the respective jobs X1 , X2 , . . . are conditionally independent given {Xn } with distributions P {V1 ≤ v|X1 = i} that have means ai , i ∈ S. Theorem 75. Associated with the ergodic Markov chain Xn , suppose Vn , n ≥ 1 are random variables that are conditionally independent given {Xn } with distributions P {V1 ≤ v|X1 = i} that have means ai , i ∈ S. Then 14

One often establishes convergence by exploiting the following properties, which follow automatically from the analogous limit statements for real numbers. If Yn → Y a.s. and Zn → Z a.s., then Yn + Zn → Y + Z a.s. and Yn Zn → Y Z a.s.. Also, Yνn → Y a.s. for integer-valued νn → ∞ a.s.

48

1 Markov Chains

lim n−1

n→∞

n 

Vm =



m=1

ai πi

a.s.,

(1.52)

i

provided the sum is absolutely convergent. Proof. We will use the notation Ni (n) and τn in Proposition 73 for a fixed state i, and assume X0 = i. Consider the sequence   Zn = τn − τn−1 , {(Xm , Vm ) : m ∈ [τn−1 , τn )} , n ≥ 1. This Zn contains all the information about the process {(Xm , Vm ) : m ≥ 0} in the time interval [τn−1 , τn ), where τ0 = 0. It follows as in Proposition 67 that the Zn are i.i.d. Consequently, the values Yk =

∞ 

Vm 1(τk−1 < m ≤ τk ),

k ≥ 1,

m=1

accumulated in the respective intervals (τk−1 , τk ] are i.i.d., since they are deterministic functions of the Zk . Then by the classical SLLN and Proposition 69, n   n−1 Yk → E[Y1 ] = πi−1 aj πj a.s. (1.53) j

k=1

Next, we show that this limit leads to (1.52). First assume the Vm are nonnegative. Note that15 n

−1

Ni (n)



Yk ≤ n

−1

n 

Vm ≤ n

−1

m=1

k=1

Ni (n)+1



Yk .

(1.54)

k=1

Applying (1.51) and (1.53) to the left-hand side of (1.54), we have

Ni (n)     −1 Ni (n)/n Ni (n) Yk → πi E[Y1 ] = aj πj

a.s.

j

k=1

A similar argument showsthat the right-hand side of (1.54) has this same  a π limit. Consequently, n−1 nm=1 Vm also has the limit j j , since it is j sandwiched between these two terms. For general Vm , we can write Vm = Vm+ − Vm− , where Vm+ = max{0, Vm } and Vm− = − min{Vn , 0}. Then by what we just proved, 15

Here and below, we use the convention that

0

n=1 (·)

= 0.

1.12 Strong Laws of Large Numbers

n−1

n 

n 

Vm = n−1

m=1

49

Vm+ − n−1

m=1

→



a+ j πj

−



j

n 

Vm−

m=1

a− j πj =

j



aj πj

a.s.

j

Example 76. Functions Involving Auxiliary Variables. In the setting of Theorem 75, a typical random function or value associated with the Markov chain has the form Vn = g(Xn , Yn ), where Yn are auxiliary variables in S  and g : S × S  → R. For instance, Yn might represent the state of an environment or cost that affects the value at time n, and g(i, y) is the value whenever the chain is in state i and the auxiliary variable is in state y. Assume the Yn are i.i.d. and independent of the chain. Then according to Theorem 75, the average value is n−1

n 

g(Xm , Ym ) →

m=1



E[g(i, Y1 )]πi

a.s.,

i

provided the last sum is absolutely convergent. This section ends with a few observations about limiting averages. Intuition suggests that a limiting average for a stochastic process should be the mean value associated with a stationary version of the process. Although this is not generally true, it is for the Markov averages above. ¯ n is a stationary version of the Remark 77. Stationary Chains. Suppose X ergodic Markov chain Xn in the preceding results (both chains have the same transition probabilities, but X¯n is stationary). By Proposition 52, each X¯n has the distribution π, and so in the context of the preceding results,  ¯ n )] = f (i)πi , E[f (X i

E[Vn ] =



¯ n = j]πj = E[Vn |X

j



aj πj .

j

That is, the limiting averages in Theorems 74 and 75 are the means of singleperiod values when the chain is in equilibrium. Do limiting averages exist for “expected values” analogous to those in the preceding SLLN? Remark 78. Limit Laws for Expectations. The limiting averages in Proposition 73 and Theorems 74 and 75 are also true for expected values. For instance, n−1 E[τn ] → 1/πi and n−1

n  m=1

E[Vm ] →

 i

ai πi .

50

1 Markov Chains

Such results for expected values follow by a discrete-time version of the key renewal theorem in Chapter 2 for functions of regenerative processes. Although these limits for means hold for regenerative processes, they generally do not hold for any non-regenerative sequence that obeys a SLLN.

1.13 Examples of Limiting Averages This section contains several corollaries and illustrations of the strong laws of large numbers in the preceding section. For this discussion, suppose Xn is an ergodic Markov chain on S with transition probabilities pij and stationary distribution π. Costs or utilities associated with n “jumps” of the Markov chain Xn are often modeled by sums of the form m=1 f (Xm−1 , Xm ), where f (i, j) is the cost of a jump from i to j. Averages of such sums are described by the following extension of Theorem 74. ˜ n = (Xn , . . . , Xn+ ) is an Corollary 79. For a fixed integer , the process X +1 ergodic Markov chain on S with stationary distribution π(i) = πi0 pi0 ,i1 · · · pi−1 ,i . Hence, for f : S +1 → R, lim n−1

n→∞

n 



˜m) = f (X

f (i)π(i)

a.s.,

i∈S +1

m=1

provided the sum is absolutely convergent. Proof. The first assertion follows by Exercise 28, and the second assertion is an example of Theorem 74. The following are basic properties of ergodic Markov chains that are manifestations of limiting averages. Example 80. Movements Between Sets. The average number of transitions the chain makes from a set A to a set B per unit time is λ(A, B) = lim n−1 n→∞

n 

1(Xm−1 ∈ A, Xm ∈ B)

a.s.

m=1

This limit exists by Corollary 79 with  = 1 and f (i, j) = 1(i ∈ A, j ∈ B), and it is   λ(A, B) = πi pij . i∈A

j∈B

This quantity is often called the rate at which the chain moves from A to B. It is also the mean number of transitions the chain makes from A to B per

1.13 Examples of Limiting Averages

51

unit time when the chain is in equilibrium; recall Remark 77. In particular, the rate at which the chain moves from state i to state j is λ(i, j) = πi pij , and therefore  λ(A, B) = λ(i, j). i∈A j∈B

Example 81. Balance Equations. In terms of rates of movements, the total balance equations for the Markov chain have the following interpretation: For i ∈ S,  πi = πj pji is equivalent to λ(i, S) = λ(S, i). j

This says that the rate at which the chain moves out of i is equal to the rate at which it moves into state i. More generally, λ(A, Ac ) = λ(Ac , A),

(1.55)

which says that the rate at which the chain moves out of a set A is the rate at which it moves into A. To prove (1.55), first observe that summing the balance property πi = j πj pji on i ∈ A, we have λ(A, S) = λ(S, A). Then subtracting λ(A, A) from these terms yields (1.55). The next example is an extension of Proposition 73. Example 82. Entrance Rates into Sets. The number of entrances the Markov chain makes into a set A ⊂ S up to time n is defined by νA (n) =

n 

1(Xm−1 ∈ A, Xm ∈ A).

m=1

By Example 80, the average number of entrances per unit time into A is  λ(A) = lim n−1 νA (n) = πi pij . (1.56) n→∞

i∈Ac ,j∈A

This is the same as the rate λ(Ac , A) at which the chain enters A. A related quantity is the nth time the chain enters the set A, which is τA (n) = min{m : νA (m) = n}. The limiting averages of these times is lim n−1 τA (n) = 1/λ(A) a.s.

n→∞

This follows since νA (τA (n)) = n and τA (n) → ∞; and so by (1.56), n−1 τA (n) =

 ν (τ (n)) −1 A A → 1/λ(A) a.s. τA (n)

52

1 Markov Chains

Example 83. Average Sojourn Time in a Set. Let Wn (A) denote the amount of time the Markov chain spends in a set A on its nth sojourn in that set. Then the average sojourn or waiting time in A is W (A) = lim n−1 n→∞

n 

Wm (A) a.s.

m=1

We will prove that this limit exists and W (A) =

1  πi . λ(A)

(1.57)

i∈A

First note that by the definition of the waiting times and the τA (n) from the preceding example, n 

Wm (A) =

m=1



Ni (τA (n)),

i∈A

where Ni (n) is the cumulative sojourn time in state i up to time n. Then (1.57) follows, since by the preceding example and n−1 Ni (n) → πi from Proposition 73, we have n−1

n 

  Wm (A) = (τA (n)/n) τA (n)−1 Ni (τA (n))

m=1

i∈A

1  πi . → λ(A) i∈A

Example 84. Machine Deterioration: Average Time Machine is Good. Consider the ergodic Markov chain Xn in Example 57 that denotes the state of deterioration of a machine. Its stationary distribution is πi = 

ci

k=0 ck

,

0 ≤ i ≤ ,

 where c0 = 1 and ci = (1 − pii )−1 i−1 j=1 cj pji . Suppose the sets of “acceptable” states and “bad” states of the machine are A = {0, 1, 2} and B = { − 2,  − 1, }, respectively (assume  ≥ 5). Then the preceding examples yield the following properties. The rate at which the machine jumps from being acceptable to being bad is λ(A, B) =

 i∈A,j∈B

πi pij =

2  i=0



ci

 

k=0 ck j=−2

pij .

1.14 Optimal Design of Markovian Systems

53

The rate at which the machine enters an acceptable state is λ(A) =

  i=3



2 

ci

k=0 ck j=0

pij .

The average time the machine is in an acceptable state is W (A) =

1 + c1 + c 2 1  πk =  . 2 λ(A) i=3 ci j=0 pij k∈A

1.14 Optimal Design of Markovian Systems This section explains how one can use average cost formulas for Markov chains to determine optimal design parameters for systems. Consider a Markov chain on a space S whose transition probabilities pij (x) are a function of a variable x in a set. The x is a design parameter (a vector or any type of variable) that can be selected to minimize the cost of running the Markov chain. In particular, suppose, for each x that pij (x) determines an ergodic Markov chain and denote its stationary distribution by πi (x), i ∈ S. Assume there is a cost f (i, x) whenever the chain visits state i, and a cost g(i, j, x) whenever the chain jumps from i to j. Then by Theorem 74 above, the average cost under the design variable x is   πi (x)[f (i, x) + pij (x)g(i, j, x)], (1.58) φ(x) = i

j

provided the sum is absolutely convergent. The aim is to find a value of x that minimizes this cost. For instance, in the (s, S) inventory model in Example 17, one may want to find values of s and S that minimize the average cost. In some cases, it might be convenient to first determine the stationary distributions π(x) for each x and then minimize φ(x) by an appropriate algorithm. When there are only a small number of x-values, a total enumeration approach might be feasible. Another more general approach is to determine the stationary distributions simultaneously with minimizing the cost by the following non-linear mathematical program:   min πi (x)[f (i, x) + pij (x)g(i, j, x)] x

i

j

54

1 Markov Chains

subject to πi (x) = 



πj (x)pji (x),

i ∈ S,

j

πi (x) = 1,

πi (x) > 0, i ∈ S.

i

A procedure for solving this would depend on the structure of the problem. The preceding is a generic design problem for a Markovian system. It is a “static” optimization problem in that an optimal x is selected once (at time 0) and the system is run indefinitely with x set at this value. This is different from a “dynamic” optimization problem in which one can vary a parameter x as the system evolves to minimize the cost. For instance, one might vary the service rate in a queueing system as the queue length varies. Such a problem is a Markov decision problem (or a stochastic control, or dynamic programming problem). Example 85. Optimal Machine Replacement. Consider the machine deterioration model in Example 16 in which Xn denotes the state of deterioration of a machine (or equipment) at time n. The Xn is a Markov chain on S = {0, 1, . . . , } with transition matrix ⎤ ⎡ p00 p01 p02 . . . . ⎢ 0 p11 p12 p13 . . . ⎥ ⎥ ⎢ ⎥ P=⎢ ⎢ 0 0 0 p22 p23 . ⎥ ⎣. . . . . . . . .⎦ p0 . . . 0 0 p Now, suppose there is a cost h(i) for operating the machine in state i and a cost C for replacing the machine, where h(0) ≤ h(1) ≤ · · · ≤ h() < C. Because of these costs, it may be beneficial to replace the machine before it reaches state . Accordingly, let us consider the following control-limit policy: Replace the machine in the next time period putting its state to 0 when and only when the machine’s state equals or exceeds a state x (the control limit). Under this policy, the transition probabilities of the chain are if 0 ≤ i ≤ x − 1 pij pij (x) = 1(j = 0) if x ≤ i ≤ . This chain is clearly ergodic. By Example 57, its stationary distribution is ci (x) πi (x) =  , i=0 ci (x)

0 ≤ i ≤ ,

1.15 Closed Network Model

55

but under the control, the ci (x) are given recursively by c0 (x) = 1 and  i−1 (1 − pii )−1 j=0 cj (x)pji , 1 ≤ i ≤ x − 1, ci (x) = x−1 x ≤ i ≤ . j=0 cj (x)pji , Now by (1.58), the average cost of the system is φ(x) =

 

πi (x)[h(i) + C1(x ≤ i ≤ )].

i=0

For specific costs and transition probabilities, one can readily compute φ(x) from the preceding formulas for each x = 1, . . . , , and then select a control limit x that minimizes the cost. Although this optimal control-limit policy is a static policy, it is often optimal in the dynamic sense. Indeed, assume that, for each k,   i =k

pii ≤

 

pji ,

when i ≤ j.

i =k

In other words, the deterioration is higher in worse states, which would generally be true. Under this monotonicity condition, it is known from the theory of Markov decision processes (e.g., see [90] and examples in [33, 76, 109]) that within the class of all dynamic machine replacement policies, there is a control-limit policy that is optimal. Thus, the optimal control-limit policy described above is also optimal in the dynamic sense.

1.15 Closed Network Model We will now present two Markov chain models of networks in which discrete items move among nodes. This section describes a model for a closed network in which a fixed number of items move indefinitely among the nodes. The next section describes a model for an open network in which items enter from outside and move among the nodes for a while and eventually exit the network. The notation here is different from above. A typical state i will now be denoted by a vector x = (x1 , . . . , xm ) with nonnegative integer entries, and transition and stationary probabilities pij and πi will now be denoted by p(x, y) and π(x). Consider a network in which ν items move in discrete time among m nodes (or processing stations) in the set M = {1, . . . , m}. The state of the network 1 m i at time n is denoted by the random vector X n = (Xn , . . . , Xn ), where Xn  m i denotes the number of items at node i, and i=1 Xn = ν. The state space is the set S of all vectors x = (x1 , . . . , xm ) with nonnegative integer-valued m entries such that i=1 xi = ν.

56

1 Markov Chains

At each time period, exactly one item moves from its current node to another node or returns to the same node according to prescribed probabilities. Specifically, whenever the process is in a state x, one item is selected to move from node i with probability pi (xi ), and that item moves to a node j with a prescribed routing probability pij , for i, j in M. In other words, one item moves from node i to node j with probability pi (xi )pij , and this movement is independent of the past history. These selection and routing probabilities are such that pi (0) = 0, pi (k) > 0 if k > 0; and m 

m 

pi (xi ) = 1,

i=1

pij = 1,

x ∈ S, i ∈ M.

j=1

Under these assumptions, Xn is a Markov chain. Now, a typical transition is of the form x → x−ei +ej (one item moves from node i to node j) with probability pi (xi )pij . Here ei is the ith unit vector with 1 in position i and 0 elsewhere. Therefore, the transition probabilities p(x, y) = P {X1 = y|X0 = x} are pi (xi )pij if y = x − ei + ej for some i, j ∈ M p(x, y) = 0 otherwise. The communication properties of this chain are determined by the routing probabilities pij . Indeed, Exercise 40 shows that the chain is irreducible or aperiodic if and only if pij has these respective properties. One can envision these routing probabilities as defining a virtual or artificial Markov chain that depicts the state of a single item moving in M with a service time at each node being exactly one time unit. This virtual chain evolves like the network chain Xn with ν = 1. For simplicity, we will assume the routing probabilities pij are ergodic with stationary distribution wi . That is, wi satisfies the traffic equations  wj pji , i ∈ M. wi = j

Exercise 40 ensures that Xn is ergodic when it is aperiodic, since its state space is finite. The stationary distribution of the network chain is as follows.16 Theorem 86. The stationary distribution for the closed-network Markov chain Xn described above is π(x) = cf1 (x1 ) · · · fm (xm ), x ∈ S, (1.59)  i where fi (xi ) = wixi xk=1 pi (k)−1 . The normalization constant c is given by c−1 =



f1 (x1 ) · · · fm (xm ).

x∈S 16

Here and below, we use the convention that

0

k=1 (·)

= 1.

1.15 Closed Network Model

57

Proof. It suffices to show that π given by (1.59) satisfies the balance equations π(x) = y π(y)p(y, x). Since any transition into x has the form x + ei − ej → x,   π(y)p(y, x) = π(x + ei − ej )p(x + ei − ej , x)1(xj > 0). y

i,j

From the definitions of π and the transition probabilities, when xj > 0, wi pj (xj ) , wj pi (xi + 1) p(x + ei − ej , x) = pi (xi + 1)pij . π(x + ei − ej ) = π(x)

Substituting these expressions in the preceding display yields      π(y)p(y, x) = π(x) pj (xj ) wj−1 wi pij = π(x). y

j

i



The last equality follows since j pj (xj ) = 1, and  the stationary distribution wi of the routing probabilities satisfies wj = i wi pij . Thus π satisfies the balance equations π = πP, and hence it is the stationary distribution. In the preceding result, the stationary distribution π(x1 , . . . , xm ) = cf1 (x1 ) · · · fm (xm ) is the “joint distribution” for the network process in equilibrium. This distribution provides expressions for many performance parameters of the network. To describe c and the marginal distributions of π, we will use the functions   fi (xi ), 0 ≤ k ≤ ν, I ⊂ M. fI (k) = 

i∈I

xi =k i∈I

This fI is the convolution17 of the functions fi , i ∈ I. First note that c = fM (ν)−1 . Next, note that the equilibrium distribution of the number of items in a subset of nodes I is   πI (k) = π(x1 , . . . , xm )1( xi = k) x∈S

i∈I

= cfI (k)fI c (ν − k),

0 ≤ k ≤ ν.

In particular, if I = {i}, the ith “marginal distribution” of the number of items in node i is πi (k) = fi (k)fM\{i} (ν − k)/fM (ν), 17

0 ≤ k ≤ ν.

Properties of convolutions, which are not needed here, are in the Appendix.

58

1 Markov Chains

A standard performance parameter for the network is the average time items wait at a node or in a set of nodes. Example 87. Rates of Item Movements. We first consider the rate at which items move from node i to node j, which is r(i, j) = lim n−1 n→∞

n 

1(Xm = Xm−1 − ei + ej ).

(1.60)

m=1

By Corollary 79 and the definitions of π(x) and p(x, y),  π(x)p(x, x − ei + ej )1(xi ≥ 1) r(i, j) = x∈S

= wi pij



π(x − ei )1(xi ≥ 1).

x∈S

Now the last sum equals fM (ν − 1)/fM (ν), the normalization constant for the closed network with ν items divided by the constant for the network with ν − 1 items. Therefore r(i, j) =

fM (ν − 1) wi pij . fM (ν)

Furthermore, the rate r(I, J) at which items move from a set of nodes I to a set of nodes J is (1.60) summed over i ∈ I and j ∈ J. Consequently, r(I, J) =

fM (ν − 1)   wi pij . fM (ν) i∈I

j∈J

Example 88. Average Empty Times. Next, consider the average length of time that a set of nodes I is empty, which is the average time W (SI ) that Xn spends in SI = {x ∈ S : i∈I xi = 0}. By Example 83, W (SI ) =

 1 π(x), λ(SIc , SI ) x∈SI

where λ(SIc , SI ) =

 

π(y)p(y, x)

y∈SIc x∈SI

=

 

π(x − ej + ei )p(x − ej + ei , x)

x∈SI j∈I c i∈I

=



x∈SI

π(x)



j∈I c

pj (xj )wj−1

 i∈I

wi pij .

1.16 Open Network Model

59

Example 89. Equally-Likely Item Selection. Consider the closed network described above in which at each time period, any one of the ν items is randomly chosen (or is equally-likely) to move. That is, the probability that one of the xi items at node i is selected is pi (xi ) = xi /ν. In this case, the stationary distribution of the ergodic network Markov chain Xn is the multinomial distribution18 π(x) =

ν! xm wx1 · · · wm , x1 ! · · · xm ! 1

x ∈ S.

This result says that the joint distribution of the quantities of items in the nodes in equilibrium is equal to the multinomial distribution of quantities of items in m boxes, when ν items are independently put in the m boxes with respective probabilities w1 , . . . , wm . From the structure of the multinomial distribution, it follows that the equilibrium distribution of the quantity of items ina subset of nodes I is a binomial distribution with parameters ν and p = i∈I wi . More generally, if I, J, K is a partition of M, then the joint equilibrium distribution of the numbers of items in these subsets is the multinomial distribution πI,J,K (i, j, k) = where pI =

 i∈I

ν! i j k p p p , i!j!k! I J K

i + j + k = ν,

wi , and pJ and pK are defined similarly.

1.16 Open Network Model We will now consider an open m-node network in which items occasionally enter the network from outside and move among the nodes, as in the closed network above, but eventually the items exit the network. Much of the notation will be similar to that in the last section. The state of the network is denoted by the random vector Xn = (Xn1 , . . . , Xnm ), where Xni denotes the number of items at node i at time n, and the state space is S = Zm +. For a typical state x = (x1 , . . . , xm ), an item at node i is selected to move (as in the closed network) with probability pi (xi ), and pij is the probability the item is routed to node j. In addition, p0 (|x|) will denote the probability 18

Here the normalization constant in (1.59) is c = ν! because of w1 + · · · + wm = 1 and the multinomial formula 

(a1 + · · · + am )ν = x:



i xi =ν

ν! m ax1 · · · ax m . x1 ! · · · xm ! 1

60

1 Markov Chains

m that an item enters the network from outside, as a function of |x| = i=1 xi , the number of items in the network. Also, p0j and pi0 will denote the respective probabilities that an item from outside is routed to node j and an item at node i exits the network (think of 0 as the “outside node”). These probabilities are such that p0 (|x|) +

m 

pi (xi ) = 1,

x ∈ S,

i=1 m 

pij = 1,

0 ≤ i ≤ m.

j=0

A network transition at each time period is triggered by exactly one of the following events: • One item moves from outside the network to some node j in the network with probability p0 (|x|)p0j . • One item at some node i moves to a node j in the network with probability pi (xi )pij . • One item at some node i exits the network with probability pi (xi )pi0 . Under these assumptions, Xn is a Markov chain with transition probabilities ⎧ ⎨ p0 (|x|)p0j if y = x + ej for some 1 ≤ j ≤ m p(x, y) = pi (xi )pij if y = x − ei + ej for some 1 ≤ i ≤ m, 0 ≤ j ≤ m ⎩ 0 otherwise. Here e0 = 0. As in the closed network, the single-item routing probabilities pij determine the communication properties of the open-network chain Xn . Assume the routing probabilities pij on {0, . . . , m} are irreducible and aperiodic. Let w = (w0 , . . . , wm ), with w0 = 1, be an invariant measure of pij ; that is, wi =

m 

wj pji ,

0 ≤ i ≤ m.

j=0

Then an argument as in Exercise 40 proves that Xn is irreducible and aperiodic. The state space may be finite or infinite. It is infinite if p0 (k) > 0 for each k ≥ 0. On the other hand, assume that if p0 (k) = 0 for some k, then p0 () = 0, for  ≥ k. Then the total quantity of items in the network cannot exceed max{k : p0 (k) > 0} and the state space is finite (assuming the initial quantity is below this maximum). Theorem 90. The open-network Markov chain Xn described above is ergodic if and only if the state space is finite or

1.17 Reversible Markov Chains

c−1 =

61

 |x|−1 

p0 (k)

xi k=1

fi (xi ) < ∞,

i=1

x∈S k=0

where fi (xi ) = wixi

m 

pi (k)−1 . In that case, its stationary distribution is |x|−1

π(x) = c



p0 (k)

k=0

m 

fi (xi ),

x ∈ S.

(1.61)

i=1

Proof. Similarly to the proof of Theorem 86, one can show that π given by (1.61) with c = 1 is an invariant measure for the chain; this is left as Exercise 41. This observation, in light of Theorem 54, proves the assertions. Marginal distributions and rates of item movements for open networks are similar to those for closed networks. However, there are a few simplifications. Example 91. Independent Quantities at Nodes. Suppose the entry probabilities for the open network arep0 (k) = p, k ≥ 0. Then the joint distribution m (1.61) is the product π(x) = i=1 πi (xi ) of the marginal distributions πi (k) = ci (pwi )k

k 

pi ()−1 ,

k ≥ 0,

=1

∞ k = k=0 (pwi )k =1 pi ()−1 . So in equilibrium the state compowhere c−1 i nents Xn1 , . . . , Xnm are independent for each fixed n. Also, similarly to Example 87, the rate at which items move from node i to node j is 0 ≤ i, j ≤ m. r(i, j) = wi pij ,

1.17 Reversible Markov Chains We will now study an important class of “reversible” Markov chains. An ergodic Markov chain is reversible if its rate of transitions from one state to another is equal to the rate of the transitions in the reverse order. When the ergodic chain is stationary, the reversibility condition is equivalent to a “time-reversibility” property that, at any instant, the future of the process is stochastically indistinguishable from viewing the process in reverse time. A remarkable feature of an ergodic reversible Markov chain is that its equilibrium distribution has a known universal product-form (a product of a ratios of transition probabilities). Further properties of reversible Markov chains in continuous time are covered in Chapter 4. Throughout this section, Xn will denote an irreducible Markov chain on S with transition probabilities pij .

62

1 Markov Chains

Definition 92. The Markov chain Xn is reversible if there is a measure η on S that satisfies the detailed balance equations ηi pij = ηj pji ,

i = j in S.

(1.62)

We also say that Xn (or pij ) is reversible with respect to η. The η in this definition is an  invariant measure for the chain, since summing (1.62) over j yields ηi = j ηj pji . When η is finite, it can be normalized to be the stationary distribution of the chain. This definition of reversibility is related to the property of a chain being “reversible in time”; see Exercise 67. Note that if Xn is reversible, then it has the two-way communication property: for each i = j in S, the probabilities pij and pji are both positive or both equal to 0. This property yields the criterion that a chain is “not” reversible if a transition from some i to j is possible, but the reverse transition is not possible. For instance, the success runs chain in Example 19 is not reversible since a transition from a state i > 1 to 0 is possible but the reverse transition is not possible. Because of the two-way communication property, a periodic Markov chain with period greater than 2 cannot be reversible; see Exercise 57. A distinguishing characteristic of a reversible Markov chain is describable in terms of the rate of its transitions between sectors of its state space as follows; this is an immediate consequence of (1.62). Remark 93. Suppose Xn has an invariant measure η. Then Xn is reversible with respect to η if and only if   ηi pij = ηj pji , A, B ⊂ S. i∈A,j∈B

j∈A,i∈B

That is λ(A, B) = λ(B, A), using the rate notation of Example 80. Therefore, when the chain is ergodic, the rate of transitions between any two sets is equal to the rate of the reverse transitions. A quintessential reversible chain is as follows. Example 94. Random Walk. Consider a random walk Xn on S = {0, 1, . . .} with transition probabilities ⎡ ⎤ r0 p0 0 . . . . . . . ⎢ q1 r1 p1 0 . . . . ⎥ ⎥ P=⎢ ⎣ 0 q2 r2 p2 0 · · · ⎦ ............... Assume the pi and qi are positive. Then clearly this Markov chain is irreducible. Now, its detailed balance equations (1.62) (for the two respective cases j = i + 1, and j = i − 1 ) are ηi pi = ηi+1 qi+1 ,

i ≥ 0,

ηi qi = ηi−1 pi−1 ,

i ≥ 1.

1.17 Reversible Markov Chains

63

Note that these equations are the same. The last one is the recursive equation ηi = ηi−1 pi−1 /qi . Iterating this backward, we arrive at the solution ηi = η0

i 

pj−1 /qj ,

i ≥ 1,

(1.63)

j=1

where η0 is any positive number. Consequently, the chain is reversible and this η is a positive measure. Furthermore, this measure is finite if i ∞invariant and only if γ = i=1 j=1 pj−1 /qj is finite. Thus, the chain is ergodic if and only if γ < ∞. In that case, η given by (1.63) is the stationary distribution for the chain, where η0 = (1 + γ)−1 . Now, consider the chain restricted to S = {0, 1, . . . , m} so that its transition matrix is ⎤ ⎡ r0 p0 0 . . . . . . . . . . . . ⎢q1 r1 p1 0 . . . . . . . .⎥ ⎥ ⎢ ⎢ 0 q2 r2 p2 0 . . . .⎥ ⎥ P=⎢ ⎢ . . . . . . . . . . . . . . . . . .⎥ ⎥ ⎢ ⎣. . . . 0 qm−1 rm−1 pm−1 ⎦ . . . . . . . . . . qm rm The detailed balance equations for this chain are the same as those above, and  henceit is reversible and its stationary distribution is as above with i γ= m i=1 j=1 pj−1 /qj . We will see later in Chapter 4 that such restrictions of reversible Markov chains to smaller subspaces are also reversible and their stationary distributions are simply restrictions of the stationary distributions of the original chains. Exercises 50 and 52 discuss related queueing models. The most remarkable property of a reversible Markov chain is that an invariant measure for it is automatically given by expression (1.64) below, which is a product of ratios of the transition probabilities (we saw this in (1.63) above). In this result, a sequence of states i0 , . . . , in in S is a path from i0 to in if pik−1 ,ik > 0, k = 1, . . . , n. Theorem 95. If the Markov chain Xn is reversible, then an invariant measure for it is ηi0 = 1 and ηi =

  pik−1 ,ik , pik ,ik−1

i ∈ S\{i0},

(1.64)

k=1

where i0 is a fixed state and i0 , i1 , . . . , i = i is any path from i0 to i. A proof of Theorem 95 is contained below in the proof of (c) implies (a) in Theorem 97. Remark 96. One can construct the invariant measure η in (1.64) by the following recursion. Let S0 = {i0 } and

64

1 Markov Chains

Sn+1 = {i ∈ S\Sn : pij > 0 for some j ∈ Sn }. Then set ηi0 ≡ 1 and, for each n ≥ 1, define ηi =

pji ηj , pij

for i ∈ Sn+1 \Sn and any j ∈ Sn with pji > 0.

The “universal” invariant measure (1.64) of a reversible Markov chain is based on the following Kolmogorov criterion for reversibility. Statement (c) is a “ratio form” of Kolmogorov’s criterion, which justifies that (1.64) is the same for any path from i0 to i. With no loss in generality, we assume that the Markov chain Xn has the two-way communication property. Theorem 97. The following statements are equivalent. (a) The Markov chain Xn is reversible. (b) (Kolmogorov Criterion) For each i0 , . . . , i in S with i = i0 ,  

pik−1 ,ik =

k=1

 

pik ,ik−1 .

k=1

(c) For each path i0 , . . . , i in S, the product i0 , . . . , i and  only through i0 and i .



pik−1 ,ik k=1 pik ,ik−1

depends on

Proof. (a) ⇒ (b). If Xn is reversible with respect to η, then, for each i0 , . . . , i in S with i = i0 ,  

ηik−1 pik−1 ,ik =

k=1

 

ηik pik ,ik−1 .

k=1

Canceling the η’s yields (b). (b) ⇒ (c). To prove (c), it suffices to show  m   pik−1 ,ik pjk−1 ,jk = , pik ,ik−1 pjk ,jk−1

k=1

(1.65)

k=1

where i0 , . . . , i and j0 , . . . , jm are two paths with i0 = j0 and i = jm . Since i0 , . . . , i , jm−1 , . . . , j1 , j0 is a path from i0 to itself, (b) implies   k=1

pik−1 ,ik

m  k=1

pjk ,jk−1 =

m  k=1

pjk−1 ,jk

 

pik ,ik−1 .

k=1

These quantities are positive, by the definition of a path. Then dividing both sides of this equation by the second and fourth products yields (1.65). (c) ⇒ (a). Suppose (c) holds. We will show that Xn is reversible with respect to η defined by (1.64). For a fixed i, let i0 , . . . , i = i be a path from i0 to i.

1.17 Reversible Markov Chains

65

Choose any j such that pij > 0. Then   pik−1 ,ik pij pik ,ik−1 pji k=1 = pji ηj .

ηi pij = pji

These detailed balance equations also hold trivially for i, j such that pij = pji = 0. Thus Xn is reversible with respect to η. To verify the Kolmogorov criterion (b), or its ratio analogue (c), one may not have to consider all possible sequences or paths in S. In many instances, certain properties of pij and S lead to simpler versions of the Kolmogorov criterion. In particular, for some processes on vector state spaces only a small family of paths generated by the basis vectors need be considered. Here is another type of simplification. Remark 98. The Kolmogorov criterion holds for all paths, if it holds for paths consisting of distinct states (aside from the same beginning and end states). This is because any path can be partitioned into subpaths of distinct states. Reversibility of the Markov chain Xn can sometimes be recognized from the form of its communication graph. This is an undirected graph whose set of vertices is the state space S, and there is an edge linking a pair i, j if either pij or pji is not 0. The graph is connected when Xn is irreducible (which we have assumed). Theorem 99. If the Markov chain Xn has an invariant measure η, and its communication graph is a tree, then it is reversible with respect to η. Proof. Suppose i, j are vertices in the communication graph that are linked by an edge. Let Ai be the set of all the states in S reachable from i if the edge between i and j were deleted. Since the graph is a tree, it follows by the definition of Ai that ηi pij = λ(Ai , Aci ), ηj pji = λ(Aci , Ai ),  where λ(A, B) = k∈A,∈B ηk pk . Furthermore, we know from Example 81 that λ(A, Ac ) = λ(Ac , A), for any A ⊂ S. Therefore, the terms in the preceding display are equal, and so η satisfies the detailed balance equations, which proves the assertion. Note that the communication graph of the random walk Example 94 is a tree, but there are many reversible processes whose communication graphs are not trees. Example 100. Random Walk on a Circle. Suppose the Markov chain Xn takes values on the set of states S = {0, 1, . . . , } arranged clockwise on a circle. From state i the chain can move to state i + 1 with probability pi and move

66

1 Markov Chains

to state i − 1 with probability qi = 1 − pi , where  + 1 = 0 and 0 − 1 = . This circular random walk may not be reversible like the standard random walk in Example 94. Note that its communication graph is a circle (not a tree). Consequently, a path of distinct states from any state back to itself consists of all the states. In this case, the Kolmogorov criterion for reversibility is p0 · · · p = q0 · · · q .

(1.66)

In other words, the chain is reversible if and only if (1.66) holds. In this case, the stationary distribution given by (1.64) is πi = π0

i 

pk−1 /qk ,

1 ≤ i ≤ ,

k=1

  where π0−1 = 1 + i=1 ik=1 pk−1 /qk . When this chain is not reversible, its stationary distribution is still tractable; see Exercise 56. Even if the pi are all the same, the chain may not be reversible. The following is another example in which the Kolmogorov ratio criterion simplifies considerably. Example 101. McCabe’s Library. Consider a finite collection of books (or items or data) labeled 1, . . . , m that are placed in a row on a bookshelf. The successive book selections (one-at-a-time) by users are independent, and each user selects book b with probability pb . When a book at location 1 is selected, it is returned to that location. When a book at location k ≥ 2 is selected, it is returned to location k − 1, and the book there is placed in location k. This rearrangement is done before the next book is selected. The state of the library at any selection is a vector i = (i1 , . . . , im ), where ik denotes the book at location k. Then the state of the library at successive book selections is a Markov chain Xn on the set S of all m! permutations of the books (1, . . . , m). Its transition probabilities are pik if the book ik at location k is selected, for some k, pij = 0 otherwise. In the first line, j is the vector obtained from i after selecting book ik at location k. We will show the Markov chain is reversible by applying the Kolmogorov ratio criterion. Consider any path i0 , . . . , i of distinct states. Let b1 , . . . , b denote the book selections that determine this path starting with i0 . Now, for the reverse path i , . . . , i0 , let b1 , . . . , b denote the book selections that determine this reverse path starting with i . Then the Kolmogorov ratio is

1.17 Reversible Markov Chains

67

    pin−1 ,in pbn = . n n−1 p p n=1 i ,i n=1 bn

This product simplifies as follows. To move from i0 to i , each book ik with < i0k ) has to be selected at least i0k − ik times; and after the (i0k − ik )-th selection, each subsequent bn book selection has to be compensated by the associated book bn . Similarly, to move in reverse from i to i0 , each book i0k with (i0k < ik ) has to be selected at least ik − i0k times; and after the (ik − i0k )-th selection, each subsequent bn selection has to be compensated by the associated bn . Consequently, (ik

 m   pin−1 ,in (i0 −i ) = pik k . k p n n−1 n=1 i ,i

(1.67)

k=1

Here is a typical path and its reverse path, with the book selections above the arrows. 3

3

2

4

2

1

1

1

(1, 2, 3, 4) −→ (1, 3, 2, 4) −→ (3, 1, 2, 4) −→ (3, 2, 1, 4) −→ (3, 2, 4, 1) (1, 2, 3, 4) ←− (1, 3, 2, 4) ←− (3, 1, 2, 4) ←− (3, 2, 1, 4) ←− (3, 2, 4, 1) Then the product (1.67), with (i0k − ik : 1 ≤ k ≤ 4) = (−3, 0, 2, 1), is p23 p2 p4 2 = p−3 1 p3 p4 . p31 p2 Note that the product (1.67) does not depend on the interior states i1 , . . . , i−1 of the path. Then by Theorem 97, the Markov chain Xn is reversible, and an invariant measure for it is the product (1.67) evaluated at i = i, where i0 is fixed. Therefore, setting i0 = (1, . . . , m), the stationary distribution is m  (k−i ) pik k , i ∈ S, (1.68) πi = c k=1

where c is the normalization constant under which this is a distribution. A similar model applies for an infinite book collection. This model assumes the state space S is the set of all permutations of the books (1, 2, . . .) obtained by a finite number of book selections. The argument above yields the same invariant measure (1.68), where m is replaced by min{k : ik = k  , k  > k}, the first location in i after which all the books are in their starting locations. See Exercise 58 for some details on these models.

68

1 Markov Chains

1.18 Markov Chain Monte Carlo This section describes a few standard Markov chains that are used in Monte Carlo simulations. The field of simulation and related statistics (e.g., see [43]) is very important for numerical explorations of systems, but our discussion will not go beyond an introduction to Markov chain procedures for estimation of system parameters. There are a variety of statistical problems in which one uses Monte Carlo simulations for estimating expectations of the form  μ= g(i)πi , i∈S

where π is a specified probability measure and g : S → R. A standard Markov chain Monte Carlo approach is to construct an ergodic Markov transition matrix whose stationary distribution is π. Then for a sample path X1 , . . . , Xn of the chain, an estimator for μ is μ ˆn = n

−1

n 

g(Xm ).

m=1

By Theorem 74 for Markov chains, μ ˆn → μ a.s., and so μ ˆ n is a consistent estimator19 of μ. Other consistent estimators are given in Exercise 59. In some applications, the target distribution has the form πi = cηi , where η is known, but the normalization constant c is unknown and is difficult to compute. In this case, one can obtain consistent estimators for c as well as μ as in Exercise 59. For a particular application, one would simulate the Markov chain for a large number of steps n, and then take the resulting μ ˆn as an approximate value of μ. An implementation of this procedure requires the formulation of an ergodic Markov chain whose stationary distribution is π. There are many ways this can be done. One approach is to use a reversible Markov chain whose transition probabilities have the form pij = r(i, j)/πi ,

i, j ∈ S,

(1.69)

where r(i, j) = r(j, i). This chain is clearly reversible with stationary distribution π, provided the r(i, j) are chosen so that the chain is ergodic. Here are two standard examples. Example 102. Hastings-Metropolis Markov Chain. Let π be a positive probability measure on S, and consider a Markov chain with transition probabilities A statistic θn that is a function of observed values X1 , . . . , Xn , is a consistent estimator of a parameter θ if θn → θ a.s. as n → ∞. 19

1.18 Markov Chain Monte Carlo

pij =

69

πj γji /(πi γij )} if j =  i, γij min{1,  if j = i. 1 − k=i pik

(1.70)

Here γij are probabilities that one selects such that the chain is ergodic. These transition probabilities may seem rather artificial, but they are amenable to easy simulations for obtaining estimates discussed above. Note that the Markov chain is reversible with stationary distribution π, since pij can be written as in (1.69) with r(i, j) = min{πi γij , πj γji }. An important feature of the pij is that they do not depend on the normalization constant c, where πi = cηi ; it is often difficult to calculate c. Also, the probabilities simplify to pij = γij min{1, πj /πi } when γij = γji , for each i and j. A Hastings-Metropolis simulation generates transitions according to the probabilities (1.70) as follows. Whenever the chain is in state i, the next state is determined by the following steps. (1) Select a state j with probability γij . (2) If state j is selected, then choose j as the next state with probability min{1, πj γji /(πi γij )}; and otherwise choose the current state i to be the next state. These steps are repeated for successive transitions. The preceding example is for a single- or multi-dimensional state space; the next example is for the latter case. Example 103. Gibbs Sampler. Let π be a probability measure on a set S of vectors of the form i = (i1 , . . . , im ), and let (Y1 , . . . , Ym ) denote a random vector with probability measure π. Consider a Markov chain with transition probabilities pij = m−1 P {Yk = jk |Y = i ,  = k},

if j = i ,  = k, for some k. (1.71)

Also, pij = 0 otherwise. An easy check shows that these probabilities are of the form (1.69), and hence the Markov chain is reversible with stationary distribution π. A Gibbs simulation generates transitions according to (1.71) as follows. Whenever the chain is in state i, the next state is determined by changing a single component of i by the following steps. (1) Randomly select a component of i that is to be changed: component k is selected with probability 1/m. (2) For the component k to be changed, choose a value jk with probability P {Yk = jk |Y = i ,  = k}, and then take the new state j to be i with ik changed to jk . These steps are repeated for successive transitions. A variation of this simulation can be constructed by changing the components one at a time in a specified order, and repeating this indefinitely. The resulting Markov chain has the stationary distribution π, but it may not be reversible (which is not essential for estimations).

70

1 Markov Chains

Hastings-Metropolis and Gibbs Sampler Markov chains are illustrated by the following example. Example 104. Hastings-Metropolis Simulation. We will consider a probability measure π on the set of vectors S = {0, 1, . . . , L}m . Let M denote any family of subsets of {1, . . . , m}, and, for i = (i1 , . . . , im ) ∈ S and A ∈ M, let  iA = k∈A ik . Suppose that π has the form πi = c



fA (iA ),

A∈M

where fA , A ∈ M, are positive functions that are known. This type of distribution arises in stochastic networks, where iA is the number of items at the nodes in A. To construct a Hastings-Metropolis simulation, we only need to decide which transitions are to be feasible and to choose the transition probabilities γij . Assume the feasible transitions from a state i will be into the set S(i) = {i − ek + e ∈ S : k =  ∈ {0, 1, . . . , m}}. Recall that i − ek + e is the vector i with one unit subtracted from ik and one unit added to i . The number of its elements is |S(i)| ≤ m(m + 1). Next, assume γij = 1/|S(i)|, which means that each j ∈ S(i) is equally likely. Then the transition probabilities (1.70) are pij =

1 min{1, πj /πi }, |S(i)|

j ∈ S(i).

Letting Mk = {A ∈ M : k ∈ A}, it follows by the definition of πi that  πj A∈Mk \M rA (iA ) , for j = i − ek + e , =  (1.72) πi A ∈M \Mk rA (iA + 1) where rA (n) = fA (n − 1)/fA (n) and M0 is the empty set. Assume the functions fA are such that the Markov chain is ergodic. Then the simulation generates transitions as follows. Whenever the chain is in state i, the simulation obtains the next state by the following rules. (1) Randomly select a state j ∈ S(i) with probability 1/|S(i)|. (2) If j is selected, then choose it as the next state with probability min{1, πj /πi }, and otherwise, choose the current state i to be the next state. Example 105. Gibbs Sampler Simulation. Suppose that (Y1 , . . . , Ym ) has the joint distribution π in the preceding example. Then a Gibbs simulation generates transitions as follows. Whenever the chain is in state i, the next state is determined by changing a single component of i by the following steps.

1.19 Markov Chains on Subspaces

71

(1) Select a component k of i that is to be changed with probability 1/m. (2) For the component k to be changed, choose a value jk ∈ {0, 1, . . . , L} with probability  fA (iA + jk − ik ) k P {Yk = jk |Y = i ,  = k} = L A∈M ,   A∈Mk fA (iA + jk − ik ) j  =0 k

where j = i ,  = k, for some k. Then take the new state j to be i with ik changed to jk .

1.19 Markov Chains on Subspaces This section discusses two types of Markov chains that are associated with observing a Markov chain on parts of its state space. The first type of chain in the following example addresses the questions: If one observes a Markov chain only on a certain subset of states, is the observed sequence of states a Markov chain? If so, what are its transition probabilities, and does it inherit the stationary distribution of the parent chain? Example 106. Markov Chain on a Subspace. Let Xn be an irreducible Markov chain on S. Suppose the hitting time of a fixed subset S  ⊂ S is finite starting from at any state; then the successive times that it visits S  are finite. Let Xn denote the state of the chain at its nth visit to S  . By the strong Markov property for Xn at the hitting times of S  , it follows that Xn is an irreducible Markov chain on S  . This chain is called the restriction of Xn to S  . We will show that its transition probabilities are  pij = pij + pik γkj , (1.73) k∈S 

where γkj is the probability the chain Xn beginning in state k eventually hits S  in state j (such hitting probabilities are described in Section 1.7). In addition, we show that if Xn is recurrent, then so is Xn , and an invariant measure for it is an invariant measure for Xn restricted to S  . To derive the transition probabilities, note that a transition of Xn out of a state i consists of (instantaneously) selecting a sequence of states, say k1 , k2 , . . . , k , by the probabilities pik1 , pk1 ,k2 , . . . , pk ,j until some j ∈ S  is reached. Then, conditioning on the first selection, the transition probabilities of Xn are ∞   pij = pij + pik1 pk1 ,k2 · · · , pk ,j . =1 k1 ,...,k ∈S 

The last sum can also be written as in (1.73), which proves (1.73).

72

1 Markov Chains

Clearly Xn is recurrent when Xn is. Next, recall from Theorem 53 that an invariant measure for Xn is as follows: For a fixed i ∈ S, ηj = Ei

i −1

τ

 1(Xn = j) ,

j ∈ S.

(1.74)

n=0

The ηj is the expected number of visits Xn makes to state j in between visits to the fixed reference state i. However, for i, j ∈ S  , the ηj is also the expected number of visits Xn makes to j between visits to i. Thus by Theorem 53, the ηj above for i, j ∈ S  defines an invariant measure for Xn . The next type of Markov chain addresses the questions: Can the stationary distribution of a Markov chain be constructed by determining the stationary distributions of the chain restricted to certain subsets of the state space by pasting these distributions together? If so, is there a procedure for doing the pasting? Example 107. Star-Like Collage of Subchains. Let Xn be an irreducible Markov chain on S. Suppose S0 , S1 , S2 , . . . is a countable partition of S such that whenever the chain is in any Sk , it can only take transitions into S0 ∪Sk . The partition is star-like in that to move from one set in the partition to another the chain must go through S0 . Assume that the chain restricted to each set S0 ∪ Sk is ergodic with stationary distribution pki , i ∈ S0 ∪ Sk . For simplicity, assume S0 consists of the single state 0; see Exercise 60 for the case when S0 is not a singleton. Under these conditions, a natural candidate for the stationary distribution of Xn is πi = π0 ck pki ,

if i ∈ S0 ∪ Sk for some k,

where π0 and ck are to be determined. This is a collage (or pasting together) of the stationary distributions pk of the subchains. First note that ck = 1/pk0 , since π0 = π0 ck pk0 because S0 = {0}. In addition, π0 is determined by    πi = π0 + π0 ck pki . 1= i

k

i∈Sk

Thus, we have the following result. If the preceding double sum is finite, then Xn is ergodic and its stationary distribution from above would be πi = π0 pki /pk0 ,

if i ∈ Sk for some k,

(1.75)

−1    . An easy check shows that this distribution and π0 = 1+ k i∈Sk pki /pk0 satisfies π = πP. Example 108. Dual Birth-Death Subprocesses. Suppose Xn takes values in the set of integers. Assume that in order for it to move between the positive

1.20 Limit Theorems via Coupling

73

and negative integers it must pass through 0 and, it can enter 0 only from states 1 or −1. Furthermore, assume that the chain behaves like an ergodic birth-death process on the nonnegative integers with stationary distribution p1i = (1 − ρ1 )ρi−1 , where 0 < ρ1 > 1. Similarly, the chains behavior on the nonpositive integers is that of a birth-death process with stationary distribution p2i = (1 − ρ2 )ρi−1 . This process is an example of the model above in which the communication graph of the process is a star with center set S0 = {0} and point sets S1 = {1, 2, . . .} and S2 = {. . . , −2, −1}. Under the assumptions, the stationary distribution of the process is i ≥ 1, π0 ρi−1 1 , πi = , i ≤ 1, π0 ρi−1 2 where π0−1 = 1 + 1/(1 − ρ1 ) + 1/(1 − ρ2 ). Example 109. Random Walks on Intersecting Circles. The random walk on discrete points on a circle described in Example 100 has a tractable stationary distribution, even when it is not reversible (Exercise 56). Consider the generalization of a random walk on several ellipses that have 0 as a common point. Then the stationary distribution of the random walk is a collage of the stationary distributions on the ellipses.

1.20 Limit Theorems via Coupling The material in this section was used to classify states of a Markov chain (Theorem 37), and to prove the important property that a stationary distribution is a limiting distribution (Theorem 59). The first result is that if two independent irreducible and recurrent Markov chains have the same transition probabilities, then the difference between their distributions converges to 0 as time tends to infinity. It is based on constructing Markov chains on a common probability space with certain properties. This is an example of coupling, which refers to constructing stochastic processes (not necessarily on the same probability space) in order to prove convergence in distribution, stochastic ordering, rates of convergence of probability measures, etc. Theorem 110. Suppose Xn and Yn are independent, irreducible, aperiodic recurrent Markov chains on S with arbitrary initial distributions, but with the same transition probabilities. Then sup |P {Xn = i} − P {Yn = i}| → 0, i

as n → ∞.

(1.76)

Proof. For a fixed state i0 , let τ = min{n ≥ 1 : Xn = Yn = i0 }. Clearly τ is a stopping time of the process Zn = (Xn , Yn ), n ≥ 0. Under the hypotheses, Exercise 27 shows that Zn is an irreducible recurrent Markov chain on S 2 . Consequently, τ < ∞ a.s.

74

1 Markov Chains

Let pij denote the transition probabilities for the two Markov chains Xn and Yn . To prove their distributions get close to each other, the key observation is that the two chains are both equal to i0 at time τ and, at any time thereafter, they have the same (conditional) distribution, since the chains have the same transition probabilities. In particular, since Xm = Ym = i0 when τ = m, it follows by the strong Markov property that, for n > m, P {Xn = i|τ = m} = pn−m i0 ,i = P {Yn = i|τ = m}.

(1.77)

Next, define Xn∗ = Xn for n ≤ τ and Xn∗ = Yn otherwise. Clearly the chain is equal in distribution to the chain Xn , since both chains have the same transition probabilities and initial distribution. Using this fact and (1.77), Xn∗

|P {Xn = i} − P {Yn = i}| = |P {Xn∗ = i} − P {Yn = i}| = |P {Xn∗ = i, τ > n} − P {Yn = i, τ > n}| ≤ 2P {τ > n} → 0. This proves (1.76). Recall from Theorem 59 that the stationary distribution for an ergodic Markov chain is also its limiting distribution. This result follows from statement (1.78) below that the probability measure of an ergodic Markov chain converges in total variation distance20 to its stationary probability measure. This mode of convergence, of course, implies convergence in distribution. Theorem 111. (Limiting Distributions) If Xn is an ergodic Markov chain with stationary distribution π, then sup |P {Xn = i} − πi | → 0, i

as n → ∞.

(1.78)

Hence π is the limiting distribution of Xn . Proof. Let Yn be a Markov chain defined on the same probability space as Xn with the same transition probabilities as Xn . Suppose the two chains are independent and that Yn is stationary. Then P {Yn = i} = πi . Thus (1.78) follows by Theorem 110. Next, we use Theorem 110 to prove the following result, which is a restatement of Theorem 38. This characterization of null-recurrence was used for classifying states of a Markov chain. Theorem 112. For an irreducible Markov chain on S with transition probabilities pij , a recurrent state i is null-recurrent if and only if 20

The total variation distance between two probability measures P and P  on a space S is d(P, P  ) = supB |P (B) − P  (B)|, where the  supremum is over all sets B in the σ-field of S. When S is countable, d(P, P  ) = 1/2 i∈S |P (i) − P  (i)|. Probability measures Pn on S converge in total variation distance to a probability measure P if d(Pn , P ) → 0 as n → ∞.

1.20 Limit Theorems via Coupling

75

lim pnii = 0.

(1.79)

n→∞

In this case, limn→∞ pnji = 0, for j ∈ S. Proof. We will prove the equivalent statement that i is positive recurrent if and only if lim sup pnii > 0. (1.80) n→∞

First assume (1.80) holds. Then the diagonal selection principle21 applied to the probabilities pnij yields the existence of a strictly increasing subsequence nm and constants γj such that γi > 0 and limm→∞ pnijm = γj , j ∈ S. Furthermore, k ∈ S. lim pnkjm = γj , m→∞

Indeed, applying Theorem 110 to the middle term in the following, we have |pnkjm − γj | ≤ |pnkjm − pnijm | + |pnijm − γj | → 0. We will now show that γ is a finite invariant measure. The γ is finite, since Fatou’s lemma for sums22 yields    γj = lim pnijm ≤ lim inf pnijm = 1. j

j

m→∞

m→∞

j

Next, by the Chapman-Kolmogorov equations,   nm +1 = pnikm pkj = pik pnkjm . pij k

k

Then applying Fatou’s lemma to the first sum and the dominated convergence theorem to the second sum yields   γk pkj ≤ pik γj = γj . k

k

If this inequality is a strict inequality for some j’s, then summing on j,    γk = γk pkj < γj , k

j

which is a contradiction. Thus, γj = measure. 21

j

k

 k

γk pkj , and hence γ is an invariant

The diagonal selection principle for bounded real numbers {aj (n) : j ∈ S, n ≥ 1} states that there exists a strictly increasing subsequence of integers nm such that the limit limm→∞ aj (nm ) exists for each j.   22 From Theorem 12 in the Appendix, j lim inf n→∞ aj (n) ≤ lim inf n→∞ j aj (n), for aj (n) that are bounded from below.

76

1 Markov Chains

Now γ is a multiple of the invariant measure η in Theorem  53. From (1.42), = the mean time between entrances to state i is given by μ i k ηk , and this,  being a multiple of k γk , is finite. Thus i is positive recurrent. The next step is to show that if i is positive recurrent then (1.80) holds. The positive recurrence of i implies (using the notation in the last paragraph) that μi = k ηk is finite. Then πj = ηj /μi is a stationary distribution for the chain. Now if the chain is aperiodic, Theorem 110 with P {Y0 = j} = πj , j ∈ S, yields pnii → πi > 0, which proves (1.80). Also, when the chain is periodic, (1.80) follows from a slight variation of Exercise 29. Finally, assume (1.79) holds. Using the strong Markov property, pnji =

n 

m n−m fji pii ,

i, j ∈ S.

m=1 m Recall that fji is the probability that, starting at j, the first entrance of the chain to i is at time m. Then as n → ∞, the dominated convergence theorem m n−m pii 1(m ≤ n) → 0. yields pnji → 0, since (1.79) ensures fji

1.21 Criteria for Positive Recurrence We know by Theorem 54 that an irreducible Markov chain is positive recurrent if and only if it has a stationary distribution. For a complicated Markov chain, it may not be feasible to obtain a closed-form expression for its invariant measure or stationary distribution. However, it is still of interest to establish that an irreducible Markov chain is positive recurrent. One criterion for this is that the time to return to a fixed state has a finite mean (recall Theorem 54). This section presents more general criteria for positive recurrence based on showing that the time to hit a finite subset of states (instead of a fixed state) has a finite mean. As usual, let Xn denote a Markov chain on S with transition probabilities pij . Consider the hitting time τF = min{n ≥ 1 : Xn ∈ F } of F ⊂ S. We begin with a preliminary result. Proposition 113. If Xn is irreducible, and there is a finite set F ⊂ S such that b = maxj∈F Ej [τF ] < ∞, then Xn is positive recurrent. Proof. It suffices to show that a single state in F is positive recurrent. To this end, consider the restriction of the chain to F defined by Xn = XτF (n) , where τF (n) is the nth time Xn visits F and Xn is the state of the chain at that visit. Example 106 showed that Xn is an irreducible Markov chain on F ; and it is positive recurrent since F is finite. Now, for a fixed i ∈ F , the hitting time τi = min{n ≥ 1 : Xn = i} has the form

1.21 Criteria for Positive Recurrence

τi =

∞ 

77

(τF (m + 1) − τF (m))1(τi > m),

m=0

where τF (0) = 0 and τi = min{n ≥ 1 : Xn = i}. Using the strong Markov property of Xn at τF (m), Ei [τi ] = =

∞  m=0 ∞ 



  Ei E (τF (m + 1) − τF (m))1(τi > m)X0 , X1 , . . . , XτF (m)

   [τF ]1(τ > m) . Ei EXm i

m=0

The last line uses the pull-through formula (1.90) and the fact that the event {τi > m} = {Xτk = i; k ≤ τF (m)} is a function of X0 , X1 , . . . , XτF (m) . The positive recurrence of Xn ensures that Ei [τi ] < ∞. Using this and the  [τF ] ≤ b in the preceding, we have assumption EXm Ei [τi ] ≤ b

∞ 

Pi {τi > m} = bEi [τi ] < ∞.

m=0

Thus i is positive recurrent for Xn . The next result shows that the finite mean-hitting-time condition in Proposition 113 is satisfied if there exist a function v and a set F that satisfy (1.82). The function v assigns a real number to each state, which provides an “artificial” v-ordering of the states. The assumption (1.82) is equivalent to the existence of an ε > 0 such that Ei [v(X1 )] < v(i) − ε,

i ∈ F c.

(1.81)

That is, the mean v-order of the chain decreases at each of its jumps initiated outside the finite set F . This ensures (the main part of the proof) that Ej [τF ] < ε−1 v(j) < ∞,

j ∈ F c.

Finding such a function v is often done by trial and error since there are no known procedures for constructing it. Sufficient conditions for v(i) = i are in the follow-on example. Theorem 114. (Foster’s Criterion) Suppose Xn is an irreducible Markov chain, and there are a function v : S → R+ and a set F such that Ei [v(X1 )] < ∞, i ∈ F , and sup Ei [v(X1 ) − v(X0 )] < 0.

(1.82)

i∈F c

Then Ei [τF ] < ∞, i ∈ S. Furthermore, Xn is positive recurrent if F is finite.

78

1 Markov Chains

Proof. Let τ = τF . We first prove Ej [τ ] < ∞, j ∈ F c . Fix j ∈ F c and let γj (n) = Ej [v(Xn )1(τ > n)]. Conditioning on Hn = (X0 , . . . , Xn ) and using {τ > n + 1} ⊆ {τ > n}, we have

 γj (n + 1) ≤ Ej Ej [v(Xn+1 )1(τ > n)|Hn ] . Since 1(τ > n) is a function of Hn , and Xn ∈ F c when τ > n, it follows by (1.81) that there is an ε > 0 such that Ej [v(Xn+1 )1(τ > n)|Hn ] = 1(τ > n)EXn [v(Xn+1 )] < 1(τ > n)(v(Xn ) − ε). Using this inequality in the preceding display yields γj (n + 1) < γj (n) − εPj {τ > n}. Iterating this inequality for n, n − 1, . . . , 1, we have 0 ≤ γj (n + 1) < γj (0) − ε

n 

Pj {τ > k}.

k=0

Letting n → ∞, the sum converges to Ej [τ ], and therefore εEj [τ ] < γj (0) = v(j) < ∞,

j ∈ F c.

(1.83)

The aim is to prove Ei [τ ] < ∞ for all i ∈ S, and so it remains to show Ei [τ ] < ∞, i ∈ F . But this follows since conditioning on X1 and using (1.83),    pij + pij (1 + Ej [τ ]) ≤ 1 + ε−1 pij v(j) < ∞. Ei [τ ] = j∈F

j∈F c

j∈F c

Finally, if F is finite, then Xn is positive recurrent by Proposition 113. Example 115. Pake’s Criterion. An irreducible Markov chain Xn on the nonnegative integers S is positive recurrent if Ei [X1 ] < ∞, i ∈ S, and lim sup Ei [X1 − i] < 0.

(1.84)

i→∞

This result is a special case of Foster’s criterion with v(i) = i, since (1.84) implies that there is an i∗ such that supi≥i∗ Ei [X1 − i] < 0. Example 116. Fork-Join Processing System. Consider a fork-join network shown in Figure 1.4 that processes jobs as follows. Jobs arrive according to a Bernoulli process, where p is the probability of an arrival at any discrete time. Each job arriving to the system instantaneously splits into m tasks, which are simultaneously assigned to the m nodes for processing. The m nodes operate

1.21 Criteria for Positive Recurrence

79

like M/M/1 service systems in discrete time as in Example 21, but not more than one job can be completed in each time period. The service time at node i has a geometric distribution where qi is the probability of a service completion in any discrete time. When all of its m tasks are finished, the job is completed and exits the system. Fork-join networks are natural models for a variety of computer, telecommunications and manufacturing systems that involve parallel processing. For instance, a fork-join computer or telecommunications network typically represents the processing of computer programs, data packets, telephone calls, etc. that involve parallel multi-tasking and splitting and joining of information. A manufacturing fork-join network represents the assembly of a product or system that requires several parts processed simultaneously at separate work stations or plant locations. A supply chain fork-join network typically represents filling an order by obtaining several products simultaneously from vendors (or warehouses or manufacturing plants).

Tasks

Tasks

1

2

Jobs Fork -Node

Join -Node

Tasks

m

Fig. 1.4 Fork-Join Network.

The state of the fork-join network in Figure 1.4 at time n is represented by a vector-valued process Xn with states x = (x1 , . . . , xm ), where xi denotes the quantity of tasks at node i. Under the assumptions above, Xn is a Markov chain with transition probabilities m 

p(x, x + (1, 1, . . . , 1)) = p

(1 − qj )

j=1 xj >0

p(x, x − ei ) = qi 1(xi > 0)(1 − p)

m 

(1 − qj ),

j=1, j=i xj >0

1 ≤ i ≤ m.

80

1 Markov Chains

Also, p(x, x) = 1− the sum of the preceding probabilities, and p(x, y) = 0 elsewhere. Here ei is the vector with 1 in position i and 0 elsewhere. This network chain Xn is one of those infamous queueing processes whose stationary distribution is intractable. However, we can use Foster’s criterion to prove that the chain is positive recurrent under the assumption p < qi ,

1 ≤ i ≤ m.

(1.85)

That is, the arrival rate is less than the service rate at each node. Exercise 68 shows that (1.85) is a necessary as well as a sufficient condition for Xn to be positive recurrent. A natural firstguess is that Foster’s criterion will work with the linear m function v(x) = i=1 xi , but it does not as Exercise A second  68 verifies. 2 choice is to consider the quadratic function v(x) = m i=1 xi . Clearly, Ex [v(X1 )] ≤

m 

(xi + 1)2 < ∞,

x ∈ S.

i=1

Next, for x ≥ (1, 1, . . . , 1), consider  p(x, y)(v(y) − v(x)) D(x) = Ex [v(X1 )] − v(x) = y=x m 

= p(x, x + (1, 1, . . . , 1))

(2xi + 1) +

i=1

=

m 

(1 − qj )

j=1

m 

p(x, x − ei )(−2xi + 1)

i=1

m  g(xi ) , 1 − qi i=1

where g(xi ) = 2xi (p − qi ) + p(1 − qi ) + qi (1 − p). The aim is to find a finite set F for which supx∈F c D(x) < 0. Note that assumption (1.85) ensures that D(x) is decreasing in x. Now, the smallest vector b = (b1 , . . . , bm ) for which D(b) < 0 is defined by bi = min{xi ≥ 1 : g(xi ) < 0}. Then setting F = {x : xi < bi , 1 ≤ i ≤ m}, it follows that D(x) ≤ D(b) < 0,

x ∈ F c.

Thus the network chain Xn is positive recurrent by Foster’s criterion.

1.22 Review of Conditional Probabilities

81

1.22 Review of Conditional Probabilities Applied probability involves extensive use of conditional probabilities and expectations. This section reviews these concepts for discrete random variables; analogous properties for continuous and general random variables are in the Appendix. For this discussion, X and Y are discrete random variables that take values in countable sets S and S  , respectively; and X and Y are defined on the same underlying probability space. For instance, X and Y could be discrete, realvalued random variables or vectors. The conditional probability measure of Y given X = x, for x ∈ S, is p(y|x) = P {Y = y|X = x} =

P {X = x, Y = y} , P {X = x}

y ∈ S ,

provided P {X = x} > 0. This proviso will be assumed for all conditional probabilities without mention. For a real-valued Y (S  ⊂ R), the conditional expectation (or mean) of Y given X = x is  E[Y |X = x] = yp(y|x), x ∈ S. y

provided the sum is absolutely convergent. Unless specified otherwise, all the expectations in this section are assumed to be finite. Consider a random variable of the form g(Y ), where g : S  → R. Then as above, the conditional expectation of g(Y ) given X = x is  E[g(Y )|X = x] = g(y)p(y|x), x ∈ S. y

Conditional probabilities are often used to determine the distribution or mean of a random variable as follows. Suppose the issue is to find the distribution of Y or the mean of g(Y ), when the conditional probabilities p(y|x) given X = xare known and p(x) = P {X = x} are also known. Using P {Y = y} = x P {X = x, Y = y} and the definitions above,     p(y|x)p(x), E[g(Y )] = g(y)p(y|x) p(x). (1.86) P {Y = y} = x

x

y

Examples are given in Exercises 1 and 3. We will often use the following standard shorthand notation for conditional probabilities and expectations. The conditional probability measure of Y given X (without specifying the value of X) is defined by P {Y = y|X} = p(y|X)

y ∈ S .

82

1 Markov Chains

Similarly, the conditional expectation of g(Y ) given X is E[g(Y )|X] = h(X),  where h(x) = E[g(Y )|X = x]. That is, E[Y |X] = y g(y)p(y|X). Note that these shorthand representations of conditional probabilities and expectations are random variables that are deterministic functions of X. With this notation, (1.86) becomes P {Y = y} = E[P {Y = y|X}],

E[g(Y )] = E[E[g(Y )|X].

The last formula is a special case of a result for general random variables as described in the Appendix. Namely, for a random variable X that takes values in a general space and a real-valued random variable Y , E[Y ] = E[E[Y |X] ].

(1.87)

This is a frequently-used formula for expectations. Many properties of probabilities and expectations extend to conditional probabilities and expectations. For instance, suppose Z is an S  -valued random variable on the same probability space as X and Y . Then23 for h : S  → R, E[g(Y ) + h(Z)|X] = E[g(Y )|X] + E[h(Z)|X], E[g(Y )|X] ≤ E[h(Z)|X] when g(Y ) ≤ h(Z). Another variation involves multiple conditioning, such as  E[h(Z)|X, Y = y]P {Y = y|X}. E[h(Z)|X] = y

An important point is that conditioning on X = x, allows one to replace X by x throughout in certain instances. In particular, from the definition of conditional expectation, for G : S × S  → R, E[G(X, Y )|X = x] = E[G(x, Y )|X = x],

x ∈ S.

(1.88)

Moreover, E[G(x, Y )|X = x] = E[G(x, Y )], when X and Y are independent. For instance, E[X(Y − X)|X = x] = xE[(Y − x)|X = x]. Similar expressions hold for conditional probabilities, such as P {X 2 + |Y − X| ≤ z|X = x} = P {x2 + |Y − x| ≤ z|X = x}. In addition, for H : S × S  → R, 23 Statements like g(Y ) ≤ h(Z), E[g(Y )|X] > V and Y = 0 hold a.s. but it is standard to suppress a.s.

1.22 Review of Conditional Probabilities

83

E[G(X, Y )|H(X, Y )]  = E[G(x, Y )|H(x, Y ), X = x]P {X = x|H(X, Y )}.

(1.89)

x

Expression (1.88) also yields the pull-through formula: For f : S → R, E[f (X)g(Y )|X] = f (X)E[g(Y )|X].

(1.90)

That is, any function of the conditioning variable X can be pulled out of the expectation. In some cases, we use conditioning statements in which only some of the variables are specified. For instance, P {Y = y|V, Z, X = x} = h(V, Z, x), where h(v, z, x) = P {Y = y|V = v, Z = z, X = x}. Another important concept related to Markov chains is that of conditional independence. Recall that X and Y are independent if P {X = x, Y = y} = P {X = x}P {Y = y},

x ∈ S, y ∈ S  .

Equivalently, for any f : S → R, E[f (X)g(Y )] = E[f (X)]E[g(Y )]. Analogously, Y and Z are conditionally independent given X if P {Y = y, Z = z|X} = P {Y = y|X}P {Z = z|X},

y ∈ S  , z ∈ S  .

Equivalently, E[g(Y )h(Z)|X] = E[g(Y )|X]E[h(Z)|X] for any functions g and h for which the expectations exist. More generally, S  -valued random variables Y1 , . . . , Yn are conditionally independent given X if P {Y1 = y1 , . . . , Yn = yn |X} =

n 

P {Yk = yk |X}.

k=1

Frequent use is made of the property that if Y is conditionally independent of X1 , . . . , Xn−1 given Xn , then P {Y = y|X1 , . . . , Xn−1 , Xn = i} = P {Y = y|Xn = i}.

84

1 Markov Chains

1.23 Exercises The first four exercises deal with conditional probabilities and expectations that are reviewed in Section 1.22. Exercise 1. The length of time in microseconds for a computer to perform a task of type i has a geometric distribution (1 − pi )pn−1 , n ≥ 1, with mean i 1/(1 − pi ), for i = 1, . . . , . Also, the type of task to be worked on is a random variable X, where p(i) = P {X = i} is known for i = 1, . . . , m. Under these assumptions, the time T to perform a task has the conditional probability measure P {T = n|X = i} = (1 − pi )pn−1 . Find expressions for E[T |X = i], i P {T = n} and ET . Exercise 2. Suppose X1 , X2 , . . . are real-valued i.i.d. random variables with mean μ and variance σ 2 that represent times to process jobs. The number of jobs to be processed in a week is a nonnegative integer-valued random variable N that is independent of the Xn . Prove that the mean and variance of the time to do the N jobs are E[

N 

Xn ] = μE[N ],

N Var[ n=1 Xn ] = E[N ]σ 2 + μ2 Var[N ].

n=1

Exercise 3. Poisson Random Variable with a Randomized Mean. The number of sales of a product in a period of length t has a Poisson probability measure pλt (n) = (λt)n e−λt /n!, n ≥ 0, with mean λt. This is true when the sales over time occur according to a Poisson process with rate λ. Consider a variation of this setting in which the rate λ is a random variable Λ that may depend on the economic environment and other factors, but it is not affected by the sales. In this case, the number of sales N in the period of length t has the properties P {N = n|Λ = λ} = pλt (n),

E[N |Λ = λ] = λt.

Letting FΛ (λ) denote the distribution of Λ, show that  E[N ] = tE[Λ], P {N = n} = pλt (n)FΛ (dλ). R+

Exercise 4. Continuation. In the context of the preceding exercise, suppose therevenue from each sale is r, and the cost associated with making n sales n is k=1 Yk , where Yk is the cost to make the kth sale. Assume Y1 , Y2 , . . . are i.i.d. with mean μ and variance σ 2 , and the Yk are independent of Λ and the number of sales N . The net revenue from the N sales is ZN =

N 

(r − Yk ).

k=1

1.23 Exercises

85

Show that the mean and variance of this revenue are E[ZN ] = tE[Λ](r − μ),

Var[ZN ] = tE[Λ][σ 2 + (r − μ)2 ].

You can use the results in Exercise 2. Recall that the mean and variance of the Poisson distribution pλt are both λt. Exercise 5. For a nonnegative integer-valued random variable N , prove E[N ] =

∞ 

P {N > n}.

n=0

Exercise 6. Geometric Sojourn Times in a State. For a Markov chain Xn with pii > 0, the distribution of the sojourn time in state i is Pi {τi = n}, where τi = min{n ≥ 1 : Xn = i}. Show that Pi {τi = n} = (1 − pii )pn−1 ii ,

n ≥ 1,

which is a geometric distribution with parameter 1 − pii . Use Exercise 5 to show E[τi ] = 1/(1 − pii ). Exercise 7. Geometric Memoryless Property. Suppose X is a random variable with values in {1, 2, . . .}. Show that X has a geometric distribution if and only if it satisfies the memoryless property P {X > n + 1|X > n} = P {X > 1},

n ≥ 0.

Hint: What is the unique solution of f (n + 1) = f (1)f (n), n ≥ 0? This memoryless property is analogous to the memoryless property of exponential random variables in Exercise 1 in Chapter 3. Exercise 8. Perishable Inventory or Perishable Service Model. Quantities of a perishable resource arrive to a system in discrete time periods to satisfy demands. The resource is only available for a single time period, but unsatisfied demands are backlogged. For instance the resource might be food, blood or human organs that perish after a certain time period, or cargo space on an airline that disappears when the airline departs. This system could also be viewed as a queueing model in which potential services are occasionally available, but they are unused when there are no units waiting for services. Let Un denote the quantity of the resource that is demanded in period n, and let Vn denote the quantity of the resource that arrives. Assume the pairs (Un , Vn ), for n ≥ 1, are i.i.d. nonnegative integer-valued random variables. Let Xn denote the quantity of the resource that is backlogged at time n. Show that Xn is an input-output Markov chain process as in Example 23. Specify its transition probabilities when U1 and V1 are independent Poisson random variables with respective means α and β.

86

1 Markov Chains

Exercise 9. Moran Storage Model. A reservoir with capacity c receives inputs V1 , V2 , . . . in discrete time periods, where Vn are i.i.d. nonnegative integervalued random variables. In a period when the reservoir has room for y additional units (the reservoir level is c − y), and v units of input occur, then (v − y)+ units of the input are discarded. In addition, the reservoir releases a non-random quantity of u units of water in each time period provided the reservoir level exceeds u, otherwise it releases all the water in the dam. Let Xn denote the amount of water in the reservoir at (the end of) time period n. Justify that Xn is a reflected random walk Markov chain and express its transition probabilities as a function of the distribution qn = P {V1 = n}. Exercise 10. Two-State Markov Chain. Consider a machine (or a production system) that alternates, in discrete time, between being in operation (state 1) or being down for repair or reloading (state 2). The successive durations of time during which the machine is in operation are i.i.d. and the successive time durations the machine is down are i.i.d. and independent of the operation times. Let Xn denote the state of the machine at time n. Determine the type of distributions for the operation times and down times in order for Xn to be a Markov chain on S = {1, 2} with transition matrix   1−a a P= , b 1−b where all the entries are positive. Specify the meaning of the probability a, and specify the relation between a and the mean of a typical machine operation time. Show that     (1 − a − b)n a −a 1 ba n P = + . −b b a+b b a a+b Find P {Xn = 1|X0 = 1}, for n ≥ 1. Establish that the chain is ergodic and its stationary distribution is π1 = b/(a+b) and π2 = a/(a+b). This two-state Markov chain can be used to model a variety of situations. Describe one. Exercise 11. Uniform Representation and Simulation of a Random Variable. For a distribution function F , its left-continuous inverse is F −1 (u) = inf{x : F (x) ≥ u}. Show that if U has a uniform distribution on the interval [0, 1], then the random variable X = F −1 (U ) has the distribution F . (Use the fact that F (x) ≥ u if and only if F −1 (u) ≤ x.) Because of this property, one can generate a random sample of values from F by using the uniform distribution. First, one generates a random sample U1 = u1 , . . . , Un = un from the uniform distribution on [0, 1]. Then the values x1 = F −1 (u1 ), . . . , xn = F −1 (un ) form a random sample from F .

1.23 Exercises

87

Exercise 12. Non-homogeneous Markov Chains. A stochastic process {Xn : n ≥ 0} on S is a non-homogeneous Markov chain if it satisfies the Markov property P {Xn+1 = j|X0 , . . . , Xn−1 , Xn = i} = P {Xn+1 = j|Xn = i} = pij (n), where the transition probabilities pij (n) are functions of n. Show that the two-dimensional process Yn = (Xn , n) is a time-homogeneous Markov chain and specify its transition probabilities. Because of this formulation, some properties of non-homogeneous Markov chains (e.g., Markov decision problems) can be obtained from the theory of homogeneous Markov chains. Exercise 13. Continuation. Suppose {Xn : n ≥ 0} is a stochastic process on S of the form Xn+1 = fn+1 (Xn , Yn+1 ), n ≥ 0, where Y1 , Y2 , . . . are independent S  -valued random variables that are independent of X0 , and fn : S × S  → S. Show that Xn is a non-time-homogeneous Markov chain with transition probabilities pij (n) = P {fn (i, Yn ) = j}. Exercise 14. Unit-Demand Inventory System. Consider an inventory or input-output system in discrete time, where Xn denotes the quantity of items in the system at the beginning of the nth period. At the beginning of each period, the inventory decreases by one unit provided the inventory level is positive, and otherwise the inventory remains at 0 until the end of the period. At the end of the nth period, the inventory is replenished by an amount Vn , where Vn are i.i.d. with distribution pi = P {V1 = i}, i ≥ 0. Under these assumptions Xn+1 = (Xn − 1 + Vn) if Xn > 0 and Xn = Vn if Xn = 0. Justify that Xn is a Markov chain and specify its matrix of transition probabilities. Is this Markov chain a special case of another one in this chapter? Exercise 15. Continuation: Unit-Supply Inventory System. A dual of the model in the preceding exercise is an inventory system, where in each period, the inventory is replenished by one unit and it decreases by Un (if possible). Then the inventory at the beginning of period n+1 is Xn+1 = (Xn −Un +1)+ . Assume Un are i.i.d. with distribution pi = P {U1 = i}, i ≥ 0. Justify that Xn is a Markov chain and specify its matrix of transition probabilities. Exercise 16. At a dock where trucks are unloaded one-at-a-time, it was observed that a small truck was followed by a large truck 10% of the time, and a large truck was followed by a small truck 60% of the time. Define a Markov chain model for representing the type of truck being unloaded and find the percentage of large trucks that are unloaded. Suppose the times to unload the small trucks are i.i.d. with mean μ and the times to unload the large trucks are i.i.d. with mean μ . Find the percentage of unloading time devoted to large trucks.

88

1 Markov Chains

Exercise 17. Extreme-value Process. Suppose Yn are i.i.d. integer-valued random variables with distribution pk = P {Y1 = k}, where this probability is positive for some k > 0. Define X0 = 0 and Xn = max{Y1 , . . . , Yn }. Let τ0 = 0 and τn+1 = min{m > τn |Xm > Xτn }, n ≥ 0. Then Xn = Xτn is the nth record value. Justify that Xn and Xn are Markov chains and specify their transition probabilities. In addition, classify their states. d

Exercise 18. The equality in distribution X = Y of random variables is equivalent to E[f (X)] = E[f (Y )] for any nonnegative function f on the set of values of the variables. This equivalence also holds for stochastic processes X and Y . Analogous equivalences hold for conditional expectations. In particular, for discrete random variables Y , Y ∗ , Z and Z ∗ in S,show that the following statements are equivalent: (a) P {Y ∈ B|Z} = P {Y ∗ ∈ B|Z ∗ }, B ∈ S (b) E[f (Y )|Z] = E[f (Y ∗ )|Z ∗ ], f : S ∞ → R. These statements are like (1.17) and (1.18). Exercise 19. Reflected Random Walks. Show by induction or substitution that the solution to the recursion Xn = a∨[b∧(Xn−1 +Vn −Un )] in Example 23 is given by (1.10). Exercise 20. Let Y1 , . . . , Yn+1 be random variables (possibly in a general space) such that Y1 , . . . , Yn are i.i.d. Show that if Yn+1 is independent of d Y1 , . . . , Yn and Yn+1 = Y1 , then Y1 , . . . , Yn+1 are i.i.d. Exercise 21. Suppose {Xn : n ≥ 0} is an S-valued stochastic process of the form Xn+1 = f (Xn , Yn+1 ), n ≥ 0, where Yn are S  -valued random variables such that, for each n ≥ 0, Yn+1 is conditionally independent of Hn ≡ {Xk−1 , Yk , k ≤ n}, and Gi (A) = P {Yn+1 ∈ A|Xn = i} is independent of n. Show that Xn is a Markov chain and specify its transition probabilities. Exercise 22. Continuation. Consider the processes Xn and Yn in the preceding exercise with the history Hn changed to Hn ≡ {Xk−1 , Yk+1 , k ≤ n− 1, Xn−1 } (it no longer contains Yn ). Is Xn is a Markov chain? If so specify its transition probabilities. Answer this under the additional assumption that Gi,i (A) = P {Yn+1 ∈ A|Xn = i, Yn = i } is independent of n. Exercise 23. Stopping Time Criteria. Let τ be a random variable that takes values in {0, 1, . . . , ∞}. Show that the following are equivalent statements: (a) τ is a stopping time for a Markov chain Xn . (b) {τ > n} is determined by X0 , . . . , Xn for any finite n. (c) {τ ≤ n} is determined by X0 , . . . , Xn for any finite n. (d) 1(τ = n) = hn (X0 , . . . , Xn ) for some hn : S n+1 → {0, 1}. Exercise 24. Communication is an Equivalence Relation. Show that i → j in S if and only if there are states i1 , . . . , in ∈ S such that pi,i1 pi1 ,i2 · · · pin ,j > 0. Show that the communication relation ↔ is an equivalence relation in that

1.23 Exercises

89

it satisfies the following properties. Reflexive: i ↔ i, i ∈ S. Symmetric: i ↔ j if and only if j ↔ i, i, j ∈ S. Transitive: If i ↔ j and j ↔ k, then i ↔ k, i, j, k ∈ S. Exercise 25. Show that, for an irreducible class C, each of its states has the same period. Hint: Suppose i, j ∈ C have periods di and dj . Let m and n be n the smallest integers such that a = pm ji pij > 0. Then use (1.27) along with m+n n m pii ≥ pij pji = a to prove di ≤ dj . Then reverse the roles of i and j. Exercise 26. Consider a Markov chain on S = {1, 2, . . . , 10} whose transition matrix has the following form, where means a positive probability. ⎤ ⎡

00 000000 ⎢0 0 0 0 0 0 0 0⎥ ⎥ ⎢ ⎢0 0 0 0 0 0 0 ⎥ ⎥ ⎢ ⎢ 0 0 0 0 0 0 0 0⎥ ⎥ ⎢ ⎢0 0 0 0 0⎥ ⎥ P=⎢ ⎢0 0 0 0 0 0 0 0⎥ ⎥ ⎢ ⎢0 0 0 0 0⎥ ⎥ ⎢ ⎢0 0 0 0 0 0 0 0 0⎥ ⎥ ⎢ ⎣0 0 0 0 0 0 0 0 ⎦ 00 00000

Draw the transition graph of this chain and identify its transient and closed irreducible sets. Display a more informative transition matrix for the chain by reordering the states according to Theorem 40. Exercise 27. Suppose Xn and Xn are independent ergodic Markov chains on the spaces S and S  with transition probabilities pij and pij and stationary distributions π and π  , respectively. Show that Zn = (Xn , Xn ) is an ergodic Markov chain on S × S  with transition probabilities pij,k = pik pj , and its stationary distribution is πij = πi πj . Exercise 28. Let Xn be an ergodic Markov chain on S with transition prob˜ n = (Xn , . . . , Xn+ ) abilities pij and stationary distribution π. Show that X is an ergodic Markov chain on S +1 with stationary distribution π(i) = πi0 pi0 ,i1 · · · pi−1 ,i . Exercise 29. For an irreducible Markov chain Xn with period d, show that Xn∗ = Xnd is an irreducible aperiodic Markov chain and specify its transition probabilities. Show that τi = min{n ≥ 1 : Xn = i} is a multiple of τi∗ = min{n ≥ 1 : Xn∗ = i}. Use Theorem 110 to prove that if i is positive recurrent, then limn→∞ pnd ij = d/Ei [τi ]. Use this result to prove (1.80).

90

1 Markov Chains

Exercise 30. Infinite Number of Stationary Distributions. Consider a Markov chain, as in Theorem 40, whose state space S is the union of closed irreducible recurrent classes C1 , . . . , Cm with associated transition matrices P1 , . . . , Pm , respectively. Suppose πi , i ∈ C , is a stationary distribution for P (π  P = π  ), = 1, . . . , m. Show that, if α is an initial distribution for X0 , then m πi = =1 αi πi 1(i ∈ C ) is a stationary distribution for the chain. In fact, all stationary distributions can be obtained this way. Exercise 31. Using the dominated convergence for sums, show that a limiting distribution for a Markov chain (which need not be ergodic) is a stationary distribution. Give an example of a non-ergodic Markov chain with a stationary distribution that is not a limiting distribution. Exercise 32. Success Runs. Let Xn denote the success runs Markov chain in Example 19. Find an expression for the probability f00 of ever reaching state 0 starting from 0. Show that the chain is irreducible, and it is recurrent if and only if i (1 − pi ) = ∞. In that case, show that an invariant measure  for the chain is ηi = i−1 j=0 pj , i ≥ 0. Exercise 33. Suppose Y1 , Y2 , . . . are i.i.d. random variables with a finite −1 Y1 → 0, and that n−1 Yn → 0, a.s. as n → ∞. Hint: mean μ. Show n that n  n−1 use Yn = m=1 Ym − m=1 Ym . Next, suppose for some positive integer k that Ym , m > k are i.i.d. with mean μ, and Y1 , . . . , Yk are general random variables with finite means (they need not be independent or independent of n the other Ym ’s). Show that n−1 m=1 Ym → μ a.s. as n → ∞. (This result also holds for a random k). Exercise 34. Limits of Expectations. Let Xn be an ergodic Markov chain on S with stationary distribution π. Show that, for B ⊂ S +1 ,  p(i), lim P {(Xn , . . . , Xn+ ) ∈ B} = n→∞

i∈B

where i = (i0 , . . . , i ) and p(i) = πi0 pi0 ,i1 · · · pi−1 ,i . Show that, for bounded f : S +1 → R,  f (i)p(i). lim E[f (Xn , . . . , Xn+ )] = n→∞

i∈S +1

Exercise 35. Brand Switching. A Markov chain model for approximating the sales of several brands of a particular product that customers continually purchase (e.g., shampoo, soda, bread) is as follows. Suppose there are four brands of a product labeled 1, 2, 3, 4, and the successive brands that a typical customer chooses to purchase over time is a Markov chain Xn with transition matrix ⎤ ⎡ .7 .1 .1 .1 ⎢.2 .4 .2 .2⎥ ⎥ P=⎢ ⎣ 0 .5 .4 .1⎦ 0 0 .2 .8

1.23 Exercises

91

For instance, upon buying brand 3, a customer’s next purchase is brand 2, 3 or 4 with respective probabilities .5, .4, .1, independent of past purchases. Show that the fraction of sales over time of brands 1 and 2 are 6/67 and 9/67, respectively. Suppose the profits from the four brands are $10, $12, $15, $16 per sale. Show that the average profit from the four brands is $984/67 ≈ $14.70 per sale, and the average profit from only brands 1 and 3 combined is $300/67 ≈ $4.48 per sale (of all brands). Is brand 4 more profitable per sale than brands 1 and 2 combined? Exercise 36. Setup Costs in Processing Systems. As in Example 71, suppose the ergodic Markov chain Xn with stationary distribution π denotes a sequence of jobs a system processes (labeled by job type). Assume that whenever the system switches from processing a type i job to processing a type j job, the system incurs a setup cost v(i, j) measured in time or Then money. n the average setup cost per job processed is γ = limn→∞ n−1 m=1 v(Xn−1 , Xn ). Find an expression for this average. Find an expression for the average setup cost γ(i) per job  processed for switching from a type i job to another type of job, where γ = i γ(i). Exercise 37. Renewal Age Process. Let Xn denote the renewal age process in Example 20, where the renewals times are i.i.d. machine lifetimes with distribution F and mean μ, and F (1) > F (0) = 0. Prove that Xn is ergodic and that its stationary distribution is πi = [1 − F (i)]/μ. Suppose there is a cost C each time the machine is replaced and a cost hi for each period in which the age is i. Give an expression for the average cost for the chain. Exercise 38. Continuation. Jobs are processed one at a time and the processing times are nonnegative integer-valued i.i.d. random variables with distribution F and mean μ. As in Example 20, let Xn denote the age of the job being processed at time n, and let Xn denote the time needed to finish the job in process at time n. Show that Xn is an ergodic Markov chain and that its stationary distribution is the same as that for Xn . Exercise 39. Let Xn denote the number of visitors at a certain web site (e.g., a food recipe site) at time n. At each time period, each visitor at the site independently leaves with probability p, and the number of new visitors that enter the site has a Poisson distribution with mean λ (independent of everything else). Show that Xn is a Markov chain and determine its transition probabilities. Find a condition on p and λ under which the chain is ergodic. Show that its stationary distribution is a Poisson distribution and specify its mean. Exercise 40. Closed Network Process. Show that the network chain Xn in Section 1.15 is irreducible or aperiodic or reversible if and only if the routing probabilities pij have these respective properties.

92

1 Markov Chains

Exercise 41. Open Network Process. For the open network process described in Theorem 90, show that π given by (1.61) with c = 1 is an invariant measure for the network chain. Start with 

π(y)p(y, x) =

y

m m  

π(x + ei − ej )p(x + ei − ej , x)1(xj > 0)

i=1 j=1

+

m 

π(x + ei )p(x + ei , x) +

i=1

m 

π(x − ej )p(x − ej , x)1(xj > 0).

j=1

Exercise 42. Doubly Stochastic Chains. A Markov chain is doubly stochastic if its transition probabilities pij satisfy i pij = 1, for each j. Show that a doubly stochastic Markov chain on a state space with m states has a stationary distribution πi = 1/m. Exercise 43. Ehrenfest Chain. Consider a system in which ν particles move in discrete time between two locations 1 and 2. At each time period, a particle chosen at random moves from its location to the other one. Let Xn denote the number of particles in location 1 at time n. Justify that Xn is an irreducible Markov chain with period 2 and show that its stationary distribution is a binomial distribution. Exercise 44. Random Walk on a Graph. Consider a connected graph with m nodes, and let Gi denote the set of nodes adjacent to node i. Whenever the particle is at node i, it moves to any j ∈ Gi with equal probability (i.e., 1/|Gi |). Let Xn denote the location of the particle at time n. Show that Xn is a Markov chain and specify its transition probabilities. Is the chain irreducible? aperiodic? recurrent? Assuming the graph is such that the chain is ergodic, show that its stationary distribution is πi = |Gi |/2|S|. Exercise 45. Expected Length of Gambling Game. In the Gambler’s ruin model in Examples 4 and 44, let vi denote the expected length of time until the gambler’s fortune Xn reaches 0 or m, starting with the fortune i. Verify vi =

1 − (q/p)i  m i − , q − p m 1 − (q/p)m

if p = 1/2,

and vi = i(m − i), if p = 1/2. Find an expression for the mean number of times vˆi the gambler’s fortune equals i before it reaches 0 or m, when X0 = i. Exercise 46. Continuation. Consider the gambler’s random walk in the preceding exercise in which there is no upper bound m at which the gambler stops playing. That is, the fortune Xn moves in S = {0, 1, . . .} until it is absorbed at 0 or remains positive forever. Let γi denote the probability of being absorbed at 0 when X0 = i. Show that γi = 1 if q ≥ p, and that the mean length of the game starting at i is i/(q − p) if q > p. Thus a gambler is bound to loose in a series of gambles if the odds are not favorable.

1.23 Exercises

93

In addition,show that γi = (q/p)i if p > q. Use the facts that γi must be bounded and ∞ i=1 γi = 1. Exercise 47. Pattern Occurrences in Bernoulli Process. Let Yn denote a Bernoulli process with P {Yn = 1} = p and P {Yn = 0} = q = 1 − p. Suppose one is interested in the occurrence of a three-letter pattern in the sequence Yn . Let Xn = Yn−2 Yn−1 Yn , for n ≥ 2. Show that Xn is an ergodic Markov chain and find its stationary distribution. In particular, consider the occurrence of the three-letter pattern 101 (various pattern occurrences like this are of interest in DNA sequences). Let τ = inf{n ≥ 2 : Xn = 101}, and define Gk (s) = E[sτ |X0 ∈ Ak ], where A0 = {000, 100},

A1 = {001, 011, 111},

A2 = {110, 010},

A3 = {101}.

Find an expression for G0 (s). Begin by justifying that G0 (s) = s[pG1 (s) + qG0 (s)],

G1 (s) = s[qG2 (s) + pG0 (s)]

and G2 (s) = s[p + qG0 (s)], and then solve for G0 (s). Find a tractable expression for E[τ |X0 ∈ A0 ]. Exercise 48. Moving Averages. Let Xn be an ergodic Markov chain with stationary distribution π. Consider the moving average process Yn =

 

am f (Xn+m ),

for k,  ≥ 0, am ∈ R, and f : S → R.

m=−k

Specify the limiting distribution of Yn for k = 2 = m, and give a formula for limn→∞ E[Yn ]. Exercise 49. Exhaustive Parallel Processing. A set of m jobs are processed in parallel until they are all completed. At the completion time, another m jobs instantaneously enter the system and are processed similarly. This is repeated indefinitely. Assume the times to process jobs are independent with a geometric distribution with mean 1/p. Let Xn denote the number of jobs being processed at time n. Show that Xn is a Markov chain on S = {1, . . . , m} and specify its transition probabilities (whenever all the jobs in the system are completed simultaneously, the next state is m). Show that the chain is ergodic and that an invariant distribution can be computed by the recursive formula

k q m−k  m−k−j j πm−k = p , π m−k−j 1 − q m−k j=1 j for k = 1, . . . , m starting with πm = 1, where q = 1−p. Use this formula (and successive substitutions) to derive an explicit expression for the stationary distribution when m = 3.

94

1 Markov Chains

Exercise 50. M/M/1 Queueing System. Suppose Xn is the queue-length process in Example 21, where p is the probability of an arrival, and q is the probability of a service completion at any discrete time. Show that an invariant measure for this Markov chain is πi = ρi , i ≥ 0, where ρ = p(1 − q)/[q(1 − p)]. Prove that the chain is ergodic if and only if p < q, and in this case πi = (1−ρ)ρi , i ≥ 0 is its stationary distribution. Is this chain reversible? Suppose there is a cost s per unit time for serving a customer and a cost h per unit time of holding a customer in the system. Show that the average cost per unit time is ρ[s + h/(1 − ρ)]. Now, assume there is a reward R for serving a customer. Show that the average net reward is ρq(1 − p)R − ρ[s + h/(1 − ρ)]. Exercise 51. Continuation. Consider the M/M/1 queueing model in the preceding exercise with p < q. Example 29 pointed out that the regenerative property of Markov chains in Proposition 67 implies that thetimes between empty epochs ξn are i.i.d. and the durations of the busy periods βn are i.i.d. for n ≥ 2. Find an expression for E0 [ξ1 ] and show that E0 [β1 ] = q(1 − p)/(q − p) − 1/p. Verify that E0 [β1 ] tends to ∞ as p ↑ q (the traffic is heavy), and it tends to 1 as p ↓ 0 (the traffic is light). Exercise 52. Sharing a Buffer. Let Xn and Xn be independent M/M/1 queue-length processes as in the preceding exercise with parameters, ρ = p(1 − q)/[q(1 − p)],

ρ = p (1 − q  )/[q  (1 − p )].

Now, consider the modification in which these queues must share a common buffer with capacity C so that a new arrival (in either system) is blocked from entering and disregarded when the buffer is full. Let Yn and Yn denote the resulting queue-length processes (which are now dependent). Show that Zn = (Yn , Yn ) is a Markov chain on the state space S˜ = {(i, i ) : 0 ≤ i + i ≤ C} and specify its transition probabilities. Show that this chain is reversible  and its stationary distribution has the form π(i,i ) = cρi (ρ )i . (These results follow from the definition of reversibility and a little algebra. Chapter 4 covers related models based on the following properties. (a) If Xn and Xn are independent reversible Markov chains, then (Xn , Xn ) is also a reversible Markov chain. (b) If Xn is a reversible Markov chain on S with respect to π, then its restriction to a subset S˜ ⊂ S is also reversible with ˜ respect to π restricted to S.) Exercise 53. Let S0 = 0 and Sn = Y1 + · · · Yn , for n ≥ 1, where Yk are i.i.d. nonnegative integer valued random variables. Let Xn denote the unit in the integer expansion of Sn (so Xn equals Sn modulo 10; if Sn = 321, then Xn = 1). Show that Xn is a Markov chain and specify its transition probabilities. Under what conditions is Xn irreducible. Assuming it is, find its stationary distribution. (Similar properties hold when Xn equals Sn modulo 10k for some k ≥ 1.)

1.23 Exercises

95

Exercise 54. In the context of Example 71, suppose the cost of processing a type-i job in time v is c(i, v), where c : S × R+ → R. Then c(Xn , Vn ) is the cost of processing job Xn . Find an expression for the average cost of n processing jobs, which is limn→∞ n−1 m=1 c(Xm , Vm ). Exercise 55. Random Walk on an Edge-Weighted Graph. Consider a Markov chain Xn on the set of vertices S of a finite graph. Associated with each pair of vertices i and j there is a nonnegative weight wij , which is 0 if (i, j) is not an edge; and the probability of a transition from i to j is directly proportional  to this weight. Therefore, its transition probabilities are pij = wij / k wik . Assume the chain is irreducible. Show that the Markov chain is reversible and its stationary distribution is   wij / wi , i ∈ S. πi = j

k,

Find a formula for the average sojourn time of the chain in a subset A of vertices. Exercise 56. Determine the stationary distribution of the random walk on a circle described in Example 100. Use the property that the balance equations for i = 1,. . . ,  − 1 are the same as those for a standard random walk, and so i πi = π0 k=1 pk−1 /qk for i ≤  − 1. Then solve for π and π0 . Consider the special case pi = p and qi = 1 − p for all i. For what value of p is the random walk reversible? Exercise 57. Show that if a Markov chain is periodic with period greater than 2, then the chain is not reversible. For the random walk in Example 94, specify its periodicity for the following cases: (a) ri = 0 for each i. (b) ri > 0 for some i. Exercise 58. McCabe Library. Consider Example 101 for a library of three books. Compute the product (1.67) for the following path of states (1, 2, 3) → (2, 1, 3) → (2, 3, 1) → (3, 2, 1) → (3, 1, 2). An alternate way of modeling the state of the library is by the vector z = (z1 , . . . , zm ), where zb denotes the location of book b. This z is the inverse of the corresponding state i (which lists how the books are arranged on the shelf); indeed, zik = k and izb = b. Use this one-to-one correspondence between i’s and z’s, to show that the successive values of this shelf variable Zn is a reversible Markov chain. Describe the transition probabilities of Zn in terms of the probabilities pb of the book selections, and show that its stationary distribution is π(z) = c

m  b=1

pzb b ,

z ∈ S.

96

1 Markov Chains

One can make use of the results in Example 101, since Zn = z if and only if the zb th component of the process Xn equals b for each b. Exercise 59. Markov Chain Monte Carlo. In the context of Section 1.18, consider the estimator of μ given by n g(Xm )η(Xm )/γ(Xm ) n , μ ˆn = m=1 m=1 η(Xm )/γ(Xm ) where γ is any fixed positive probability measure (it gives flexibility in implementing the estimation). Show that μ ˆn is a consistent estimator of μ. Suppose the target distribution has the form πi = cη(i), where the normalization constant c is unknown or is toodifficult to compute. Show that a n −1 consistent estimator for c−1 is cˆ−1 n = n m=1 η(Xm )/γ(Xm ). Exercise 60. Star-Like Collage. Consider the Markov chain Xn in Example 107 with the variation that the set S0 may consist of more that one state. In addition to the assumption that the chain restricted to each S0 ∪Sk is ergodic with stationary distribution pki , assume the chain restricted to S0 is ergodic with stationary distribution p0i . Show that Xn is ergodic and its stationary distribution is π0 p0i if i ∈ S0 πi = π0 ck pki if i ∈ Sk for some k,    −1 k −1 and ck = . Also, show that whereπ0 = 1 + k=0 (ck − 1) i∈S0 pi π0 = i∈S0 πi . Exercise 61. Branching Process Properties. Consider the branching process Xn defined by (1.33), with the generalization that X0 is a random variable that is independent of the ξni . Show that E[Xn ] = μn E[X0 ]. For the case μ > 1, show that the extinction probability is lim P {Xn = 0} =

n→∞

∞ 

z i P {X0 = i},

i=1

where z is the unique fixed point of φ(s) in (0, 1). Assuming X0 = 1 and σ 2 = Var[ξn1 ], show that Var[Xn+1 ] = μ2 Var[Xn ] + μn σ 2 , Var[Xn ] = σ 2 (μ2n−2 + μ2n−3 + · · · μn−1 ),

n ≥ 1.

Exercise 62. Geometric Branching Process. Consider a branching process in which each item produces k items with probability pk = pq k , k ≥ 0, where q = 1 − p. Determine the extinction probability z, and specify conditions under which z = 1 and z < 1. Show by induction that E[sXn ] =

p[an − qsan−1 ] , an+1 − qsan

when p = q,

1.23 Exercises

97

where an = q n − pn . Let p → 1/2 in this expression to obtain an expression for E[sXn ] when p = q = 1/2. Use this last generating function to obtain P {Xn = 0} = n/(n + 1),

when p = q.

Exercise 63. Consider a branching process in which each item produces k items with probability 1/4, where 0 ≤ k ≤ 3. First show that the extinction probability z is in the interval [.30, .50]. Then compute the extinction probability of the process by the procedure in Remark 49 for ε = .001. Alternatively, the probability can be obtained – you need not do this – by solving the cubic equation s = (1/4)(s3 + s2 + s + 1). Exercise 64. Total Progeny in Branching. Let Xn denote the branching process defined by  (1.33), where X0 = 1. The total progeny in this branching ∞ process is Y = n=0 Xn . Of course, P {Y < ∞} = z (the extinction probability). Show that when μ < 1, Var[Y ] = Var[X1 ]/(1 − μ)3 .

E[Y ] = 1/(1 − μ),

Prove that the generating function G(s) = E[sY ] satisfies G(s) = sφ(G(s)). It is also true (you need not prove it) that G(s) is the unique solution of γ = sφ(γ), γ ∈ (0, z]. Using this result, find G(s) for the geometric branching in Exercise 62. Exercise 65. Estimation of Transition Probabilities. Suppose one wants to estimate the transition probabilities of an ergodic Markov chain Xn based on observing X0 , . . . , Xn , where the number of states is known. An estimator of its transition probability pij is n 1(X = i, Xk = j) n k−1 , i, j ∈ S. pij (n) = k=1 1(X k−1 = i) k=1 This is the portion of the times the chain moves to j upon leaving i. Show that pij (n) is a consistent estimator of pij in that pij (n) → pij a.s. as n → ∞. Exercise 66. Suppose Xn is an ergodic Markov chain that is stationary. Show that the following statements are equivalent. (a) Xn is reversible. d

(b) (Xn , Xn+1 ) = (Xn+1 , Xn ),

n ≥ 1.

d

(c) (X1 , X2 , . . . , Xn ) = (Xn , . . . , X2 , X1 ),

n ≥ 1.

Exercise 67. Time-Reversible Chains. A Markov chain Xn is reversible in time if, for any ν ≥ 1, d

(Xν , Xν−1 , . . . , Xν−n ) = (X0 , X1 , . . . , Xn ),

n ≥ ν.

That is, the chain viewed in reverse time beginning at ν (like viewing a video tape in reverse) is equal in distribution to viewing the chain in the forward

98

1 Markov Chains

direction. Show that a Markov chain is time-reversible if and only if it is stationary and reversible (in the usual sense). Exercise 68. Fork-Join System. Prove that (1.85) is also a necessary condition for the fork-join Markov chain Xn in Example 116 to be positive recurrent. Also, discuss why Foster’s criteriondoes not work to prove positive recurrence with the linear function v(x) = m i=1 xi .

Chapter 2

Renewal and Regenerative Processes

Renewal and regenerative processes are models of stochastic phenomena in which an event (or combination of events) occurs repeatedly over time, and the times between occurrences are i.i.d. Models of such phenomena typically focus on determining limiting averages for costs or other system parameters, or establishing whether certain probabilities or expected values for a system converge over time, and evaluating their limits. The chapter begins with elementary properties of renewal processes, including several strong laws of large numbers for renewal and related stochastic processes. The next part of the chapter covers Blackwell’s renewal theorem, and an equivalent key renewal theorem. These results are important tools for characterizing the limiting behavior of probabilities and expectations of stochastic processes. We present strong laws of large numbers and central limit theorems for Markov chains and regenerative processes in terms of a process with regenerative increments (which is essentially a random walk with auxiliary paths). The rest of the chapter is devoted to studying regenerative processes (including ergodic Markov chains), processes with regenerative increments, terminating renewal processes, and stationary renewal processes.

2.1 Renewal Processes This section introduces renewal processes and presents several examples. The discussion covers Poisson processes and renewal processes that are “embedded” in stochastic processes. We begin with notation and terminology for point processes that we use in later chapters as well. Suppose 0 ≤ T1 ≤ T2 ≤ . . . are finite random times at which a certain event occurs. The number of the times Tn in the interval (0, t] is ∞  1(Tn ≤ t), t ≥ 0. N (t) = n=1

R. Serfozo, Basics of Applied Stochastic Processes, Probability and its Applications. c Springer-Verlag Berlin Heidelberg 2009 

99

100

2 Renewal and Regenerative Processes

We assume this counting process is finite valued for each t, which is equivalent to Tn → ∞ a.s. as n → ∞. More generally, we will consider Tn as points (or locations) in R+ (e.g., in time, or a physical or virtual space) with a certain property, and N (t) is the number of points in [0, t]. The process {N (t) : t ≥ 0}, denoted by N (t), is a point process on R+ . The Tn are its occurrence times (or point locations). The point process N (t) is simple if its occurrence times are distinct: 0 < T1 < T2 < . . . a.s. (there is at most one occurrence at any instant). Definition 1. A simple point process N (t) is a renewal process if the interoccurrence times ξn = Tn − Tn−1 , for n ≥ 1, are independent with a common distribution F , where F (0) = 0 and T0 = 0. The Tn are called renewal times, referring to the independent or renewed stochastic information at these times. The ξn are the inter-renewal times, and N (t) is the number of renewals in (0, t]. Examples of renewal processes include the random times at which: customers enter a queue for service, insurance claims are filed, accidents or emergencies happen, or a stochastic process enters a special state of interest. In addition, Tn might be the location of the nth vehicle on a highway, or the location of the nth flaw along a pipeline or cable, or the cumulative quantity of a product processed in n production cycles. A discrete-time renewal process is one whose renewal times Tn are integer-valued. Such processes are used for modeling systems in discrete time, or for modeling sequential phenomena such as the occurrence of a certain character (or special data packet) in a string of characters (or packets), such as in DNA sequences. To define a renewal process for any context, one only has to specify a distribution F with F (0) = 0 for the inter-renewal times. The F in turn defines the other random variables. More formally, there exists a probability space and independent random variables ξ1 , ξ2 , . . . defined on it that have the distribution in the Appendix). Then the other quantities  F (see Corollary 6  are Tn = nk=1 ξk and N (t) = ∞ n=1 1(Tn ≤ t), where Tn → ∞ a.s. by the strong law of large numbers (Theorem 72 in Chapter 1). Here are two illustrations. Example 2. Scheduled Maintenance. An automobile is lubricated when its owner has driven it L miles or every M days, whichever comes first. Let N (t) denote the number of lubrications up to time t. Suppose the numbers of miles driven in disjoint time periods are independent, and the number of miles in any time interval has the same distribution, regardless of where the interval begins. Then it is reasonable that N (t) is a renewal process. The inter-renewal distribution is F (t) = P {τ ∧ M ≤ t}, where τ denotes the time to accumulate L miles on the automobile. This scheduled maintenance model applies to many types of systems where maintenance is performed when the system usage exceeds a certain level L or when a time M has elapsed. For instance, in reliability theory, the Age

2.1 Renewal Processes

101

Replacement model of components or systems, replaces a component with lifetime τ if it fails or reaches a certain age M (see Exercise 19). Example 3. Service Times. An operator in a call center answers calls one at a time. The calls are independent and homogeneous in that the callers, the call durations, and the nature of the calls are independent and homogeneous. Also, the time needed to process a typical call (which may include post-call processing) has a distribution F . Then one would be justified in modeling the number of calls N (t) that the operator can process in time t as a renewal process. The time scale here refers to the time that the operator is actually working; it is not the real time scale that includes intervals with no calls, operator work-breaks, etc. Elementary properties of a renewal process N (t) with inter-renewal distribution F are as follows. The times Tn are related to the counts N (t) by {N (t) ≥ n} = {Tn ≤ t}, TN (t) ≤ t < TN (t)+1 . In addition, N (Tn ) = n and N (t) = max{n : Tn ≤ t} = min{n : Tn+1 > t}. These relations (which also hold for simple point processes) are used to derive properties of N (t) in terms of Tn , and vice versa. n We have a good understanding of Tn = k=1 ξk , since it is a sum of independent variables with distribution F . In particular, by properties of convolutions of distributions (see the Appendix), we know that P {Tn ≤ t} = F n (t), which is the n-fold convolution of F . Then {N (t) ≥ n} = {Tn ≤ t} yields P {N (t) ≤ n} = 1 − F (n+1) (t). ∞ Also, using E[N (t)] = n=1 P {N (t) ≥ n} (see Exercise 1), we have E[N (t)] =

∞ 

F n (t).

(2.1)

(2.2)

n=1

The following result justifies that this mean and all moments of N (t) are finite. Properties of moment generating functions are in the Appendix. Proposition 4. For each t ≥ 0, the moment generating function E[eαN (t) ] exists for some α in a neighborhood of 0, and hence E[N (t)m ] < ∞, m ≥ 1. Proof. It is clear that if 0 ≤ X ≤ Y and Y has a moment generating function on an interval [0, ε], then so does X. Therefore, to prove the assertion it

102

2 Renewal and Regenerative Processes

suffices to find a random variable larger than N (t) whose moment generating function exists. To this n end, choose x > 0 such that p = P {ξ1 > x} > 0. Consider the sum Sn = k=1 1(ξk > x), which is the number of successes in n independent Bernoulli trials with probability of success p. The number of trials until the mth success is Zm = min{n : Sn = m}. Clearly xSn < Tn , and so N (t) = max{n : Tn ≤ t} ≤ max{n : Sn = t/x} ≤ Z t/x+1 . Now Zm has a negative binomial distribution with parameters m and p, and its moment generating is given in Exercise 2. Thus, Z t/x+1 has a generating function, and hence N (t) has one as well. Furthermore, this existence ensures that all moments of N (t) exist (a basic property of moment generating functions for nonnegative random variables). Keep in mind that the preceding properties of the renewal process N (t) are true for any distribution F with F (0) = 0. When this distribution has a finite mean μ and finite variance σ 2 , the distribution of N (t), for large t, is approximately a normal distribution with mean t/μ and variance tσ 2 /μ3 (this follows by the central limit theorem in Example 67 below). Refined asymptotic approximations for the mean of N (t) are given in Proposition 84. The rest of this section is devoted to examples of renewal processes. The most prominent renewal process is as follows. Example 5. Poisson Process. Suppose the i.i.d. inter-renewal times of the renewal process N (t) have the exponential distribution F (t) = 1 − e−λt with rate λ (its mean is λ−1 ). Then as we will see in the next chapter, N (t) is Poisson process with rate λ. In this case, by properties of convolutions  t e−λx dx. λn xn−1 P {Tn ≤ t} = F n (t) = (n − 1)! 0 This is a gamma distribution with parameters n and λ. Alternatively, P {Tn ≤ t} = 1 −

n−1  k=0

(λt)k −λt e . k!

This is justified by noting that the derivative of the summation equals the integrand (the gamma density) in the preceding integral. Then using the relation {N (t) ≥ n} = {Tn ≤ t}, we arrive at P {N (t) ≤ n} =

n  (λt)k k=0

k!

e−λt .

This is the Poisson distribution with mean E[N (t)] = λt.

2.1 Renewal Processes

103

Poisson processes are very important in the theory and applications of stochastic processes. We will discuss them further in Chapter 3. Note that the discrete-time analogue of a Poisson process is the Bernoulli process described in Exercise 2. Example 6. Delayed Renewal Process. Many applications involve a renewal process N (t) with the slight difference that the first renewal time ξ1 does not have the same distribution as the other ξn , for n ≥ 2. We call N (t) a delayed renewal process. Elementary properties of delayed renewal processes are similar to those for renewal processes with the obvious changes (e.g., if ξ1 has distribution G, then the time Tn of the nth renewal has the distribution G F (n−1) (t)). More important, we will see that many limit theorems for renewal processes apply to delayed renewal processes. In addition to being of interest by themselves, renewal processes play an important role in analyzing more complex stochastic processes. Specifically, as a stochastic process evolves over time, it is natural for some event associated with its realization to occur again and again. When the “embedded” occurrence times of the event are renewal times, they may be useful for gleaning properties about the parent process. Stochastic processes with embedded renewal times include discrete- and continuous-time Markov chains, Markov-Renewal processes and more general regenerative processes (which are introduced in later chapters). The next example describes renewal processes embedded in ergodic Markov chains due the regenerative property of Markov chains. Example 7. Ergodic Markov Chain. Let Xn denote a discrete-time Markov chain on a countable state space that is ergodic (aperiodic, irreducible and positive recurrent). Consider any state i and let 0 < ν1 < ν2 < . . . denote the (discrete) times at which Xn enters state i. Theorem 67 in Chapter 1 showed that the times νn form a discrete-time renewal process when X0 = i. These times form a delayed renewal process when X0 = i. The Bernoulli process in Exercise 2 is a special case. Example 8. Cyclic Renewal Process. Consider a continuous-time stochastic process X(t) that cycles through states 0, 1, . . . , K − 1 in that order, again and again. That is, it starts at X(0) = 0, and its nth state is j if n = mK + j for some m. For instance, in modeling the status of a machine or system, X(t) might be the amount of deterioration of a system, or the number of shocks (or services) it has had, and the system is renewed whenever it ends a sojourn in state K − 1. Assume the sojourn times in the states are independent, and let Fj denote the sojourn time distribution for state j, where Fj (0) = 0. The time for the process X(t) to complete a cycle from state 0 back to 0 has the distribution F = F0 F1 · · · FK−1 . Then it is clear that the times at which X(t) enters state 0 form a renewal process with inter-renewal distribution F . We call X(t) a cyclic renewal process.

104

2 Renewal and Regenerative Processes

There are many other renewal processes embedded in X(t). For instance, the times at which the process enters any fixed state i form a delayed renewal process with the same distribution F . Another more subtle delayed renewal process is the sequence of times at which the processes X(t) bypasses state 0 by jumping from state K − 1 to state 1 (assuming F0 (0) > 0); see Exercise 7. It is quite natural for a single stochastic process to contain several such embedded renewal processes. Example 9. Alternating Renewal Process. An alternating renewal process is a cyclic renewal process with only two states, say 0 and 1. This might be appropriate for indicating whether a system is working (state 1) or not working (state 0), or whether a library book is available or unavailable for use.

2.2 Strong Laws of Large Numbers This section begins our study of the long run behavior of renewal and related stochastic processes. In particular, we present a framework for deriving strong laws of large numbers for a variety of processes. We have already seen SLLNs in Chapter 1 for sums of i.i.d. random variables and for functions of Markov chains.1 Throughout this section, assume that N (t) is a point process on R+ with occurrence times Tn . With no loss in generality, assume that N (t) ↑ ∞ a.s. as t → ∞. The first result says that Tn satisfies a SLLN if and only if N (t) does. Here 1/μ is 0 when μ = ∞. Theorem 10. For a constant μ ≤ ∞ (or random variable), the following statements are equivalent: lim n−1 Tn = μ

n→∞

lim t−1 N (t) = 1/μ

t→∞

a.s. a.s.

(2.3) (2.4)

Proof. Suppose (2.3) holds. We know TN (t) ≤ t < TN (t)+1 . Dividing these terms by N (t) (for large enough t so N (t) > 0), we have TN (t)+1 N (t) + 1 TN (t) t ≤ < . N (t) N (t) N (t) + 1 N (t) Supposition (2.3) along with N (t) ↑ ∞ and (N (t) + 1)/N (t) → 1 ensure that the first and last terms in this display converge to μ. Since t/N (t) is sandwiched between these terms, it must also converge to their limit μ. This proves (2.4). 1

The limit statements here and below are for the a.s. mode of convergence, but we sometimes suppress the term a.s., especially in the proofs.

2.2 Strong Laws of Large Numbers

105

Conversely, suppose (2.4) holds. When N (t) is simple, N (Tn ) = n, and so Tn /n = Tn /N (Tn ) → μ, which proves (2.3). When N (t) is not simple, N (Tn ) ≥ n and (2.3) follows by Exercise 18. Corollary 11. (SLLN for Renewal Processes) If N (t) is a renewal process whose inter-renewal times have a mean μ ≤ ∞, then t−1 N (t) → 1/μ

a.s. as t → ∞.

Proof. This follows by Theorem 10, since the classical SLLN (Theorem 72 in Chapter 1) ensures that n−1 Tn → μ. Example 12. Statistical Estimation. Suppose N (t) is a Poisson process with rate λ, but this rate is not known, and one wants to estimate it. One approach is to observe the process for a fixed time interval of length t and record N (t). Then an estimator for λ is ˆ t = t−1 N (t). λ ˆ t ] = λ. It is also a consistent estimator This estimator is unbiased in that E[λ ˆ t → λ by Corollary 11. Similarly, if N (t) is a renewal process whose since λ inter-renewal distribution has a finite mean μ, then μ ˆt = t/N (t) is a consistent estimator for μ (but it is not unbiased). Of course, if it is practical to observe a fixed number n of renewals (rather than observing over a “fixed” time), then n−1 Tn is an unbiased and consistent estimator of μ. We now present a framework for obtaining limiting averages (or SLLNs) for a variety of stochastic processes. Consider a real-valued stochastic process {Z(t) : t ≥ 0} on the same probability space as the point process N (t). Our interest is in natural conditions under which the limit of its average value t−1 Z(t) exists. For instance, Z(t) might denote a cumulative utility (e.g., cost or reward) associated with a system, and one is interested in the utility per unit time t−1 Z(t) for large t. The following theorem relates the limit of the time average t−1 Z(t) to the limit of the embedded interval average n−1 Z(Tn ). An important quantity is Mn =

sup

|Z(t) − Z(Tn−1 )|,

Tn−1 0, and it is U  (t) = ∞ n=1 f addition, P {N (t) > N (t−)} = 0, t ≥ 0. (2.10) ∞ Proof. The first assertion follows since U (t) = n=0 F n (t), and the derivative of F n (t) is f n (t). The second assertion, which is equivalent to N (t) − N (t−) = 0 a.s., will follow if E[N (t) − N (t−)] = 0. But the last equality is true since, by the monotone convergence theorem (Theorem 13 in the Appendix) and the continuity of U , E[N (t−)] = E[lim N (s)] = lim U (s) − 1 = U (t) − 1 = E[N (t]. s↑t

s↑t

Expression (2.10) tells us that the probability of a renewal at any time is 0, when the inter-renewal times are continuous. Here is an important case. Remark 19. If N (t) is a Poisson process with rate λ, then the probability of a jump at any time t is 0. Some of the results below have slight differences depending on whether the inter-renewal distribution is or is not arithmetic. The distribution F is arithmetic (or periodic) if it is piecewise constant and its points of increase are contained in a set {0, d, 2d, . . .}; the largest d > 0 with this property is

2.3 The Renewal Function

109

the span. In this case, it is clear that the distributions F n and the renewal function U (t) also have this arithmetic property. If F is not arithmetic, we call it non-arithmetic. A distribution with a continuous part is necessarily non-arithmetic. The rest of this chapter makes extensive use of Riemann-Stieltjes integrals; see the review in the Appendix. In particular, the expectation of a function g : R → R on a finite or infinite interval I with respect to F will be expressed as the Riemann-Stieltjes integral 3  g(t)dF (t). I

All the functions in this book like g are assumed to be measurable (see the Appendix); we will not repeat this assumption unless emphasis is needed. Riemann-Stieltjes integrals with respect to U are defined similarly, since U is like a distribution function. A typical integral is   g(t)dU (t) = g(0) + g(t)dU (t). [0,b]

(0,b]

The right-hand side highlights that g(0)U (0) = g(0) is the contribution from the unit jump of U at 0. Since U (t) = 0 for t < 0, we will only consider integrals with respect to U on intervals in R+ . An important property of the renewal function U (t) is that it uniquely determines the distribution F . To see this, we will use Laplace transforms. The Laplace-Stieltjes or simply the Laplace transform of F is defined by  ˆ F (α) = e−αt dF (t), α ≥ 0. R+

A basic property is that the transform Fˆ uniquely determines F and vice ˆ (α) of U (t) is defined similarly. Now, taking versa. The Laplace transform U the Laplace transform of both sides in (2.9), we have ˆ (α) = U

∞ 

n (α) = F"

n=0

∞ 

Fˆ (α)n = 1/(1 − Fˆ (α)).

n=0

This yields the following result. ˆ (α) and Fˆ (α) determine each Proposition 20. The Laplace transforms U ˆ other uniquely by the relation U (α) = 1/(1 − Fˆ (α)). Hence U and F uniquely determine each other. One can sometimes use this result for identifying that a renewal process is of a certain type. For instance, a Poisson process has a renewal function 3

!

! This integral is the usual Riemann integral I g(t)f (t)dt when F has a density f . Also, !b I h(t)dt is written as a h(t)dt when I is (a, b] or [a, b] etc.

110

2 Renewal and Regenerative Processes

U (t) = λt + 1, and so any renewal process with this type of renewal function is a Poisson process. Remark 21. A renewal process N (t), whose inter-renewal times have a finite mean, is a Poisson process with rate λ if and only if E[N (t)] = λt, for t ≥ 0. Other examples of renewal processes with tractable renewal functions are those whose inter-renewal distribution is a convolution or mixture of exponential distributions; see Exercises 6 and 12. Sometimes the Laplace transform ˆ (α) = 1/(1 − Fˆ (α)) can be inverted to determine U (t). Unfortunately, nice U expressions for renewal processes are the exception rather than the rule. In addition to characterizing renewal processes as discussed above, renewal functions arise naturally in expressions for probabilities and expectations of functions associated with renewal processes. Such expressions are the focus of much of this chapter. The next result describes an important family of functions of point processes as well as renewal processes. Expression (2.11) is a special case of Campbell’s formula in the theory of point processes (see Theorem 106 in Chapter 4). Theorem 22. Let N (t) be a simple point process with point locations Tn such that η(t) = E[N (t)] is finite for each t. Then for any function f : R+ → R, E

(t)

N 

  f (Tn ) =

f (s)dη(s),

(0,t]

n=1

t ≥ 0,

(2.11)

provided the integral exists. Moreover, if X1 , X2 , . . . are random variables defined on the same probability space as the process N (t) such that E[Xn |Tn = s] = f (s), independent of n. Then E

(t)

N 



 Xn =

n=1

f (s)dη(s), (0,t]

t ≥ 0,

(2.12)

provided the integral exists. Proof. We will prove (2.11) by a standard approach for proving formulas for integrals. For convenience, denote the equality (2.11) by Σ(f ) = I(f ). First, consider the simple piecewise-constant function f (s) =

m 

ak 1(s ∈ (sk , tk ]),

k=1

for fixed 0 ≤ s1 < t1 < · · · ≤ sm < tm ≤ t. In this case,

2.3 The Renewal Function

111

Σ(f ) = E

m



 ak [N (tk ) − N (sk )]

k=1

=

m 

ak [η(tk ) − η(sk )] = I(f ).

k=1

Next, for any nonnegative function f one can define simple functions fm as above such that fm (s) ↑ f (s) as m → ∞ for each s. For instance, fm (s) = m ∧ (2m f (s)/2m )1(s ∈ [−2m , 2m ]). Then by the monotone convergence theorem (see the Appendix, Theorem 13) and the first part of this proof, Σ(f ) = lim Σ(fm ) = lim I(fm ) = I(f ). m→∞

m→∞

Thus, (2.11) is true for nonnegative f . Finally, (2.11) is true for a general function f , since f (s) = f (s)+ − f (s)− and the preceding part of the proof for nonnegative functions yield Σ(f ) = Σ(f + ) − Σ(f − ) = I(f + ) − I(f − ) = I(f ). It suffices to prove (2.12) for nonnegative Xn . Conditioning on Tn , we have E

(t)

N 

∞ ∞     Xn = E[Xn 1(Tn ≤ t)] =

n=1

n=1

=

∞ 

n=1

E[f (Tn )1(Tn ≤ t)] = E

n=1

f (s)dF n (s)

[0,t]

(t)

N 

 f (Tn ) .

n=1

Then applying (2.11) to the last term yields (2.12). Remark 23. Theorem 22 applies to a renewal process N (t) with its renewal function U being equal to η. For instance, (2.12) would be E

(t)

N  n=1

 Xn =

 f (s)dU (s). (0,t]

Note that this integral does not include the unit jump of U at 0. An extension that includes a value X0 with f (0) = E[X0 ] would be E

(t)

N  n=0

  Xn =

[0,t]

f (s)dU (s).

(2.13)

112

2 Renewal and Regenerative Processes

This remark yields the following special case of a general Wald identity for stopping times in Corollary 25 in Chapter 5. Corollary 24. (Wald Identity for Renewals) For the renewal process N (t), t ≥ 0.

E[TN (t)+1 ] = μE[N (t) + 1],

Proof. Using Remark 23 with f (s) = E[ξn+1 |Tn = s] = μ, it follows that E[TN (t)+1 ] = E

(t)

N 

 ξn+1

n=0

= μU (t) = μE[N (t) + 1]. In light of this result, one might suspect that E[TN (t) ] = μE[N (t)]. However, this is not the case. In fact, E[TN (t) ] ≤ μE[N (t)]; and this is a strict inequality for a Poisson process; see Exercise 22. Example 25. Discounted Rewards. Suppose a renewal process N (t) has rewards associated with it such that a reward (or cost) Yn is obtained at the nth renewal time Tn . The rewards are discounted continuously over time and if a reward y occurs a time t, it has a discounted value of ye−αt . Then the total discounted reward up to time t is N (t)

Z(t) =



Yn e−αTn .

n=1

As in Theorem 22, assume there is a function f : R+ → R such that E[Yn |Tn = s] = f (s), independent of n. Then applying Remark 23 to Xn = Yn e−αTn yields  e−αs f (s)dU (s). E[Z(t)] = (0,t]

The next examples describe several systems modeled by renewal processes with the same type of inter-renewal distribution shown in (2.14) below (also see Exercise 16). Example 26. Single-Server System. Pallets are scheduled to arrive at an automatically guided vehicle (AGV) station according to a renewal process N (t) with inter-arrival distribution F . The station is attended by a single AGV, which can transport only one pallet at a time. Pallets scheduled to arrive when the AGV is already busy transporting a pallet are diverted to another station. Assume the transportation times are independent with common distribution G.

2.3 The Renewal Function

113

Let us consider the times T#n at which the AGV begins to transport a pallet (the times at which pallets arrive and the AGV is idle). For simplicity, assume a transport starts at time 0. To describe T#1 , let τ denote a transport time for the first pallet. Then T#1 equals τ plus the waiting time TN (τ )+1 − τ for the next pallet to arrive after transporting the first pallet. That is, T#1 = TN (τ )+1 . When the next pallet arrives at time TN (τ )+1 , the system is renewed and these cycles are repeated indefinitely. Thus T#n are renewal times. The inter-renewal distribution of T#1 and its mean have reasonable expressions in terms of the arrival process. Indeed, conditioning on τ , which is independent of N (t), yields  # P {T1 ≤ t} = P {TN (x)+1 ≤ t}dG(x). (2.14) R+

Also, if F has a finite mean μ, then by Wald’s identity,   # E[TN (x)+1 ]dG(x) = μ U (x)dG(x). E[T1 ] = R+

R+

Example 27. G/G/1/1 System. Consider a system in which customers arrive at a processing station according to a renewal process with inter-arrival distribution F and are processed by a single server. The processing or service times are independent with the common distribution G, and are independent of the arrival process. Also, customer arrivals during a service time are blocked from being served — they either go elsewhere or go without service. In this context the times T#n at which customers begin services are renewal times as in the preceding example with inter-renewal distribution (2.14). This system is called a G/G/1/1 system: G/G means the inter-arrival and service times are i.i.d. (with general distributions) and 1/1 means there is one server and at most one customer in the system. Example 28. Geiger Counters. A classical model of a Geiger counter assumes that electronic particles arrive at the counter according to a Poisson or renewal process. Upon recording an arrival of a particle, the counter is locked for a random time during which arrivals of new particles are not recorded. The times of being locked are i.i.d. and independent of the arrivals. Under these assumptions, it follows that the times T#n at which particles are recorded are renewal times, and have the same structure as those for the G/G/1/1 system described above. This so-called Type I model assumes that particles arriving while the counter is locked do not affect the counter. A slightly different Type II Geiger counter model assumes that whenever the counter is locked and a particle arrives, that particle is not recorded, but it extends the locked period by another independent locking time. The times at which particles are registered are renewal times, but the inter-renewal distribution is more intricate than that for the Type I counter.

114

2 Renewal and Regenerative Processes

2.4 Future Expectations We have just seen the usefulness of the renewal function for characterizing a renewal process and for describing some expected values of the process. In the following sections, we will discuss the major role a renewal function plays in describing the limiting behavior of probabilities and expectations associated with renewal and regenerative phenomena. This section outlines what to expect in the next three sections, which cover the heart of renewal theory. The analysis to follow will use convolutions of functions with respect to the renewal function U (t), such as   U h(t) = h(t − s)dU (s) = h(0) + h(t − s)dU (s), [0,t]

(0,t]

where h is bounded on finite intervals and equals 0 for t < 0. We will see that many probabilities and expectations associated with a renewal process N (t) can be expressed as a function H(t) that satisfies a recursive equation of the form  H(t) = h(t) + H(t − s)dF (s), t ≥ 0. [0,t]

This “renewal equation”, under minor technical conditions given in the next section, has a unique solution of the form H(t) = U h(t). The next topic we address is the limiting behavior of such functions as t → ∞. We will present Blackwell’s theorem, and an equivalent key renewal theorem, which establishes  1 lim U h(t) = h(s)ds. t→∞ μ R+ This is for non-arithmetic F ; an analogous result holds for arithmetic F . Also, the integral is slightly different from the standard Riemann integral. We cover the topics outlined above — Renewal Equations, Blackwell’s Theorem and the Key Renewal Theorem — in the next three sections. Thereafter, we discuss applications of these theorems that describe the limiting behavior of probabilities and expectations associated with renewal, regenerative and Markov chains.

2.5 Renewal Equations We begin our discussion of renewal equations with a concrete example.

2.5 Renewal Equations

115

Example 29. Let X(t) be a cyclic renewal process on 0, 1, . . . , K − 1, and consider the probability H(t) = P {X(t) = i} as a function of time, for a fixed state i. To show H(t) satisfies a renewal equation, the standard approach is to condition on the time T1 of the first renewal (the first entrance to state 0). The result is H(t) = P {X(t) = i, T1 > t} + P {X(t) = i, T1 ≤ t},

(2.15)

where the last probability, conditioning on the renewal at T1 , is   P {X(t) = i|T1 = s}dF (s) = H(t − s)dF (s). [0,t]

[0,t]

Therefore, the recursive equation (2.15) that H(t) satisfies is H(t) = h(t) + F H(t), where h(t) = P {X(t) = i, T1 > t}. This type of equation is a renewal equation, which is defined as follows. Definition 30. Let h(t) be a real-valued function on R that is bounded on finite intervals and equals 0 for t < 0. The renewal equation for h(t) and the distribution F is  H(t) = h(t) + H(t − s)dF (s), t ≥ 0, (2.16) [0,t]

where H(t) is a real-valued function. That is H = h + F H. We say H(t) is a solution of this equation if it satisfies the equation, and is bounded on finite intervals and equals 0 for t < 0. We first observe that a renewal equation has a unique solution. Proposition 31. The function U h(t) is the unique solution to the renewal equation (2.16). Proof. Clearly U h(t) = 0 for t < 0, and it is bounded on finite intervals since sup |U h(s)| ≤ sup |h(s)|U (t) < ∞, t ≥ 0. s≤t

s≤t

Also, U h is a solution to the renewal equation, since by the definition of U and F 0 h = h, ∞    U h = F 0 + F

F (n−1) h = h + F (U h). n=1

To prove U h is the unique solution, let H(t) be any solution to the renewal equation, and consider the difference D(t) = H(t)−U h(t). From the renewal

116

2 Renewal and Regenerative Processes

equation, we have D = F D, and so iterating this yields D = F n D. Now, the finiteness of U (t) implies F n (t) → 0, as n → ∞, and hence D(t) = 0 for each t. This proves that U h(t) is the unique solution of the renewal equation. The standard approach for deriving a renewal equation is by conditioning on the first renewal time to obtain the function h(t) (recall Example 29). Upon establishing that a function H(t) satisfies a renewal equation, one automatically knows that H(t) = U h(t) by Proposition 31. For instance, Example 29 showed that the probability P {X(t) = i} for a cyclic renewal process satisfies a renewal equation for h(t) = P {X(t) = i, T1 > t}, and hence P {X(t) = i} = U h(t). (2.17) Although H(t) = U h(t) is a solution of the renewal equation, it is not an explicit expression for the function H(t) in that h(t) generally depends on H(t). For instance, h(t) = P {X(t) = i, T1 > t} in (2.17) is part of the probability H(t) = P {X(t) = i}. Only in very special settings is the formula H(t) = U h(t) tractable enough for computations. On the other hand, we will see in Section 2.7 that the function U h(t) is the framework of the Key Renewal Theorem that yields limit theorems for a variety of stochastic processes.

2.6 Blackwell’s Theorem The next issue is to characterize the limiting behavior of functions of the form U h(t) as t → ∞. This is based on the limiting behavior of U (t), which we now consider. Throughout this section, assume that N (t) is a renewal process with renewal function U (t) and mean inter-renewal time μ, which may be finite or infinite. In Section 2.2, we saw that N (t)/t → 1/μ a.s., and so N (t) behaves asymptotically like t/μ as t → ∞ (recall that 1/μ = 0 when μ = ∞). This suggests U (t) = E[N (t)] + 1 should also behave asymptotically like t/μ. Here is a confirmation. Theorem 32. (Elementary Renewal Theorem) t−1 U (t) → 1/μ,

as t → ∞.

Proof. For finite μ, using t < TN (t)+1 and Wald’s identity (Corollary 24), t < E[TN (t)+1 ] = μU (t). This yields the lower bound 1/μ < t−1 U (t). Also, this inequality holds trivially when μ = ∞. With this bound in hand, to finish proving the assertion

2.6 Blackwell’s Theorem

117

it suffices to show

lim sup t−1 U (t) ≤ 1/μ.

(2.18)

t→∞

To this end, for a constant b, define a renewal process N (t) with interrenewal times ξ n = ξn ∧ b. Define T n and U (t) accordingly. Clearly, U (t) ≤ U (t). Also, by Wald’s identity and T N (t)+1 ≤ t + b (since the ξ n are bounded by b), E[ξ1 ∧ b]U (t) = E[T N(t)+1 ] ≤ t + b. Consequently, t−1 U (t) ≤ t−1 U (t) ≤

1 + b/t . E[ξ1 ∧ b]

Letting t → ∞ and then letting b → ∞ (whereupon the last fraction tends to 1/μ, even when μ = ∞), we obtain (2.18), which finishes the proof. A more definitive description of the asymptotic behavior of U (t) is given in the following major result. Theorem 33. (Blackwell) For non-arithmetic F and a > 0, U (t + a) − U (t) → a/μ,

as t → ∞.

If F is arithmetic with span d, the preceding limit holds with a = md for any integer m. Proof. A proof for non-arithmetic F using a coupling argument is in Section 2.15 below. A simpler proof for the arithmetic case is as follows. Suppose F is arithmetic and, for simplicity, assume the span is d = 1. Then renewals occur only at integer times, and pi = F (i) − F (i − 1) is the probability that an inter-renewal time is of length i, where p0 = 0. We will represent the renewal times by the backward recurrence time process {A(t) : t = 0, 1, 2, . . .}, which we know is a Markov chain with transition probabilities pi pi0 = ∞ = 1 − pi,i+1 , i ≥ 0. j=i pj (recall Example 20 and Exercises 37 and 38 in Chapter 1). This chain is irreducible, and hits state 0 at and only at the renewal times. Then the chain is ergodic since the time between renewals has a finite mean μ. Theorem 54 in Chapter 1 yields P {A(t) = 0} → 1/μ (the representation for π0 ). Then because U (t + m) − U (t) is the expected number of renewals exactly at the times t + 1, . . . , t + m, it follows that U (t + m) − U (t) =

m 

P {A(t + k) = 0} → m/μ,

as t → ∞.

k=1

Blackwell’s theorem says that the renewal function U (t) is asymptotically linear. This raises the question: “Does the asymptotic linearity of U (t) lead

118

2 Renewal and Regenerative Processes

to a nice limit for functions of the form U h(t)?” The answer is yes, as we will see shortly. As a preliminary, let us investigate the limit of U h(t) for a simple piecewise-constant function h(s) =

m 

ak 1(s ∈ [sk , tk )),

k=1

where 0 ≤ s1 < t1 ≤ s2 < t2 < · · · ≤ sm < tm < ∞. In this case,  U h(t) = [0,t]

=

m 

h(t − s)dU (s) =

m 

 ak

k=1

t 0

1(t − s ∈ [sk , tk ))dU (s)

ak [U (t − sk ) − U (t − tk )].

(2.19)

k=1

The last equality follows since the integral is over s ∈ [t − tk , t − sk ), and U (t) = 0 when t < 0. By Theorem 33, we know U (t − sk ) − U (t − tk ) → (tk − sk )/μ. Applying this to (2.19) yields 1 1 lim U h(t) = ak (tk − sk ) = t→∞ μ μ m



k=1

h(s)ds.

(2.20)

R+

This result suggests that a limit of this form would also be true for general functions h(t). That is what we will establish next.

2.7 Key Renewal Theorem This section will complete our development of renewal functions and solutions of renewal equations. The issue here is to determine limits of functions of the form U h(t) as t → ∞. We begin with preliminaries on integrals ! t of functions on the infinite axis R+ . Recall that the Riemann integral 0 h(s)ds is constructed by Riemann sums on grids that become finer and finer (see Definition 86 below). The integral exists when h is continuous on [0, t], or is bounded and has a countable number of discontinuities. Furthermore, the Riemann integral of h on R+ is defined by   t

h(s)ds = lim R+

t→∞

h(s)ds, 0

provided the limit exits. In that case, h is Riemann integrable on R+ .

(2.21)

2.7 Key Renewal Theorem

119

The Key Renewal Theorem requires a slightly different notion of a function being directly Riemann integrable on R+ . A DRI function is defined in Section 2.17, where an integral is constructed “directly” on the entire axis R+ by Riemann sums, analogously to the construction of a Riemann integral on a finite interval. A DRI function is Riemann integrable in the usual sense, but the converse is not true; see Exercise 32. For our purposes, we only need the following properties from Proposition 88 below (also see Exercise 33). Remark 34. A function h : R+ → R is DRI in the following cases. (a) h(t) ≥ 0 is decreasing and Riemann integrable. (b) h is continuous except possibly on a set of Lebesgue measure 0, and |h(t)| ≤ b(t), where b is DRI. Here is the main result. Its proof is in Section 2.17. Theorem 35. (Key Renewal Theorem) If F is non-arithmetic and h(t) is DRI, then  1 lim U h(t) = h(s)ds. (2.22) t→∞ μ R+ Remark 36. This theorem is equivalent to Blackwell’s Theorem 33, which asserts that U (t+a)−U (t) → a/μ. Indeed, Section 2.17 shows that Blackwell’s theorem implies the Key Renewal Theorem. Conversely, (2.22) applied to h(s) = 1(a < s < t + a) (so h(t − s) = 1(t − a < s ≤ t)) yields Blackwell’s theorem. An analogous key renewal theorem for arithmetic F is as follows. It can also be proved by Blackwell’s renewal theorem — with fewer technicalities — as suggested in Exercise 31. Theorem 37. (Arithmetic Key Renewal Theorem) If F is arithmetic with span d, then for any u < d, ∞

lim U h(u + nd) =

n→∞

d h(u + kd), μ k=0

provided the sum is absolutely convergent. The next order of business is to show how the limit statement in the key renewal theorem applies to limits of time-dependent probabilities and expected values of stochastic processes. We know that any function H(t) that satisfies a renewal equation has the form H(t) = U h(t). It turns out that this functional form is “universal” in the following sense. Proposition 38. Any function H(t) that is bounded on finite intervals and is 0 for t < 0 can be expressed as H(t) = U h(t),

where

h(t) = H(t) − F H(t).

120

2 Renewal and Regenerative Processes

Proof. This follows since U = F 0 + U F , and so H = F 0 H = (U − U F ) H = U (H − F H). Knowing that U h(t) is a universal form for any function that is bounded on finite intervals, the remaining issue is, “How to relate U h(t) to probabilities and expectations of stochastic processes?” A natural vehicle is the following type of stochastic process. Definition 39. A real-valued stochastic process X(t) is crudely regenerative at a positive random time T if t ≥ 0,

E[X(T + t)|T ] = E[X(t)],

(2.23)

and these expectations are finite. An important connection between crudely regenerative processes and functions U h(t) is as follows. Proposition 40. Suppose that X(t) is a crudely regenerative process at T , which has the distribution F . If E[X(t)] is bounded on finite intervals, then E[X(t)] = U h(t),

where

h(t) = E[X(t)1(T > t)].

Proof. Applying Proposition 38 to H(t) = E[X(t)], it follows that E[X(t)] = U h(t), where  h(t) = H(t) − F H(t) = E[X(t)] − E[X(t − s)]dF (s). [0,t]

By the crude regeneration property, E[X(t)|T = s] = E[X(t − s)], s ≤ t, and  h(t) = E[X(t)] − E[X(t)|T = s]dF (s) = E[X(t)1(T > t)]. [0,t]

This completes the proof. The family of crudely regenerative processes is very large; it includes ergodic Markov chains in discrete and continuous time, regenerative processes, and many functions of these processes as well. More details on these processes are in the next sections. Typically, X(t) is a real-valued function of one or more stochastic processes. An important example is a probability P {Y (t) ∈ A} = E[X(t)], when X(t) = 1(Y (t) ∈ A). The following major result is a version of the key renewal theorem that characterizes limiting distributions and expectations. Many applications in the next sections are based on this formulation. Theorem 41. (Crude Regenerations) Suppose that X(t) is a real-valued process that is crudely regenerative at T , and define M = sup{|X(t)| : t ≤ T }. If T is non-arithmetic and M and M T have finite means, then

2.8 Regenerative Processes

121

lim E[X(t)] =

t→∞

1 μ

 h(s)ds,

(2.24)

R+

where h(t) = E[X(t)1(T > t)]. Proof. Since E[X(t)] = U h(t) by Proposition 40, where T has the nonarithmetic distribution F , the assertion (2.24) will follow by the key renewal theorem provided h(t) is DRI. To prove this, note that |h(t)| ≤ b(t) = E[M 1(T > t)]. Now, by the dominated convergence theorem in the Appendix and E[M ] < ∞, we have b(t) ↓ 0. Also, 



T

b(s)ds = E R+

0

 M ds = E[M T ] < ∞.

(2.25)

Then b(t) is DRI by Remark 34 (a), and so h(t) is DRI by Remark 34 (b).

2.8 Regenerative Processes The primary use of the key renewal theorem is in characterizing the limiting behavior of regenerative processes and their relatives via Theorem 41. This section covers limit theorems for regenerative processes, and the next three sections cover similar results for Markov chains, and processes with regenerative increments. We begin by defining regenerative processes. Loosely speaking, a discreteor continuous-time stochastic process is regenerative if there is a renewal process such that the segments of the process between successive renewal times are i.i.d. More precisely, let {X(t) : t ≥ 0} denote a continuous-time stochastic process with a state space S that is a metric space (e.g., the Euclidean space Rd or a Polish space; see the Appendix). This process need not be a jump process like the continuous-time Markov chains we discuss later. However, we assume that the sample paths of X(t) are right-continuous with left-hand limits a.s. This ensures that the sample paths are continuous except possibly on a set of Lebesgue measure 0. Let N (t) denote a renewal process on R+ , defined on the same probability space as X(t), with renewal times Tn and inter-renewal times ξn = Tn −Tn−1 , which have a distribution F with a finite mean μ. Definition 42. For the process {(N (t), X(t)) : t ≥ 0}, its sample path in the time interval [Tn−1 , Tn ) is described by ζn = (ξn , {X(Tn−1 + t) : 0 ≤ t < ξn }).

(2.26)

This ζn is the nth segment of the process. The process X(t) is regenerative over the times Tn if its segments ζn are i.i.d.

122

2 Renewal and Regenerative Processes

Classic examples of regenerative processes are ergodic Markov chains in discrete and continuous time. An important fact that follows directly from the definition is that functions of regenerative processes inherit the regenerative property. # Remark 43. Inheritance of Regenerations. If X(t) with state space S# is re# generative over Tn , then X(t) = f (X(t)) is also regenerative over Tn , for any f : S# → S. # For instance, we can express the distribution of a regenerative process X(t) # # as the expectation P {X(t) ∈ B} = E[X(t)], where X(t) = 1(X(t) ∈ B) (a # is a real-valued regenerative process. function of X) To include the possibility that the first segment of the process X(t) in the preceding definition may differ from the others, we say X(t) is a delayed regenerative process if ζn are independent, and ζ2 , ζ3 , . . . have the same distribution, which may be different from the distribution of ζ1 . We discuss more general regenerative-like processes with stationary segments in Section 2.19. Remark 44. Regenerative processes are crudely regenerative, but not vice versa. Indeed, if X(t) is regenerative over the times Tn , then X(t) is crudely regenerative at T1 . Next, consider the process X(t) = Xn (t), if t ∈ [n − 1, n] for some n, where {Xn (t) : t ∈ [0, 1]} for n ≥ 1, are independent stochastic processes with identical mean functions (E[Xn (t)] = E[X1 (t)] for each n), but non-identical variance functions. Clearly X is crudely regenerative at T = 1, but it is not regenerative. To proceed, a few comments are in order concerning convergence in distribution. For a process X(t) on a countable state space S, a probability measure P on S is the limiting distribution of X(t) if lim P {X(t) ∈ B} = P(B),

t→∞

B ⊂ S.

(2.27)

This definition, however, is too restrictive for uncountable S, where (2.27) is not needed for all subsets B. In particular, when the state space S is the Euclidean space Rd , then P on S = Rd is defined to be the limiting distribution of X(t) if (2.27) holds for B ∈ S (the Borel sets of S) such that P(∂B) = 0, where ∂B is the boundary of B. Equivalently, P on S is the limiting distribution of X(t) if  lim E[f (X(t))] = f (x)P(dx), (2.28) t→∞

S

for any continuous function f : S → [0, 1]. This means that the distribution of X(t) converges weakly to P (see Section 6.9 in the Appendix for more details on weak convergence).

2.8 Regenerative Processes

123

We are now ready to apply Theorem 41 to characterize the limiting distribution of regenerative processes. For simplicity, assume throughout this section that the inter-renewal distribution F (for the times between regenerations) is non-arithmetic. Theorem 45. (Regenerative Processes) Suppose the process X(t) on a metric state space S (e.g. Rd ) with Borel σ-field S is regenerative over Tn . For f : S → R define M = sup{|f (X(t))| : t ≤ T1 }. If M and M T1 have finite means, then   1 T1 lim E[f (X(t))] = E f (X(s))ds . (2.29) t→∞ μ 0 In particular, the limiting distribution of X(t) is 1 P(B) = lim P {X(t) ∈ B} = E t→∞ μ



T1

0

 1(X(s) ∈ B)ds ,

B ∈ S.

(2.30)

Proof. Assertion (2.29) follows by Theorem 41, since f (X(t)) is regenerative over Tn and therefore it satisfies the crude-regeneration property. Clearly, (2.30) is a special case of (2.29). Theorems 41 and 45 provide a framework for characterizing limits of expectations and probabilities of regenerative processes. For expectations, one must check that the maximum M of the process during an inter-renewal interval has a finite mean. The!main step in applying these theorems, however, ! is to evaluate the integrals R+ h(s)ds or S f (x)P(dx). Keep in mind that one need not set up a renewal equation or check the DRI property for each application — these properties have already been verified in the proof of Theorem 41. Theorem 45 and most of those to follow are true, with slight modifications, for delayed regenerative processes. This is due to the property in Exercise 42 that the limiting behavior of a delayed regenerative process is the same as the limiting behavior of the process after its first regeneration time T1 . Here is an immediate consequence of Theorem 45 and Exercise 42. Corollary 46. (Delayed Regenerations) Suppose the process X(t) with a metric state space S is a delayed regenerative process over Tn . If f : S → R is such that the expectations of M = sup{|f (X(t))| : T1 ≤ t ≤ T2 } and M ξ2 are finite, then 1 lim E[f (X(t))] = E t→∞ μ



T2

 f (X(s)) ds .

T1

In particular, the limiting distribution of X(t) is P(B) =

1 E μ



T2

T1

 1(X(s) ∈ B)ds ,

B ∈ S.

124

2 Renewal and Regenerative Processes

We end this section with applications of Theorem 45 to three regenerative processes associated with a renewal process. Definition 47. Renewal Process Trinity. For a renewal process N (t), the following three processes provide more information about renewal times: A(t) = t − TN (t) , the backward recurrence time at t (or the age), which is the time since the last renewal prior to t. B(t) = TN (t)+1 − t, the forward recurrence time at t (or the residual renewal time), which is the time to the next renewal after t. L(t) = ξN (t)+1 = A(t) + B(t), length of the renewal interval covering t. For instance, a person arriving at a bus stop at time t would have to wait B(t) minutes for the next bus to arrive, or a call-center operator returning to answer calls at time t would have to wait for a time B(t) before the next call. Also, if a person begins analyzing an information string at a location t looking for a certain character (or pattern), then A(t) and B(t) would be the distances to the left and right of t where the next character occurs. Note that the three-dimensional process (A(t), B(t), L(t)) is regenerative over Tn , and so is each process by itself. Each of the processes A(t) and B(t) is a continuous-time Markov process with piece-wise deterministic paths on the state space R+ ; see Exercises 34 and 35. A convenient expression for their joint distribution is, for 0 ≤ x < t, y ≥ 0, P {A(t) > x, B(t) > y} = P {N (t + y) − N ((t − x)−) = 0}.

(2.31)

This is simply the probability of no renewals in [t − x, t + y]. Although this probability is generally intractable, one can show that it is the solution of a renewal equation, and so it has the form U h(t); see Exercises 36 and 37. Example 48. Trinity in Equilibrium. One can obtain the limiting distributions of A(t) and B(t) separately from Theorem 45. Instead, we will derive their joint limiting distribution. Since (A(t), B(t)) is regenerative over Tn , Theorem 41 yields  1 x+y lim P {A(t) > x, B(t) > y} = 1 − [1 − F (s)]ds, (2.32) t→∞ μ 0 since, by the definitions of the variables, h(t) = P {A(t) > x, B(t) > y, T1 > t} = P {T1 > t + y}1(t > x). From (2.32), it immediately follows that 1 lim P {A(t) ≤ x} = lim P {B(t) ≤ x} = t→∞ t→∞ μ

 0

x

[1 − F (s)]ds.

(2.33)

2.8 Regenerative Processes

125

This limiting distribution, which is called the equilibrium distribution associated with F , is important in other contexts. We will see its significance in Section 2.15 for stationary renewal processes. One can also obtain the limiting distribution of L(t) = A(t) + B(t) by Theorem 41. Namely,  1 sdF (s), (2.34) lim P {L(t) ≤ x} = t→∞ μ [0,x] since h(t) = P {L(t) ≤ x, T1 > t} = P {T1 ≤ x, T1 > t} = (F (x) − F (t))1(x > t). Alternatively, one can derive (2.34) directly from (2.32). Additional properties of the three regenerative processes A(t), B(t) and L(t) are in Exercises 34–41. These processes are especially nice for a Poisson process. Example 49. Poisson Recurrence Times. If N (t) is a Poisson process with rate λ, then from (2.31) P {A(t) > x, B(t) > y} = e−λ(x+y) ,

0 ≤ x < t, y ≥ 0,

(2.35)

which is the Poisson probability of no renewals in an interval of length x + y. In particular, setting x = 0, and then y = 0, yields P {B(t) > y} = e−λy ,

P {A(t) > x} = e−λx 1(x < t).

Thus B(t) is exponentially distributed with rate λ; this also follows by the memoryless property of the exponential distribution (Exercise 1 in Chapter 3). Note that A(t) has the same exponential distribution, but it is truncated at x = t. The limiting distribution of each of these processes, however, is exponential with rate λ. Since L(t) = A(t) + B(t), its distribution can be obtained from (2.35); its mean is shown in Exercise 39. Even though recurrence time processes A(t) and B(t) are typically not tractable for a fixed t, their equilibrium distribution Fe in (2.33) may be. Example 50. Uniformly Distributed Renewals. Suppose N (t) is a renewal process with uniform inter-renewal distribution F (x) = x, for x ∈ [0, 1]. Its associated equilibrium distribution (2.33) is simply Fe (x) = 2x − x2 . Interestingly, Fe (x) ≥ F (x) for each x. That is, the distribution Fe for the forward recurrence time B(t) in equilibrium is greater than the distribution F of the forward recurrence time B(0) = ξ1 at time 0. This means that B(t) in equilibrium is stochastically smaller than B(0). This is due to the fact that the failure rate F  (x)/(1 − F (x)) = 1/(1 − x) of F is increasing. Compare this property with the inspection paradox in Exercise 39.

126

2 Renewal and Regenerative Processes

2.9 Limiting Distributions for Markov Chains This section covers the classical renewal argument for determining the limiting distributions of ergodic Markov chains. The argument uses limit theorems in the preceding section, which are manifestations of the key renewal theorem for regenerative processes. We present a similar characterization of limiting distributions for continuous-time Markov chains in Chapter 4. Assume that Xn is an ergodic Markov chain on a countable state space S, with limiting distribution πj = lim P {Xn = j}, n→∞

j ∈ S,

which does not depend on X0 . Recall that Theorems 59 and 54 in Chapter 1 established that the limiting distribution is also the stationary distribution and it is the unique distribution π that satisfies the balance equation π = πP. They also showed (via a coupling proof) that the stationary distribution has the following form, which we will now prove by a classical renewal argument. Theorem 51. (Markov Chains) The ergodic Markov chain Xn has a unique limiting distribution given as follows: for a fixed i ∈ S, τ1 (i)−1   1   1(Xn = j)X0 = i , πj = E μi n=0

j ∈ S,

(2.36)

where μi = E[τ1 (i)|X0 = i]. Another expression for this probability is πj =

1 , μj

j ∈ S.

(2.37)

Proof. We will prove this by applying the key renewal theorem. The main idea is that the strong Markov property ensures that Xn is a (discrete-time) delayed regenerative process over the times 0 < τ1 (i) < τ2 (i) < . . . at which Xn enters a fixed state i (Theorem 67 in Chapter 1). In light of this fact, the assertion (2.36) follows by Corollary 46. Also, setting i = j in (2.36) yields (2.37), since the sum in (2.36) is the sojourn time in state j, which is 1.

2.10 Processes with Regenerative Increments Many cumulative cost or utility processes associated with ergodic Markov chains and regenerative processes can be formulated as processes with regenerative increments. These processes are basically random walks with auxiliary paths or information. We will show that the classical SLLN and central limit theorem for random walks extend to processes with regenerative increments.

2.10 Processes with Regenerative Increments

127

This section presents a SLLN based on material in Section 2.2, and Section 2.13 presents a central limit theorem. Functional central limit theorems for random walks and processes with regenerative increments are the topic of Section 5.9 in Chapter 5. For this discussion, N (t) will denote a renewal process whose inter-renewal times ξn = Tn − Tn−1 have a distribution F and finite mean μ. Our focus will be on the following processes that are typically associated with cumulative information of regenerative processes. Definition 52. Let Z(t) be a real-valued process with Z(0) = 0 defined on the same probability space as a renewal process N (t). For the two-dimensional process {(N (t), Z(t)) : t ≥ 0}, its increments in the time interval [Tn−1 , Tn ) are described by ζn = (ξn , {Z(t + Tn−1 ) − Z(Tn−1 ) : 0 ≤ t < ξn }). The process Z(t) has regenerative increments over the times Tn if ζn are i.i.d. A process with “delayed” regenerative increments is defined in the obvious way, where the distribution ζ1 is different from the others. Under this definition, (Tn − Tn−1 , Z(Tn ) − Z(Tn−1 )) are i.i.d. These i.i.d. increments of N and Z leads to many nice limit theorems based on properties of random walks. A primary example !of a process with regenerative increments is a cumulat tive functional Z(t) = 0 f (X(s))ds, where X(t) is regenerative over Tn and f (i) is a cost rate (or utility rate) when the process X(t) is in state i. Hereafter, assume that Z(t) is a process with regenerative increments over Tn . Keep in mind that Z(0) = 0. Although the distribution and mean of Z(t) are generally not tractable for computations, we do have a Wald identity for some expectations. Proposition 53. (Wald Identity for Regenerations) For the process Z(t) with regenerative increments and finite a = E[Z(T1 )], E[Z(TN (t)+1 )] = aE[N (t) + 1], Proof. By Theorem 22,

⎡

E[Z(TN (t)+1 )] = E ⎣

(2.38)

⎤

N (t)



t ≥ 0.

[Z(Tn+1 ) − Z(Tn )]⎦ = a ∪ (t).

n=0

By the classical SLLN, we know that n−1 Z(Tn ) = n−1

n  

Z(Tk ) − Z(Tk−1 ) → E[Z(T1 )],

a.s. as n → ∞.

k=1

This extends to Z(t) as follows, which is a special case of Corollary 14. Here Mn =

sup Tn−1 0. Assume the times Tn form a point process on R+ such that the limit μ = limn→∞ n−1 Tn exists and is positive.4 This simply says that the system empties out infinitely often, and it does so at times that have a limiting average. 4

Keep in mind that the arrival process N (t) is “not” the counting process associated with these empty times Tn .

130

2 Renewal and Regenerative Processes

Theorem 57. (Little Law) Suppose the system described above satisfies the empty-system assumption. If any two of the averages L, λ or W exists, then the other one also exists, and L = λW . Proof. With no loss in generality, we may assume the system is empty at time 0 and an item arrives. We begin with the key observation that in the time interval [0, Tn ), all of the νn = N (Tn −) items that arrive in the interval also depart by the empty-system time Tn , and their total waiting time is νn 

Wk =

k=1

∞  

Tn

0

k=1

 1(τk ≤ s < τk + Wk )ds =

Tn

Q(s)ds.

(2.41)

0

The first equality follows since the system is empty just prior to Tn , and the second equality follows from the definition of Q(t). Also, observe that under the assumptions t−1 N (t) → λ and Tn /n → μ, n−1 νn = Tn−1 N (Tn −)(n−1 Tn ) → λμ.

(2.42)

First assume that λ and W exist. Then by (2.41), we have n−1

 0

Tn

Q(s) ds = (νn /n)νn−1

νn 

Wk → λμW.

k=1

Therefore, an application of Theorem 13 to the nondecreasing process Z(t) = !t 0 Q(s) ds and the times Tn yields L = lim t−1 Z(t) = λW. t→∞

Next, assume that λ and L exist. Then by (2.41), n−1

νn  k=1

  Wk = (n−1 Tn ) Tn−1

Tn

0

 Q(s)ds → μL,

a.s. as t → ∞.

Now, by na discrete-time version of Theorem 13 for the nondecreasing process Zn = k=1 Wk and integer-valued indices νn , which satisfy (2.42), it follows that W = lim n−1 Zn = L/λ. n→∞

Thus, W exists and L = λW . Exercise 23 shows that if L and W exist then λ exists and L = λW . The preceding Little law applies to a wide variety of queueing systems as long as two of the averages λ, L or W exist. Here are a few examples. Example 58. Regenerative Processing System. Suppose the system described above satisfies the empty-system assumption, the arrival process is a renewal

2.11 Average Sojourn Times in Regenerative Processes

131

process with a finite mean 1/λ, and Q(t) is a regenerative process over the !T empty-system times Tn . Assume that T1 and 0 1 Q(s)ds have finite means. By the SLLN for the renewal input process, the arrival rate is λ. Also, !t !T applying Theorem 54 to Z(t) = 0 Q(s)ds, we have L = E[ 0 1 Q(s)ds]/E[T1 ]. Therefore, by Theorem 57, the average waiting time W exists and L = λW ; !T that is, W = E[ 0 1 Q(s)ds]/(λE[T1 ]). In some queueing systems, the Little law L = λW we have been discussing for averages has an analogue in which the averages are means. Example 59. Little Laws for Means. Consider the system in the preceding example with the additional assumption that the sequence of sojourn times Wn is regenerative over the discrete times νn = N (Tn −). Since Q(t) is regenerative over Tn , and Wn is regenerative over νn , d

Q(t) → Q as t → ∞,

and

d

Wn → W as n → ∞,

where the distributions of Q and W are described in Theorem 45. Furthermore, by Theorem 55,  t −1 L = lim t Q(s)ds = E[Q] a.s., t→∞

0

W = lim n−1 n→∞

n 

Wk = E[W ] a.s.

k=1

˜ (1)], where Also, the renewal arrival rate λ can be represented as λ = E[N ˜ (t) is a stationary version of N (t) as described in Theorem 76 below. Then N the Little law L = λW that holds for averages has the following analogue for expected values: ˜ (1)]E[W ]. E[Q] = E[N Example 60. G/G/1 Queueing System. A general example of a regenerative queueing system is a G/G/1 system, where arrivals form a renewal process with mean inter-arrival time 1/λ, the services times are i.i.d., independent of the arrivals, and customers are served by a single server under a first-infirst-out (FIFO) discipline. Assume that the mean service time is less than !T the mean inter-arrival time, and that T1 and 0 1 Q(s)ds have finite means. In this case, the sojourn times Wn are regenerative over νn = N (Tn −), and W exists by Theorem 118 in Chapter 4. Then it follows by Theorem 57 that the average queue length L exists and L = λW . Special cases of the G/G/1 system are an M/G/1 system when the arrival process is a Poisson process, a G/M/1 system when the service times are exponentially distributed, and an M/M/1 system when the arrivals are Poisson and the service times are exponential.

132

2 Renewal and Regenerative Processes

Theorem 57 also yields expected waiting times in Jackson networks, which we discuss in Chapter 5. There are several Little laws for input-output systems and general utility processes not related to queueing [101]. The next result is an elementary but very useful example. Let X(t) be a regenerative process over Tn with state space S. Assume X(t) is a pure jump process (piecewise constant paths, etc.) with a limiting distribution p(B) = limn→∞ P {X(t) ∈ B}, which is known. Let B denote a fixed subset of the state space whose complement B c is not empty. The expected number of times that X(t) enters B between regenerations is 

  1 X(τn−1 ) ∈ B c , X(τn ) ∈ B, τn ∈ (T1 , T2 ] , γ(B) = E n

where τn is the time of the nth jump of X(t). The expected number of transitions of X(t) between regenerations is γ(S), which we assume is finite. Consider the average sojourn time of X(t) in B defined by W (B) = lim n−1 n→∞

n 

Wk (B),

k=1

where Wn (B) is its sojourn time in B at its nth visit to the set. Proposition 61. (Sojourns in Regenerative Processes) For the regenerative process X(t) defined above, its average sojourn time in B exists and is W (B) = p(B)γ(S)/γ(B). Proof. Consider Q(t) = 1(X(t) ∈ B) as an artificial queueing process that only takes values 0 or 1. Clearly Q(t) is regenerative over Tn , since X(t) is regenerative over Tn ; and Q(t) satisfies the empty-system assumption. Now, the limiting average of Q(t) is L = lim t t→∞

−1

 0

t

1(X(s) ∈ B) ds = p(B).

The arrival rate λ is the rate γ(B)/γ(S) at which X(t) enters B. Thus, Theorem 57 yields p(B) = λW (B) = (γ(B)/γ(S))W (B), which proves the assertion.

2.12 Batch-Service Queueing System For service systems that process items in batches, a basic problem is to determine when to serve batches and how many items should be in the batches. This is a dynamic control problem or a Markov decision problem. We will

2.12 Batch-Service Queueing System

133

address this problem for a particular setting and show how to obtain certain control parameters by using a SLLN for regenerative processes. Consider a single-server station that serves items or customers in batches as follows. Items arrive to the station according to a Poisson process with rate λ and they enter a queue where they wait to be served. The server can serve items in batches, and the number of items in a batch can be any number less than or equal to a fixed number K ≤ ∞ (the service capacity). The service times of the batches are independent, identically distributed and do not depend on the arrival process or the batch size (think of a computer, bus, or truck). Only one batch can be served at a time and, during a service, additional arrivals join the queue. The server observes the queue length at the times at which an arrival occurs and the server is idle, or whenever a service is completed. At each of these observation times, the server takes one of the following actions: • No items are served. • A batch consisting of all or a portion of the items waiting is served (the batch size cannot exceed i ∧ K, where i is the queue length). These actions control the batch sizes and the timing of the services. If the server takes the first action, the next control action is taken when the next item arrives, and if the server takes the second action to serve a batch, the next control action is taken when the service is completed. A control policy is a rule for selecting one of these actions at each observation time. The general problem is to find a control policy that minimizes the average cost (or discounted cost) of serving items over an infinite time horizon. This Markov decision problem was solved in [33] for natural holding and service cost functions for both the average-cost and discounted-cost criteria. In either case, the main result is that there is an optimal M -policy of the following form: At each observation time when the queue length is i, do not serve any items if i < M , and serve a batch of i ∧ K items if i ≥ M . Here M is an “optimal” level that is a function of the costs. We will now describe an optimal level M for a special case. Suppose the system is to operate under the preceding M -policy, where the capacity K is infinite, and the service times are exponentially distributed with rate γ. Assume there is cost C for serving a batch and a cost hi per unit time for holding i items in the queue. Theorem 62. Under the preceding assumptions, the average cost per unit time is minimized by setting the level M to be M = min{m ≥ 0 : m(m + 1) ≥ 2[(λ/γ)2 pm + Cλ/h]},

(2.43)

where p = λ/(λ + γ). Proof. Let Xm (t) denote the number of items in the queue at time t, when the system is operated under an m-policy. Let Tn denote the time at which the server initiates the nth service, and let N (t) denote the associated counting

134

2 Renewal and Regenerative Processes

process. For simplicity, assume that a service has just been completed at time 0, and let T0 = 0. We will show that, under the m-policy with exponential service times, the Tn are renewal times; and the service plus holding cost in [0, t] is  Zm (t) = CN (t) + h

t

0

Xm (s) ds.

Next, we will establish the existence of the average cost f (m) = lim t−1 Zm (t), t→∞

and then show that f (m) is minimized at the M specified in (2.43). Let Qn denote the queue length at the nth service initiation time Tn , for n ≥ 0. Note that Qn is just the number of arrivals that occur during the nth service period, since all the waiting items are served in the batch. Because of the exponential service times, Qn , for n ≥ 1, are i.i.d. with  (λt)i e−λt −γt P {Qn = i} = γe dt = pi (1 − p), i ≥ 0. (2.44) i! R+ For notational convenience, assume the initial queue length Q0 has this distribution and is independent of everything else. Next, observe that the quantity Qn determines the time ξn+1 = Tn+1 − Tn until the next service initiation. Specifically, if Qn ≥ m, then ξn+1 is simply a service time; and if Qn = i < m, then ξn+1 is the time it takes for m − i more items to arrive plus a service time. Since the Qn are i.i.d., it follows that Tn are renewal times. Furthermore, conditioning on Q0 , the inter-renewal distribution is P {ξ1 ≤ t} = P {Q0 ≥ m}Gγ (t) +

m−1 

(m−i)

P {Q0 = i}Gλ

Gγ (t),

i=0

where Gλ is an exponential distribution with rate λ. Then using the distribution (2.44), the inter-renewal distribution and its mean (indexed by m) are:5 P {ξ1 ≤ t} = pm Gγ (t) + (1 − p)

m−1 

(m−i)

pi Gλ

Gγ (t),

i=0 m

μm = γ −1 + mλ−1 − (1 − p )γ −1 . Now, the increasing process Zm (t) is such that Zm (Tn ) − Zm (Tn−1 ), for n ≥ 1, are i.i.d. with mean 5

The identity

k i=1

ipi−1 =

d k ( i=0 dp

pi ) is used to derive the formula for μm .

2.13 Central Limit Theorems

135

 E[Zm (T1 )] = C + hE

0

T1

 Xm (s) ds .

Then by Theorem 13, the average cost, as a function of m, is f (m) = lim t−1 Zm (t) = μ−1 m E[Zm (T1 )]. t→∞

˜ To evaluate this limit, let N(t) denote the Poisson arrival process with exponential inter-arrival times ξ˜n , and let τ denote an exponential service time with rate γ. Then we can write 



T1

Xm (s) ds = Q0 τ +

0

τ

˜ N(s) ds + 0

m−1 

1(Q0 = i)

i=0

m−i 

(i + k − 1)ξ˜k . (2.45)

k=1

The first two terms on the right-hand side represent the holding time of items during the service period, and the last term represents the holding time of items (which is 0 if Q0 ≥ m) prior to the service period. Then from the independence of Q0 and τ and Exercise 14,

 E 0

T1

m−1    m−i  Xm (s) ds = 1/(1 − p)γ + λ/γ 2 + (1 − p)λ−1 pi (i + k − 1) . i=0

k=1

Substituting this in the expression above for f (m), it follows from lengthy algebraic manipulations that f (m + 1) − f (m) = h(1 − pm+1 )Dm /(λ2 μm μm+1 ), where Dm = m(m + 1) − 2[(λ/γ)2 pm + Cλ/h]. Now, Dm is increasing in m and the other terms in the preceding display are positive. Therefore f (m) is monotone decreasing and then increasing and has a unique minimum at M = min{m : Dm ≥ 0}, which is equivalent to (2.43). Analysis similar to that above yields a formula for the optimal level M when the service capacity K is finite; see Exercise 52 in Chapter 4.

2.13 Central Limit Theorems For a real-valued process Z(t) with regenerative increments over Tn , we know that under the conditions in Theorem 54, Z(t)/t → a = E[Z(T1 )]/E[T1 ] a.s.

as t → ∞.

In other words, Z(t) behaves asymptotically like at. Further information about this behavior can be obtained by characterizing the limiting

136

2 Renewal and Regenerative Processes

distribution of the difference Z(t) − at as t → ∞. We will now present a central limit theorem that gives conditions under which this limiting distribution is a normal distribution. Special cases of this result are CLT’s for renewal and Markovian processes. We will obtain the CLT for regenerative processes by applying the following classical CLT for sums of independent random variables (which is proved in standard probability texts). The analysis will involve the notion of convergence in distribution of random variables; see Section 6.9 in the Appendix. Theorem 63. (Classical CLT) Suppose X1 , X2 , . . . are i.i.d. n random variables with mean μ and variance σ2 > 0, and define Sn = m=1 (Xm − μ). Then  x −y2 /(2σ2 ) e √ dy, x ∈ R. P {Sn /n1/2 ≤ x} → σ 2π −∞ This convergence in distribution is denoted by Sn /n1/2 → N (0, σ 2 ), d

as n → ∞,

where N (0, σ 2 ) is a normal random variable with mean 0 and variance σ 2 . We will also use the following result for randomized sums; see for instance p.216 in [26]. This result and the ones below are contained in the functional central limit theorems in Chapter 5, which focus on the convergence of entire stochastic processes instead of random variables. Theorem 64. (Anscombe) In the context of Theorem 63, let N (t) be an integer-valued process defined on the same probability space as the Xn , where d N (t) may depend on the Xn . If t−1 N (t) → c, where c is a positive constant, then d SN (t) /t1/2 → N (0, cσ 2 ), as t → ∞. The following is a regenerative analogue of the classical CLT. Theorem 65. (Regenerative CLT) Suppose Z(t) is a real-valued process with regenerative increments over Tn such that μ = E[T1 ] and a = E[Z(T1 )]/μ are finite. In addition, let Mn =

|Z(t) − Z(Tn−1 )|,

sup

n ≥ 1,

Tn−1 0. Then (Z(t) − at)/t1/2 → N (0, σ 2 /μ), d

as t → ∞.

(2.46)

Proof. The process Z(t) is “asymptotically close” to Z(TN (t) ), when dividing them by t1/2 , because their difference is bounded by MN (t)+1 , which is a regenerative process that is 0 at regeneration times. Consequently, the normalized process

2.13 Central Limit Theorems

137

˜ = (Z(t) − at)/t1/2 Z(t) should have the same limit as the process Z  (t) = (Z(TN (t) ) − aTN (t) )/t1/2 . Based on this conjecture, we will prove Z  (t) → N (0, σ 2 /μ), d

as t → ∞,

˜ − Z  (t)| → 0, |Z(t) d

as t → ∞.

(2.47) (2.48)

Then it will follow by a standard property of convergence in distribution (see Exercise 53 of Chapter 5), that d ˜ = Z  (t) + (Z(t) ˜ − Z  (t)) → Z(t) N (0, σ 2 /μ).

To prove (2.47), note that N (t)

Z  (t) = t−1/2



Xn ,

n=1

where Xn = Z(Tn ) − Z(Tn−1 ) − a(Tn − Tn−1 ). Since Z(t) has regenerative increments over Tn , the Xn are i.i.d. with mean 0 and variance σ 2 . Also, t−1 N (t) → 1/μ by the SLLN for renewal processes. In light of these observations, Anscombe’s theorem above yields (2.47). To prove (2.48), note that ˜ − Z  (t) = t−1/2 [Z(t) − Z(TN (t) ) − a(t − TN (t) )]. Z(t) Then letting Yn = Mn + a(Tn+1 − Tn ), it follows that   $ ˜ − Z  (t)| ≤ t−1/2 YN (t) = N (t)/t N (t)−1/2 YN (t) . |Z(t) Since Z(t) has regenerative increments, the Yn are i.i.d., and so n−1/2 Yn = n−1/2 Y1 → 0 a.s. d

Using this and N (t)/t → 1/μ a.s. in the preceding proves (2.48). An important use of a CLT is to find confidence intervals for certain parameters. Here is an example. Example 66. Confidence Interval for the Mean. Under the assumptions of Theorem 65, let us construct a confidence interval for the mean a of the regenerative-increment process Z(t) based on observing the process up to a fixed time t. Assume (which is reasonable) that we do not know the variance parameter σ 2 = Var[Z(T1 ) − aT1 ].

138

2 Renewal and Regenerative Processes

We first note that by the SLLN for Z(t), $ √ t−1/2 Z(t) − at → σ/ μ t−1 (Z(t) − at) → 0,

a.s. as t → ∞.

Then the latter combined with Theorem 65 yields (Z(t) − at)/

$

d

Z(t) − at → N (0, 1).

Therefore, a confidence interval with approximate confidence coefficient 1− α (e.g., see [99]) is

Z(t) − at  Z(t) − at $ − zα/2 , $ + zα/2 , Z(t) − at Z(t) − at where P {−zα/2 ≤ N (0, 1) ≤ zα/2 } = 1 − α. Insights on simulation procedures for this and related models are in [43]. What would be an analogous confidence interval when Z(t) is observed only at regeneration times? See Exercise 52. Applying Theorem 65 to a regenerative-increment process involves determining conditions on the process under which the main assumptions are satisfied and then finding expressions for the normalization constants a and σ. Here are some examples. Example 67. CLT for Renewal Processes. Suppose that N (t) is a renewal process whose inter-renewal distribution has a finite mean μ and variance σ 2 . Then Z(t) = N (t) satisfies the assumptions in Theorem 65, and so (N (t) − t/μ)/t1/2 → N (0, σ 2 /μ3 ), d

where a = 1/μ,

as t → ∞,

Var[Z(T1 ) − aT1 ] = σ 2 μ−2 .

Example 68. CLT for Markov Chains. Let Xn be an ergodic Markov chain on S with limiting distribution π. Consider the sum Zn =

n 

f (Xm ),

n ≥ 0,

m=1

where f (j) is a real-valued cost or utility for the process being in state j. For simplicity, fix an i ∈ S and assume X0 = i a.s. Then Zn has regenerative increments over the discrete times νn at which Xn enters state i. We will apply a discrete-time version of Theorem 65

to Zn .  Accordingly, assume μi = E[ν1 ] and E max1≤n≤ν1 |Zn | are finite. The

  ν1 latter is true when E |f (X )| is finite. In addition, assume n n=1

2.14 Terminating Renewal Processes

a=

139

 1 Ei [Zν1 ] = πj f (j), μi

and σ 2 =

j∈S

1 Var[Zν1 − aν1 ] μi

are finite and σ > 0. Letting f˜(j) = f (j) − a, Exercise 54 shows that 2

σ = E[f˜(X0 )2 ] + 2

∞ 

E[f˜(X0 )f˜(Xn )],

(2.49)

n=1

where P {X0 = i} = πi . Then Theorem 65 (in discrete time) yields (Zn − an)/n1/2 → N (0, σ 2 ), d

as n → ∞.

(2.50)

This result also applies to random functions of Markov chains as follows (see Exercise 33 in Chapter 4 for a related continuous-time version). Suppose Zn =

n 

f (Xm , Ym ),

n ≥ 0,

m=1

where f : S × S  → R, and Ym are conditionally independent given Xn (n ≥ 0), and P {Ym ∈ B|Xn , n ≥ 0} only depends on Xm and B ∈ S  . Here S  need not be discrete. In this setting, the cost or utility f (Xm , Ym ) at time m is partially determined by the auxiliary or environmental variable  Ym . Then the argument above yields the CLT (2.50). In this case, a = j∈S πj α(j), α(j) = E[f (j, Y1 )], and σ 2 = E[(f (X0 , Y1 ) − α(X0 ))2 ] + 2

∞ 

E[m(X0 , Xn )],

n=1

where m(j, k) = E[(f (j, Y1 ) − α(j))(f (k, Y2 ) − α(k))].

2.14 Terminating Renewal Processes In this section, we discuss renewal processes that terminate after a random number of renewals. Analysis of these terminating (or transient) renewal processes uses renewal equations and the key renewal theorem applied a little differently than above. Consider a sequence of renewal times Tn with inter-renewal distribution F . Suppose that at each time Tn (including T0 = 0), the renewals terminate with probability 1 − p, or continue until the next renewal epoch with probability p. These events are independent of the preceding renewal times, but may depend on the future renewal times.

140

2 Renewal and Regenerative Processes

Under these assumptions, the total number of renewals ν over the entire time horizon R+ has the distribution P {ν ≥ n} = pn ,

n ≥ 0,

and E[ν] = p/(1 − p). The number of renewals in [0, t] is N (t) =

∞ 

1(Tn ≤ t, ν ≥ n),

t ≥ 0.

n=1

Of course N (t) → ν a.s. Another quantity of interest is the time Tν at which the renewals terminate. We will also use the following equivalent formulation of this terminating renewal process. Assume that N (t) counts renewals in which the independent inter-renewal times have an improper distribution G(t), with p = G(∞) < 1. Then p is the probability of another renewal and 1 − p = 1 − G(∞) is the probability that an inter-renewal time is “infinite”, which terminates the renewals. This interpretation is consistent with that above since necessarily G(t) = pF (t), where F as described above is the conditional distribution of an inter-renewal time given that it is allowed (or is finite). Similarly to renewal processes, we will address issues about the process N (t) with the use of its renewal function V (t) =

∞ 

Gn (t) =

n=0

∞ 

pn F n (t).

n=0

We first observe that the counting process N (t) and the termination time Tν are finite a.s., and their distributions and means are P {N (t) ≥ n} = Gn (t), P {Tν ≤ t} = (1 − p)V (t),

E[N (t)] = V (t) − 1, E[Tν ] = pμ/(1 − p).

(2.51)

To establish these formulas, recall that the events ν = n (to terminate at n) and ν > n (to continue to the n+1st renewal) are assumed to be independent of T1 , . . . , Tn . Then P {N (t) ≥ n} = P {ν ≥ n, Tn ≤ t} = pn F n (t) = Gn (t), ∞  P {N (t) ≥ n} = V (t) − 1. E[N (t)] = n=1

Similarly, using the independence and Tν =

∞ n=0

1(ν = n)Tn ,

2.14 Terminating Renewal Processes

P {Tν ≤ t} = E[Tν ] =

∞  n=0 ∞ 

141

P {ν = n, Tn ≤ t} = (1 − p)

∞ 

pn F n (t),

n=0

P {ν = n}E[Tn ] = μp/(1 − p).

n=1

Although a regular renewal function tends to infinity, the renewal function for a terminating process has a finite limit. Remark 69. As t → ∞ V (t) =

∞ 

pn F n∗ (t) → 1/(1 − p).

n=0

Corollary 71 below describes the convergence rate. We will now discuss limits of certain functions associated with the terminating renewal process. As in Proposition 31, it follows that H(t) = V h(t) is the unique solution to the renewal equation H(t) = h(t) + G H(t). We will consider the limiting behavior of H(t) for the case in which the limit h(∞) = lim h(t) t→∞

exists, which is common in applications. Since V (t) → 1/(1 − p) and h(t) is bounded on compact sets and converges to h(∞), it follows by dominated convergence that  [h(t − s) − h(∞)]dV (s) H(t) = h V (t) = h(∞)V (t) + [0,t]

→ h(∞)/(1 − p)

as t → ∞.

(2.52)

The next result describes the rate of this convergence under a few more technical conditions. Assume there is a positive β such that  eβt dG(t) = 1. R+

The existence of a unique β is guaranteed under the weak condition that ! βt e dG(t) is finite for some β > 0. Indeed, this function of β is continuous R+ and increasing and, being finite at one point, its range contains the set [p, ∞); thus, it must equal 1 for some β. We also assume the distribution  eβs dG(s) F # (t) = [0,t]

is non-arithmetic and has a mean μ# .

142

2 Renewal and Regenerative Processes

Theorem 70. In addition to the preceding assumptions, assume the function eβt [h(t) − h(∞)] is DRI. Then H(t) = h(∞)/(1 − p) + ce−βt /μ# + o(e−βt ), ! where c = R+ eβs [h(s) − h(∞)] ds − h(∞)/β.

as t → ∞,

(2.53)

Proof. Multiplying the renewal equation H = h + G H by eβt yields the renewal equation H # = h# + F # H # where H # (t) = eβt H(t) and h# (t) = eβt h(t). We can now describe the limit of H(t) − h(∞)/(1 − p) by the limit of H # (t) − v(t), where v(t) = eβt h(∞)/(1 − p). From Lemma 83 below,  1 H # (t) = v(t) + # h(s) ds + o(1), as t → ∞, (2.54) μ R+ provided h(t) = h# (t) − v(t) + F # v(t) is DRI. In this case, h(t) = eβt [h(t) − h(∞)] −

h(∞)eβt 1−p

 (p − G(t)) .

(2.55)

Now, the first term on the right-hand side is DRI by assumption. Also,  eβt (p − G(t)) ≤ eβs dG(s) = 1 − F # (t). (t,∞)

This bound is decreasing to 0 and its integral is μ# , and so the last term in brackets in (2.55) is DRI. Thus h(t) is DRI. Finally, an easy check shows ! that R+ h(s) ds = c, the constant in (2.53). Substituting this in (2.54) and dividing by eβt yields (2.53). Corollary 71. Under the assumptions preceding Theorem 70, V (t) = 1/(1 − p) − e−βt /(βμ# ) + o(e−βt ), P {Tν > t} = (1 − p)e−βt /(βμ# ) + o(e−βt ),

as t → ∞.

Proof. The first line follows by Theorem 70 with h(t) = 1, since by its definition, V (t) = 1 + G V (t). The second follows from the first line and (2.51). Example 72. Waiting Time for a Gap in a Poisson Process. Consider a Poisson process with rate λ that terminates at the first time a gap of size ≥ c occurs. That is, the termination time is Tν , where ν = min{n : ξn+1 ≥ c}, where ξn = Tn − Tn−1 and Tn are the occurrence times of the Poisson process. Now, at each time Tn , the process either terminates if ξn+1 ≥ c, or it continues until the next renewal epoch if ξn+1 < c. These events are clearly independent of T1 , . . . , Tn .

2.14 Terminating Renewal Processes

143

Under these assumptions, the probability of terminating is 1 − p = P {ξn+1 ≥ c} = e−λc . The conditional distribution of the next renewal period beginning at Tn is F (t) = P {ξn+1 ≤ t|ξn+1 < c} = p−1 (1 − e−λt ),

0 ≤ t ≤ c.

Then from (2.51), the distribution and mean of the waiting time for a gap of size c are P {Tν ≤ t} = e−λc V (t), E[Tν ] = (eλc − 1)/λ. ! Now, assume λc > 1. Then the condition R+ eβt pdF (t) = 1 above for

defining β reduces to λe(β−λ)c = β, for β < λ. Such a β exists as in Figure 1.3 in Chapter 1 for the branching model. Using this formula and integration by parts, we have  teβt pdF (t) = p(cβ − 1)/(β − λ). μ# = [0,c]

Then by Corollary 71, P {Tν > t} =

 1 − β/λ  1 − βc

e−βt + o(e−βt ),

as t → ∞.

Example 73. Cram´er-Lundberg Risk Model. Consider an insurance company that receives capital at a constant rate c from insurance premiums, investments, interest etc. The company uses the capital to pay claims that arrive according to a Poisson process N (t) with rate λ. The claim amounts X1 , X2 , . . . are i.i.d. positive random variables with mean μ, and are independent of the arrival times. Then the company’s capital at time t is N (t)

Zx (t) = x + ct −



Xn ,

t ≥ 0,

n=1

where x is the capital at time 0. An important performance parameter of the company is the probability R(x) = P {Zx (t) ≥ 0, t ≥ 0}, that the capital does not go negative (the company is not ruined). We are interested in approximating this survival probability when the initial capital x is large. Exercise 25 shows that R(x) = 0, regardless of the initial capital x, when c < λμ (the capital input rate is less than the payout rate). We will now consider the opposite case c > λμ. Conditioning on the time and size of the first claim, one can show (e.g., see [37, 92, 94]) that R(x) satisfies a certain differential equation whose corresponding integral equation

144

2 Renewal and Regenerative Processes

is the renewal equation R(x) = R(0) + R G(x),

(2.56)

where R(0) = 1 − λμ/c and G(y) = λc

−1

 0

y

P {X1 > u} du.

The G is a defective distribution with G(∞) = λμ/c < 1. Then applying (2.52) to R(x) = h V (x) = R(0)V (x), we have R(x) → R(0)/(1 − λμ/c) = 1,

as x → ∞.

We now consider the rate at which the “ruin” probability 1 − R(x) converges to 0 as x → ∞. Assume there is a positive β such that  λc−1 eβx P {X1 > x} dx = 1, R+

and that #

μ = λc

−1

 R+

xeβx P {X1 > x} dx < ∞.

Then by Theorem 70 (with R(x), R(0) in place of H(t), h(t)), the probability of ruin has the asymptotic form 1 − R(x) =

1 (1 − λμ/c)e−βx + o(e−βx ), βμ#

as x → ∞.

2.15 Stationary Renewal Processes Recall that a basic property of an ergodic Markov chain is that it is stationary if the distribution of its state at time 0 is its stationary distribution (which is also its limiting distribution). This section addresses the analogous issue of determining an appropriate starting condition for a delayed renewal process so that its increments are stationary in time. We begin by defining the notion of stationarity for stochastic processes and point processes. A continuous-time stochastic process {X(t) : t ≥ 0} on a general space is stationary if its finite-dimensional distributions are invariant under any shift in time: for each 0 ≤ s1 < . . . < sk and t ≥ 0, d

(X(s1 + t), . . . , X(sk + t)) = (X(s1 ), . . . , X(sk )).

(2.57)

2.15 Stationary Renewal Processes

145 d

Remark 74. A Markov process X(t) is stationary if X(t) = X(0), t ≥ 0. This simpler criterion follows as in the proofs of Proposition 52 in Chapter 1 and Exercise 55.  Now, consider a point process N (t) = n 1(τn ≤ t) on R+ , with points at 0 < τ1 < τ2 < . . . Another way of representing this process is by the  family N = {N (B) : B ∈ B+ }, where N (B) = n 1(τn ∈ B) is the number of points τn in the Borel set B. We also define B + t = {s + t : s ∈ B}. The process N is stationary (i.e., it has stationary increments) if, for any B1 , . . . , Bk ∈ B+ , d

(N (B1 + t), . . . , N (Bk + t)) = (N (B1 ), . . . , N (Bk )),

t ≥ 0.

(2.58)

A basic property of a stationary point process is that its mean value function is linear. Proposition 75. If N is a stationary point process and E[N (1)] is finite, then E[N (t)] = tE[N (1)], t ≥ 0. Proof. To see this, consider E[N (s + t)] = E[N (s)] + E[N (s + t) − N (s)] = E[N (s)] + E[N (t)]. This is a linear equation f (s + t) = f (s) + f (t), s, t ≥ 0. The only nondecreasing function that satisfies this linear equation is f (t) = ct for some c. In our case, c = f (1) = E[N (1)], and hence E[N (t)] = tE[N (1)]. We are now ready to characterize stationary renewal processes. Assume that N (t) is a delayed renewal process, where the distribution of ξ1 is G, and the distribution of ξn , n ≥ 2, is F , which has a finite mean μ. The issue is how to select the initial distribution G such that N is stationary. The answer, according to (iv) below, is to select G to be Fe , which is the limiting distribution of the forward and backward recurrence times for a renewal process with inter-renewal distribution F . The following result also shows that the stationarity of N is equivalent to the stationarity of its forward recurrence time process. Theorem 76. The following statements are equivalent. (i) The delayed renewal process N is stationary. (ii) The forward recurrence time process B(t) = TN (t)+1 − t is stationary. (iii) E[N (t)] = t/μ, for t ≥ 0. !t (iv) G(t) = Fe (t) = μ1 0 [1 − F (s)]ds. When these statements are true, P {B(t) ≤ x} = Fe (x), for t, x ≥ 0. Proof. (i) ⇔ (ii): Using Tn = inf{u : N (u) = n}, we have B(t) = TN (t)+1 − t = inf{u − t : N (u) = N (t) + 1} = inf{t : N ((0, t ] + t) = 1}. d

(2.59)

146

2 Renewal and Regenerative Processes d

Consequently, the stationarity property (2.58) of N implies B(t) = B(0), t ≥ 0. Then B is stationary by Remark 74, because it is a Markov process (Exercise 55). Conversely, since N counts the number of times B(t) jumps upward,  1(B(u + t) > B((u + t)−)). (2.60) N (A + t) = u∈A

Therefore, the stationarity of B implies N is stationary. (i) ⇒ (iii): If N is stationary, Proposition 75 ensures E[N (t)] = tE[N (1)]. Also, E[N (1)] = 1/μ since t−1 E[N (t)] → 1/μ by Proposition 32. Therefore, E[N (t)] = t/μ. (iii) ⇒ (iv): Assume E[N (t)] = t/μ. Exercise 53 shows U Fe (t) = t/μ, and so E[N (t)] = U Fe (t). Another expression for this expectation is E[N (t)] =

∞ 

G F (n−1) (t) = G U (t).

n=1

Equating these expressions, we have U Fe (t) = G U (t). Taking the Laplace transform of this equality yields ˆ U ˆ (α), ˆ (α)Fˆe (α) = G(α) U

(2.61) !

ˆ where the hat symbol denotes Laplace transform; e.g., G(α) = R+ e−αt dG(t). ˆ (α) = 1/(1 − Fˆ (α)) is positive. Using this in By Proposition 20, we know U ˆ ˆ (2.61) yields Fe (α) = G(α). Since these Laplace transforms uniquely determine the distributions, we obtain G = Fe . (iv) ⇒ (ii): By direct computation as in Exercise 37, it follows that  [1 − F (t + x − s)]dV (s), (2.62) P {B(t) > x} = 1 − G(t + x) + [0,t]

where V (t) = E[N (t)] = G U (t). Now, the assumption G = Fe , along with U Fe (t) = t/μ from Exercise 53, yield V (t) = G U (t) = U G(t) = U Fe (t) = t/μ. Using this in (2.62), along with a change of variable in the integral, we have P {B(t) > x} = 1 − G(t + x) + Fe (x + t) − Fe (x).

(2.63)

Since G = Fe , this expression is simply P {B(t) > x} = 1 − Fe (x), t ≥ 0. Thus, the distribution of B(t) is independent of t. This condition is sufficient for B(t) to be stationary since it is a Markov process (see Exercise 55). Example 77. Suppose the inter-renewal distribution for the delayed renewal process N is the beta distribution

2.15 Stationary Renewal Processes

 F (t) = 30 0

147 t

s2 (1 − s)2 ds,

t ∈ [0, 1].

The equilibrium distribution associated with F is clearly Fe (t) = 2t − 5t4 + 6t5 − 2t6 ,

t ∈ [0, 1].

Then by Theorem 76, N is stationary if and only if G = Fe . One consequence of Theorem 76 is that Poisson processes are the only nondelayed renewal processes (whose inter-renewal times have a finite mean) that are stationary. Corollary 78. The renewal process N (t) with no delay, and whose interrenewal times have a finite mean, is stationary if and only if it is a Poisson process. Proof. By Theorem 76 (vi), N (t) is stationary if and only if E[N (t)] = t/μ, t ≥ 0, which is equivalent to N (t) being a Poisson process by Remark 21. An alternate proof is to apply Theorem 76 (iii) and use the fact (Exercise 4 in Chapter 3) that F = Fe if and only if F is an exponential distribution. Here is another useful stationarity property. Remark 79. If N (t) is a stationary renewal process, then   1 t E f (Tn ) = f (s)ds. μ 0 n=1 (t)

N 

This follows by Theorem 22 and E[N (t)] = t/μ. Many stationary processes arise naturally as functions of stationary processes (two examples are in the proof of Theorem 76). A general statement to this effect is as follows; it is a consequence of the definition of stationarity. Remark 80. Hereditary Property of Stationarity. Suppose X(t) is a stationary process. Then the process Y (t) = f (X(t)) is also stationary, where f is a function on the state space of X to another space. More generally, Y (t) = g({X(s+t) : s ≥ 0}) is stationary, where g is a function on the space of sample paths of X to some space. Analogously, N is a stationary point process if, for any bounded set B and t > 0, N (B + t) = g({X(s + t) : s ≥ 0}, B)

(2.64)

(see for instance (2.59) and (2.60)). Example 81. Let X(t) be a delayed regenerative process (e.g., a continuoustime Markov chain as in Chapter 4) over the times 0 < T1 < T2 < . . . at which X(t) enters a special state x∗ . Let N denote the point process of these

148

2 Renewal and Regenerative Processes

times. If X(t) is stationary, then N is a stationary renewal process. This follows since, like (2.64),  1(X((s + t)−) = x∗ , X(s + t) = x∗ ). N (B + t) = s∈B

Although the bounded set B may be uncountable, only a finite number of its values will contribute to the sum. Because a stationary renewal process N (t) has a stationary forward recurrence time process, it seems reasonable that the backward recurrence time process A(t) = t − TN (t) would also be stationary. This is not true, since the distribution of A(t) is not independent of t; in particular, A(t) = t, for t < T1 . However, there is stationarity in the following sense. Remark 82. Stationary Backward Recurrence Time Process. Suppose the stationary renewal process is extended to the negative time axis with (artificial or virtual) renewals at times . . . < T−1 < T0 < 0. One can think of the renewals occurring since the beginning of time at −∞. Consistent with the definition above, the backward recurrence process is A(t) = t − Tn ,

if t ∈ [Tn , Tn+1 ), for some n ∈ R.

Assuming N is stationary on R+ , the time A(0) = T1 to the first renewal has the distribution Fe . Then one can show, as we proved (i) ⇔ (ii) in Theorem 76, that the process {A(t) : t ∈ R} is stationary with distribution Fe .

2.16 Refined Limit Laws We will now describe applications of the key renewal theorem for functions that are not asymptotically constant. The applications of the renewal theorem we have been discussing are for limits of functions H(t) = U h(t) that converge to a constant (i.e., H(t) = c + o(1)). However, there are many situations in which H(t) tends to infinity, but the key renewal theorem can still be used to describe limits of the form H(t) = v(t) + o(1) as t → ∞, where the function v(t) is the asymptotic value of H(t). For instance, a SLLN Z(t)/t → b suggests E[Z(t)] = bt + c + o(1) might be true, where the constant c gives added information on the convergence. In this section, we discuss such limit theorems. We first note that an approach for considering limits H(t) = v(t) + o(1) is simply to consider a renewal equation for the function H(t) − v(t) as follows. Lemma 83. Suppose H(t) = U h(t) is a solution of a renewal equation for a non-arithmetic distribution F , and v(t) is a real-valued function on R that

2.16 Refined Limit Laws

149

is bounded on finite intervals and is 0 for negative t. Then  1 h(s) ds + o(1), as t → ∞, H(t) = v(t) + μ R+

(2.65)

provided h(t) = h(t) − v(t) + F v(t) is DRI. In particular, for a linear function v(t) = bt,  1 b(σ 2 + μ2 ) + H(t) = bt + (h(s) − bμ) ds + o(1), as t → ∞, (2.66) 2μ μ R+ where σ 2 is the variance of F , provided h(t) − bμ is DRI. Proof. Clearly H − v satisfies the renewal equation H − v = (h − v + F v) + F (H − v). Then H − v = U h by Proposition 31, and its limit (2.65) is given by the key renewal theorem. ! Next, suppose v(t) = bt and h(t) − bμ is DRI. Then using μ = R+ [1 − F (x)] dx and the change of variable x = t − s in the integral below, we have  h(t) = h(t) − bt + b

0

t

F (t − s) ds

= h(t) − bμ + bg(t), where g(t) =

!∞ t

[1 − F (x)] dx. Now g(t) is continuous and decreasing and 

∞

g(t) dt = 0

1 2



t2 dF (t) =

R+

σ 2 + μ2 . 2

(2.67)

Then g(t) is DRI by Proposition 88 (a), and hence h(t) = h(t) − bμ + bg(t) is DRI. Thus, by what we already proved, (2.65) is true but it reduces to (2.66) in light of (2.67). Our first use of the preceding result is a refinement of t−1 U (t) → 1/μ from Proposition 32. Proposition 84. If N (t) is a renewal process whose inter-renewal times have a non-arithmetic distribution with mean μ and variance σ 2 , then U (t) = t/μ + (σ 2 + μ2 )/2μ2 + o(1),

as t → ∞.

Proof. This follows by Lemma 83 with H(t) = U (t), h(t) = 1, and v(t) = t/μ (that h(t) − bμ is DRI need not be verified since it equals 0). We will now apply Lemma 83 to a real-valued stochastic process Z(t) whose sample paths are right-continuous with left-hand limits. Assume

150

2 Renewal and Regenerative Processes

that Z(t), has crude regenerative increments at T in the sense that E[Z(T + t) − Z(T )|T ] = E[Z(t)],

t ≥ 0.

(2.68)

If Z(t) has regenerative increments over Tn , then Z(t) has crude regenerative increments at T1 . Theorem 85. For the process Z(t) defined above, let M = sup{|Z(T ) − Z(t)| : t ≤ T }. !T If the expectations of M , M T , T 2 , |Z(T )|, and 0 |Z(s)|ds are finite, then as t → ∞, E[Z(t)] = at/μ + a(σ 2 + μ2 )/2μ2 + c + o(1),

!  T where a = E[Z(T )] and c = μ1 E 0 Z(s)ds − T Z(T ) .

(2.69)

Proof. Because Z(t) has crude regenerative increments, it would be natural that that t−1 E[Z(t)] → a/μ. So to prove (2.69), we will apply Lemma 83 with v(t) = at/μ. We first derive a renewal equation for E[Z(t)]. Conditioning on T ,  E[Z(t)] = E[Z(t)1(T > t)] + E[Z(t)|T = s]dF (s). [0,t]

Using E[Z(t)|T = s] = E[Z(t − s)] + E[Z(s)|T = s] from assumption (2.68) and some algebra, it follows that the preceding is a renewal equation H = h + F H, where H(t) = E[Z(t)] and

 h(t) = a + E (Z(t) − Z(T ))1(T > t) . Now, by Lemma 83 for v(t) = at/μ, we have  1 σ 2 + μ2 E[Z(t)] = at/μ + + g(s) ds + o(1), 2μ2 μ R+

as t → ∞,

(2.70)

 provided g(t) = h(t) − a = E (Z(t) − Z(T ))1(T > t) is DRI. Clearly |g(t)| ≤ b(t) = E[M 1(T > t)]. ! Now, b(t) ↓ 0; and as in (2.25), R+ b(s)ds = E[M T ] is finite. Then b(t) is DRI by Proposition 88 (a). Hence g(t) is also DRI by Proposition 88 (c). Finally, observe that 

 T  g(t) dt = E Z(s)ds − T Z(T )] . R+

0

Substituting this formula in (2.70) proves (2.69).

2.17 Proof of the Key Renewal Theorem*

151

2.17 Proof of the Key Renewal Theorem∗ This section proves the key renewal theorem by applying Blackwell’s theorem, which is proved in the next section. The key renewal theorem involves real-valued functions that are integrable on the entire axis R+ as follows. Definition 86. Similarly to the definition of a Riemann integral on a finite interval, it is natural to approximate the integral of a real-valued function h(t) on the entire domain R+ over a grid 0, δ, 2δ, . . . by the upper and lower Riemann sums I δ (h) = δ Iδ (h) = δ

∞  k=0 ∞ 

sup{h(s) : kδ ≤ s < (k + 1)δ}, inf{h(s) : kδ ≤ s < (k + 1)δ}.

k=0

The function h(t) is directly Riemann integrable (DRI) if I δ (h) and Iδ (h) are finite for each δ, and they both converge to the same limit as δ → 0. The limit is necessarily the usual Riemann integral   t h(s)ds = lim h(s)ds, t→∞

R+

0

where the last integral is the limit of the Riemann sums on [0, t]. A DRI function is clearly Riemann integrable in the usual sense, but the converse is not true; see Exercise 28. From the definition, it is clear that h(t) is DRI if it is Riemann integrable and it is 0 outside a finite interval. Also, h(t) is DRI if and only if its positive and negative parts h+ (t) and h− (t) are both DRI. Further criteria for DRI are given in Proposition 88 and Exercise 33. We are now ready for the main result. Theorem 87. (Key Renewal Theorem) If h(t) is DRI and F is nonarithmetic, then  1 h(s)ds. lim U h(t) = t→∞ μ R+ Proof. Fix δ > 0 and define hk = sup{h(s) : kδ ≤ s < (k + 1)δ} and h(t) =

∞ 

hk 1(kδ ≤ t < (k + 1)δ).

k=0

∗

The star at the end of a section title means the section contains advanced material that need not be covered in a first reading.

152

2 Renewal and Regenerative Processes

Define h(t) and hk similarly, with sup replaced by inf. Obviously, U h(t) ≤ U h(t) ≤ U h(t).

(2.71)

Letting dk (t) = U (t − kδ) − U (t − (k + 1)δ), we can write (like (2.19)) U h(t) =

∞ 

hk dk (t).

k=0

Now limt→∞ dk (t) = δ/μ by Theorem 33, and dk (t) ≤ U (δ) by Exercise 28. Then by the dominated convergence theorem (see the Appendix, Theorem 14) and the DRI property of h, ∞ δ hk δ→0 μ

lim lim U h(t) = lim

δ→0 t→∞

k=0

1 1 = lim I δ (h) = δ→0 μ μ

 h(s)ds. R+

This (double) limit is the same with h(t) and I δ (h) replaced by h(t) and the upper and lower bounds in (2.71) for U h(t) have the Iδ (h). Therefore, ! same limit μ1 R+ h(s)ds, and so U h(t) must also have this limit. This proves the assertion. We end this section with criteria for a function to be DRI. Proposition 88. Any one of the following conditions is sufficient for h(t) to be DRI. (a) h(t) ≥ 0 is decreasing and is Riemann integrable on R+ . (b) h(t) is Riemann integrable on [0, a] for each a, and I δ (h) < ∞ for some δ > 0. (c) h(t) is continuous except possibly on a set of Lebesgue measure 0, and |h(t)| ≤ b(t), where b(t) is DRI. Proof. Suppose condition (a) holds. Since the usual Riemann integral of h on R+ exists, we have  Iδ (h) ≤

R+

h(s)ds ≤ I δ (h).

Also, the decreasing property of h(t) implies I δ (h) − Iδ (h) = δh(0) → 0 as δ → 0. These observations prove h(t) is DRI. Next, suppose (b) holds. We will write I δ (h) = I δ [0, a/δ) + I δ [a/δ, ∞), ∞ where I δ [x, ∞) = δ k= x−1 sup{h(s) : kδ ≤ s < (k + 1)δ}. We will use a similar expression for Iδ (h). Since h(t) is Riemann integrable on [0, a], it

2.18 Proof of Blackwell’s Theorem*

153

follows that I δ [0, a/δ) and Iδ [0, a/δ) both converge to Therefore, I δ (h) − Iδ (h) = o(1) + I δ [a/δ, ∞) − Iδ [a/δ, ∞),

!a 0

h(s)ds as δ → 0.

as δ → 0.

(2.72)

Let γ be such that I γ (h) < ∞. Then for any ε > 0, there is a large enough a such that I γ [a/γ, ∞) < ε. Then clearly, for sufficiently small δ, Iδ [a/δ, ∞) ≤ I δ [a/δ, ∞) ≤ I γ [a/γ, ∞) < ε. Using this in (2.72), we have I δ (h) − Iδ (h) ≤ o(1) + 2ε,

as δ → 0.

Since this holds for any ε, it follows that h(t) is DRI. Finally, (c) implies (b) since I δ (h) ≤ I δ (b). Thus h(t) is DRI.

2.18 Proof of Blackwell’s Theorem* This section describes a coupling proof of Blackwell’s theorem. The proof is more complicated than the one we presented above for arithmetic interrenewal times. The classical proof of Blackwell’s theorem based on analytical properties of the renewal function and integral equations is in Feller (1971). Lindvall (1977) and Athreya, McDonald and Ney (1978) gave another probabilistic proof involving “coupling” techniques. A nice review of various applications of coupling is in Lindvall (1992). A recent refinement of the coupling proof is given in Durrett (2005). The following is a sketch of his presentation when the inter-renewal time has a finite mean (he gives a different proof for the case of an infinite mean). Let N (t) be a renewal process with renewal times Tn whose inter-renewal times ξn have a non-arithmetic distribution and a finite mean μ. For convenience, we will write Blackwell’s theorem (Theorem 33) as lim E[N (t, t + a]] = a/μ,

t→∞

(2.73)

where N (t, t + a] = N (t + a) − N (t). Now, this statement would trivially hold if N (t) were a stationary renewal process, since in this case E[N (t, t + a]] would equal a/μ by Proposition 75. So if one could construct a version of N (t) that approximates a stationary process as close as possible, then (2.73) would be true. That is the approach in the proof that we now describe. On the same probability space as N (t), let N  (t) be a stationary renewal process with renewal times Tn , whose inter-renewal ξn times for n ≥ 2 have the same distribution as the ξn . The first and most subtle part of the proof

154

2 Renewal and Regenerative Processes

is to construct a third renewal process N  (t) on the same probability space that is equal in distribution to the original process N (t) and approximates the stationary process N  (t). We will not describe the construction of these processes, but only specify their main properties. In particular, for a fixed ε > 0, the proof begins by defining random indices ν and ν  such that |Tν − Tν  | < ε. Then a third renewal process N  (t) is defined (on the same probability space) with inter-renewal times ξ1 , . . . , ξν , ξν  , ξν  +1 . . . This process has the following properties: d

(a) {N  (t) : t ≥ 0} = {N (t) : t ≥ 0} (i.e., their finite-dimensional distributions are equal). (b) On the event {Tν ≤ t}, N  (t + ε, t + a − ε] ≤ N  (t, t + a] ≤ N  (t − ε, t + a + ε].

(2.74)

This construction is an ε-coupling in that N  (t) is a coupling of N (t) that is within ε of the targeted stationary version N  (t) in the sense of condition (b). With this third renewal process in hand, the rest of the proof is as follows. Consider the expectation E[N (t, t + a]] = E[N  (t, t + a]] = V1 (t) + V2 (t), where

 V1 (t) = E N  (t, t + a]1(Tν ≤ t) ,

(2.75)

 V2 (t) = E N  (t, t + a]1(Tν > t) .

Condition (b) and E[N  (c, d]] = (d − c)/μ (due to the stationarity) ensure

 V1 (t) ≤ E N  (t − ε, t + a + ε]1(Tν ≤ t) ≤ (a + 2ε)μ. Next, observe that E[N  (t, t + a]|Tν > t] ≤ E[N  (a)], since the worse-case scenario is that there is a renewal at t. This and condition (b) yield V2 (t) ≤ P {Tν > t}E[N  (a)]. Similarly,

 V1 (t) ≥ E N  (t + ε, t + a − ε] − N  (t, t + a]1(Tν > t) ≥ (a − 2ε)/μ − P {Tν > t}E[N  (a)]. Here we take ε < a/2, so that t + ε < t + a − ε. Combining the preceding inequalities with (2.75), and using P {Tν > t} → 0 as t → ∞, it follows that (a − 2ε)/μ + o(1) ≤ E[N (t, t + a]] ≤ (a + 2ε)/μ + o(1).

2.19 Stationary-Cycle Processes*

155

Since this is true for arbitrarily small ε, we obtain E[N (t, t + a]] → a/μ, which is Blackwell’s result.

2.19 Stationary-Cycle Processes* Most of the results above for regenerative processes also apply to a wider class of regenerative-like processes that we will now describe. For this discussion, suppose {X(t) : t ≥ 0} is a continuous-time stochastic process with a general state space S, and N (t) is a renewal process defined on the same probability space. As in Section 2.8, we let ζn = (ξn , {X(Tn−1 + t) : 0 ≤ t < ξn }) denote the segment of these processes on the interval [Tn−1 , Tn ). Then {ζn+k : k ≥ 1} is the future of (N (t), X(t)) beginning at time Tn . This is what an observer of the processes would see beginning at time Tn . Definition 89. The process X(t) is a stationary-cycle process over the times Tn if the future {ζn+k : k ≥ 1} of (N (t), X(t)) beginning at any time Tn is independent of T1 , . . . , Tn , and the distribution of this future is independent of n. Discrete-time and delayed stationary-cycle processes are defined similarly. The defining property ensures that the segments ζn form a stationary sequence, whereas for a regenerative process, the segments are i.i.d. Also, for a regenerative process X(t), its future {ζn+k : k ≥ 1} beginning at any time Tn is independent of the entire past {ζk : k ≤ n} (rather than only T1 , . . . , Tn as in the preceding definition). All the strong laws of large numbers for regenerative processes in this chapter also hold for stationary-cycle processes. A law’s limiting value would be a constant as usual when ζn is ergodic (as in Section 4.18 in Chapter 4), but the value would be random when ζn is not ergodic. We will not get into these details. As in Section 2.10, one can define processes with stationary-cycle increments. Most of the results above such as the CLT have obvious extensions to these more complicated processes. We end this section by commenting on limiting theorems for probabilities and expectations of stationary-cycle processes. Remark 90. Theorem 45 and Corollary 46 are also true for stationary-cycle processes. This follows since such a process satisfies the crude-regeneration property in Theorem 41 leading to Theorem 45 and Corollary 46. There are many intricate stationary-cycle processes that arise naturally from systems that involve stationary and regenerative phenomena. Here is an elementary illustration.

156

2 Renewal and Regenerative Processes

Example 91. Regenerations in a Stationary Environment. Consider a process X(t) = g(Y (t), Z(t)) where Y (t) and Z(t) are independent processes and g is a function on their product space. Assume Y (t) is a regenerative process over the times Tn (e.g., an ergodic continuous-time Markov chain as in Chapter 4) with a metric state space S. Assume Z(t) is a stationary process. One can regard X(t) = g(Y (t), Z(t)) as a regenerative-stationary reward process, where g(y, z) is the reward rate from operating a system in state y in environment z. Now, the segments ζn defined above form a stationary process, and hence X(t) is a stationary-cycle process. In light of Remark 90, we can describe the limiting behavior of X(t) as we did for regenerative processes. In particular, assuming for simplicity that g is real-valued and bounded, Theorem 45 for stationary-cycle processes tells us that   1 T1 lim E[X(t)] = E g(Y (s), Z(s))ds . t→∞ μ 0

2.20 Exercises Exercise 1. Show that if X is nonnegative with distribution F , then  (1 − F (x))dx. E[X] = R+

!x  One approach is to use E[X] = R+ 0 dy dF (x). (For an integer-valued X,  the preceding formula is E[X] = ∞ n=0 P {X > n} .) For a general X with finite mean, use X = X + − X − to prove  0  (1 − F (x))dx − F (x)dx. E[X] = !

R+

−∞

Exercise 2. Bernoulli Process. Consider a sequence of independent Bernoulli trials in which each trial results in a success or failure with respective probabilities p and q = 1 − p. Let N (t) denote the number of successes in t trials, where t is an integer. Show that N (t) is a discrete-time renewal process, called a Bernoulli Process. (The parameter t may denote discrete-time or any integer referring to sequential information.) Justify that the inter-renewal times have the geometric distribution P {ξ1 = n} = pq n−1 , n ≥ 1. Find the distribution and mean of N (t), and do the same for the renewal time Tn . Show that the moment generating function of Tn is E[eαTn ] =

 peα n , 1 − qeα

0 < α < − log q.

2.20 Exercises

157

Exercise 3. Exercise 1 in Chapter 3 shows that an exponential random variable X satisfies the memoryless property P {X > s + t|X > s} = P {X > t},

s, t > 0.

Prove the analogue P {X > τ + t|X > τ } = P {X > t}, for t > 0, where τ is a positive random variable independent of X. Show that, for a Poisson process N (t) with rate λ, the forward recurrence time B(t) = TN (t)+1 − t at time t has an exponential distribution with rate λ. Hint: condition on TN (t) . Consider the forward recurrence time B(τ ) at a random time τ independent of the Poisson process. Show that B(τ ) also has an exponential distribution with rate λ. Exercise 4. A system consists of two components with independent lifetimes X1 and X2 , where X1 is exponentially distributed with rate λ, and X2 has a uniform distribution on [0, 1]. The components operate in parallel, and the system lifetime is max{X1 , X2 } (the system is operational if and only if at least one component is working). When the system fails, it is replaced by another system with an identical and independent lifetime, and this is repeated indefinitely. The number of system renewals over time forms a renewal process N (t). Find the distribution and mean of the system lifetime. Find the distribution and mean of N (t) (reduce your formulas as much as possible). Determine the portion of time that (a) two components are working, (b) only type 1 component is working, and (c) only type 2 component is working. Exercise 5. Continuation. In the context of the preceding exercise, a typical system initially operates for a time Y = min{X1 , X2 } with two components and then operates for a time Z = max{X1 , X2 } − Y with one component. Thereupon it fails. Find the distributions and means of Y and Z. Find the distribution of Z conditioned that X1 > X2 . You might want to use the memoryless property of the exponential distribution in Exercise 3. Find the distribution of Z conditioned that X2 > X1 . Exercise 6. Let N (t) denote a renewal process with inter-renewal distribution F and consider the number of renewals N (T ) in an interval (0, T ] for some random time T independent of N (t). For instance, N (T ) might represent the number of customers that arrive at a service station during a service time T . Find general expressions for the mean and distribution of N (T ). Evaluate these expressions for the case in which T has an exponential distribution with rate μ and F = G2 , where G is an exponential distribution with rate λ. Exercise 7. Let X(t) denote the cyclic renewal process in Example 8, where F0 , . . . , FK−1 are the sojourn distributions in states 0, 1, . . . , K − 1. Assume p = F0 (0) > 0, but Fi (0) = 0, for i = 1, . . . , K − 1. Let Tn denote the times at which the process X(t) jumps from state K − 1 directly to state 1 (i.e., it

158

2 Renewal and Regenerative Processes

spends no time in state 0). Justify that the Tn form a delayed renewal process with inter-renewal distribution F (t) = p

∞ 

F1 · · · FK−1 F˜ j (t),

j=0

where F˜ (t) = F˜0 F1 · · · FK−1 (t), and F˜0 (t) is the conditional distribution of the sojourn time in state 0 given it is positive. Specify a formula for F˜0 (t), and describe what F˜ (t) represents. Exercise 8. Large Inter-renewal Times. Let N (t) denote a renewal process with inter-renewal distribution F . Of interest are occurrences of inter-renewal times that are greater than a value c, assuming F (c) < 1. Let T˜n denote the subset of times Tn for which ξn > c. So T˜n equals some Tk if ξk > c. (Example 72 addresses a related problem of determining the waiting time for a gap of size c in a Poisson process.) Show that T˜n are delayed renewal times and the inter-renewal distribution has the form F˜ (t) =

∞ 

Fck G(t),

k=0

where Fc (t) = F (t)/F (c), 0 ≤ t ≤ c (the conditional distribution of an interrenewal time given that it is ≤ c), and specify the distribution G(t) as a function of F . Exercise 9. Partitioning and Thinning of a Renewal Process. Let N (t) be a renewal process with inter-renewal distribution F . Suppose each renewal time is independently assigned to be a type i renewal with probability pi , for i = 1, . . . , m, where p1 + · · · + pm = 1. Let Ni (t) denote the number of type i renewals  up to time t. These processes form a partition of N (t) in that N (t) = m i=1 Ni (t). Each Ni (t) is a thinning of N (t), where pi is the probability that a point of N (t) is assigned to Ni (t). Show that Ni (t) is a renewal process with inter-renewal distribution Fi (t) =

∞ 

(1 − pi )k−1 pi F k (t).

k=1

Show that, for n = n1 + · · · nm , P {N1 (t) = n1 , . . . , Nm (t) = nm }

 n! pn1 1 · · · pnmm F (n) (t) − F (n+1) (t) . = n1 ! · · · nm ! For m = 2, specify an F for which N1 (t) and N2 (t) are not independent. Exercise 10. Multi-type Renewals. An infinite number of jobs are to be processed one-at-a-time by a single server. There are m types of jobs, and the

2.20 Exercises

159

probability that any job is of type i is pi , where p1 + · · · + pm = 1. The service time of a type i job has a distribution Fi with mean μi . The service times and types of the jobs are independent. Let N (t) denote the number of jobs completed by time t. Show that N (t) is a renewal process and specify its inter-renewal distribution and mean. Let Ni (t) denote the number of type i jobs processed up to time t. Show that Ni (t) is a delayed renewal process and specify limt→∞ t−1 Ni (t). Exercise 11. Continuation. In the context of Exercise 10, let X(t) denote the type of job being processed at time t. Find the limiting distribution of X(t). Find the portion of time devoted to type i jobs. Exercise 12. Continuation. Consider the multi-type renewal process with two types of renewals that have exponential distributions with rates λi , and type i occurs with probability pi , , i = 1, 2. Show that the renewal function has the density U  (t) =

λ1 λ2 + p1 p2 (λ1 − λ2 )2 e−(p1 λ2 +p2 λ1 )t , p1 λ2 + p2 λ1

t > 0.

Exercise 13. System Availability. The status of a system is represented by an alternating renewal process X(t), where the mean sojourn time in a working state 1 is μ1 and the mean sojourn time in a non-working state 0 is μ0 . The system availability is measured by the portion of time it is working, which is !t limt→∞ t−1 0 X(s)ds. Determine this quantity and show that it is equal to !T the cycle-availability measured by limn→∞ Tn−1 0 n X(s)ds. Exercise 14. Integrals of Renewal Processes. Suppose N (t) is a renewal process with renewal times Tn and μ = E[T1 ]. Prove

 E 0

Tn

 N (s) ds = μn(n − 1)/2.

For any non-random t > 0, it follows by Fubini’s theorem that   t

 t N (s) ds = E[N (s)] ds. E 0

0

Assuming τ is an exponential random variable independent of N with rate γ, prove

 τ   E N (s) ds = e−γt E[N (t)] dt. 0

R+

Show that if N is a Poisson process with rate λ, then the preceding expectation equals λ/γ 2 . (Integrals like these are used to model holding costs; see Section 2.12 and the next exercise.)

160

2 Renewal and Regenerative Processes

Exercise 15. Continuation. Items arrive to a service station according to a Poisson process N (t) with rate λ. The items are stored until m have accumulated. Then the m items are served in a batch. The service time is exponentially distributed with rate γ. During the service, items continue to arrive. There is a cost hi per unit time of holding i customers in the system. Assume the station is empty at time 0. Find the expected cost C1 of holding the customers until m have arrived. Find the expected cost C2 for holding the added arrivals in the system during the service. Exercise 16. Customers arrive to a service system according to a Poisson process with rate λ. The system can only serve one customer at a time and, while it is busy serving a customer, arriving customers are blocked from getting service (they may seek service elsewhere or simply go unserved). Assume the service times are independent with common distribution G and are independent of the arrival process. For instance, a contractor may only be able to handle one project at a time (or a vehicle can only transport one item at a time). Determine the following quantities: (a) The portions of time the system is busy, and not busy. (b) The number of customers per unit time that are served. (c) The portion of customers that are blocked from service. Exercise 17. Delayed Renewals. A point process N (t) is an m-step delayed renewal process if the inter-occurrence times ξm+k , for k ≥ 1, are independent with a common distribution F , and no other restrictions are placed on ξ1 , . . . , ξm . That is, Nm (t) = N (t) − N (Tm ), for t ≥ Tm is a renewal process. Show that Corollary 11 and Theorem 13 hold for such processes. Use the fact that N (t) is asymptotically equivalent to Nm (t) in that Nm (t)/N (t) = 1 − N (Tm )/N (t) → 1,

a.s. as t → ∞.

Exercise 18. For a point process N (t) that is not simple, show that if t−1 N (t) → 1/μ as t → ∞, then n−1 Tn → μ, as n → ∞. Hint: For a fixed positive constant c, note that N ((Tn − c)+ ) ≤ n ≤ N (Tn ). Divide these terms by Tn and take limits as n → ∞. Exercise 19. Age Replacement Model. An item (e.g., battery, vehicle, tool, or electronic component) whose use is needed continuously is replaced whenever it fails or reaches age a, whichever comes first. The successive items are independent and have the same lifetime distribution G. The cost of a failure is cf dollars and the cost of a replacement at age a is cr . Show that the average cost per unit time is  a C(a) = [cf G(a) + cr (1 − G(a))]/ (1 − G(s))ds. 0

Find the optimal age a that minimizes this average cost.

2.20 Exercises

161

Exercise 20. Point Processes as Jump Processes. Consider a point process N (t) = ∞ k=1 1(Tk ≤ t), where T1 ≤ T2 ≤ · · · . It can also be formulated as an integer-valued jump process of the form N (t) =

∞ 

νn 1(Tˆn ≤ t),

n=1

where Tˆn are the “distinct” times at which N (t) takes a jump, and νn is the size ofthe jump. That is, Tˆn = min{Tk : Tk > Tˆn−1 }, where Tˆ0 = 0, and ∞ νn = k=1 1(Tk = Tˆn ), n ≥ 1. For instance, suppose Tn are times at which data packets arrive to a computer file. Then Tˆn are the times at which batches of packets arrive, and at time Tˆn , a batch of νn packets arrive. Suppose Tˆn are renewal times, and νn are i.i.d. and independent of the Tˆn . Show that the number of packets that arrive per unit time is E[ν1 ]/E[Tˆ1 ] a.s., provided these expectations are finite. Next, assume Tˆn form a Poisson process with rate λ, and νn has a Poisson distribution. Find E[N (t)] by elementary reasoning, and then show that N (t) has a Poisson distribution.  Exercise 21. Batch Renewals. Consider times Tn = nk=1 ξk , where the ξk are i.i.d. with distribution F and F (0) = P {ξk = 0} > 0. The associated point process N (t) is a renewal process with instantaneous renewals (or batch  ˆ ν renewals). In the notation of Exercise 20, N (t) = ∞ n=1 n 1(Tn ≤ t), where ˆ νn is the number of renewals exactly at time Tn . Specify the distribution of νn . Are the νn i.i.d.? Are they independent of Tˆn ? Specify the distribution of Tˆ1 in terms of F . Exercise 22. Prove E[TN (t) ] = μE[N (t) + 1] − E[ξN (t) ]. If N (t) is a Poisson process, show that E[TN (t) ] < μE[N (t)]. Exercise 23. Little Law. In the context of the Little law in Theorem 57, show that if L and W exist, then λ exists and L = λW . Exercise 24. Superpositions of Renewal Processes. Let N1 (t) and N2 (t) be independent renewal processes with the same inter-renewal distribution, and consider the sum N (t) = N1 (t) + N2 (t) (sometimes called a superposition). Assuming that N (t) is a renewal process. prove that it is a Poisson process if and only if N1 (t) and N2 (t) are Poisson processes. Exercise 25. Production-Inventory Model. Consider a production-inventory system that produces a product at a constant rate of c units per unit time and the items are put in inventory to satisfy demands. The products may be discrete or continuous (e.g., oil, chemicals). Demands occur according to a Poisson process N (t) with rate λ, and the demand quantities X1 , X2 , . . . are independent, identically distributed positive random variables with mean μ,

162

2 Renewal and Regenerative Processes

and are independent of the arrival times. Then the inventory level at time t would be N (t)  Xn , t ≥ 0, Zx (t) = x + ct − n=1

where x is the initial inventory level. Consider the probability R(x) = P {Zx (t) ≥ 0, t ≥ 0} of never running out of inventory. Show that if c < λμ, then R(x) = 0 no matter how high the initial inventory level x is. Hint: apply a SLLN to show that Zx (t) → −∞ as t → ∞ if c < λμ, where x is fixed. Find the limit of Zx (t) as t → ∞ if c > λμ. (The process Zx (t) is a classical model of the capital of an insurance company; see Example 73.) Exercise 26. Let H(t) = E[N (t)−N (t−a)1(a ≤ t)]. Find a renewal equation that H satisfies. Exercise 27. Non-homogeneous Renewals. Suppose N (t) is a point process with distrion R+ whose inter-point times ξn = Tn − Tn−1 are independent ∞ butions Fn . Assuming it is finite, prove that E[N (t)] = n=1 F1 · · · Fn (t). Exercise 28. Subadditivity of Renewal Function. Prove that U (t + a) ≤ U (a) + U (t),

a, t ≥ 0.

Hint: Use a ≤ TN (a)+1 in the expression N (t + a) − N (a) =

∞ 

1(TN (a)+k ≤ t + a).

k=1

Exercise 29. Arithmetic Blackwell Theorem. The proof of Theorem 33 for arithmetic inter-arrival distributions was proved under the standard condition that p0 = F (0) = 0. Prove the same theorem when 0 < p0 < 1. Use a similar argument including the fact that renewals occur in batches and a batch size has a geometric distribution with parameter 1 − p0 . Exercise 30. Elementary Renewal Theorem via Blackwell. Prove the elementary renewal theorem (Theorem 32) by an application of Blackwell’s theorem. One approach, for non-arithmetic F , is to use E[N (t)] =

t 

[U (k) − U (k − 1)] + E[N (t)] − E[N (t)].

k=1

Then use the fact n−1

n

k=1 ck

→ c when ck → c.

Exercise 31. Arithmetic Key Renewal Theorem. Represent U h(u + nd) as a sum like (2.19), and then prove Theorem 37 by applying Blackwell’s theorem.

2.20 Exercises

163

∞ Exercise 32. Let h(t) = ∞ n=1 an 1(n−εn ≤ t < n+εn ), where an → ∞ and 1/2 > εn ↓ 0 such that n=1 an εn < ∞. Show that h is Riemann integrable, but not DRI. Exercise 33. Prove that a continuous function h(t) ≥ 0 is DRI if and only if I δ (h) < ∞ for some δ > 0. The next eight exercises concern the renewal process trinity: the backward and forward recurrence times A(t) = t − TN (t) , B(t) = TN (t)+1 − t, and the length L(t) = ξN (t)+1 = A(t) + B(t) of the renewal interval containing t. Assume the inter-renewal distribution is non-arithmetic. Exercise 34. Draw a typical sample path for each of the processes A(t), B(t), and L(t). Exercise 35. Prove that B(t) is a Markov process by showing it satisfies the following Markov property, for x, y, t, u ≥ 0: P {B(t + u) ≤ y|B(s) : s < t, B(t) = x} = P {B(u) ≤ y|B(0) = x}. Exercise 36. Formulate a renewal equation that P {B(t) > x} satisfies. Exercise 37. Bypassing a renewal equation. Use! Proposition 40 (without using a renewal equation) to prove P {B(t) > x} = [0,t] [1−F (t+x−s)]dU (s). Exercise 38. Prove E[B(t)] = μE[N (t) + 1] − t. Assuming F has a finite variance σ 2 , prove lim E[A(t)] = lim E[B(t)] =

t→∞

t→∞

σ 2 + μ2 . 2μ

Is this limit also the mean of the limiting distribution Fe (t) = of A(t) and B(t)?

1 μ

!t

0 [1−F (s)]ds

Exercise 39. Inspection Paradox. Consider the length L(t) = ξN (t)+1 of the renewal interval at any time t (this is what an inspector of the process would see at time t). Prove the paradoxical result that L(t) is stochastically larger than the length ξ1 of a typical renewal interval; that is P {L(t) > x} ≥ P {ξ1 > x},

t, x ≥ 0.

This inequality is understandable upon observing that the first probability is for the event that a renewal interval bigger than x “covers” t, and this is more likely to happen than a fixed renewal interval being bigger than x. A consequence of this result is E[L(t] ≥ E[ξ1 ], which is often a strict inequality. Suppose μ = E[T1 ] and σ 2 = Var[T ! x1 ] are finite. Recall from (2.34) that the limiting distribution of L(t) is μ1 0 sdF (s). Derive the mean of this distribution (as a function of μ and σ 2 ), and show it is ≥ μ.

164

2 Renewal and Regenerative Processes

Show that if N (t) is a Poisson process with rate λ, then E[L(t)] = λ−1 [2 − (1 + λt)e−λt ]. In this case, E[L(t] > E[ξ1 ]. Exercise 40. Prove limt→∞ P {A(t)/L(t) ≤ x} = x, 0 ≤ x ≤ 1. Prove this result with B(t) in place of A(t). Exercise 41. Show that lim E[A(t)k B(t) (A(t) + B(t))m ] =

t→∞

E[T1k++m ]  . μ(k +  + 1) k+ k

Find the limiting covariance, limt→∞ Cov(A(t), B(t)). ˜ be a realExercise 42. Delayed Versus Non-delayed Regenerations. Let X(t) ˜ valued delayed regenerative process over Tn . Then X(t) = X(T1 +t), t ≥ 0 is a ˜ is bounded, show that if limt→∞ E[X(t)] regenerative process. Assuming X(t) ˜ exists (such as by Theorem 45), then E[X(t)] has the same limit. Hint: Take the limit as t → ∞ of  ˜ ˜ E[X(t − s)]dF (s) + E[X(t)1(T E[X(t)] = 1 > t)]. [0,t]

Exercise 43. Dispatching System. Items arrive at a depot (warehouse or computer) at times that form a renewal process with finite mean μ between arrivals. Whenever M items accumulate, they are instantaneously removed (dispatched) from the depot. Let X(t) denote the number of items in the depot at time t. Find the limiting probability that there are pj items in the system (j = 0, . . . , M − 1). Find the average number of items in the system over an infinite time horizon. Suppose the batch size M is to be selected to minimize the average cost of running the system. The relevant costs are a cost C for dispatching the items, and a cost h per unit time for holding an item in the depot. Let C(M ) denote the average dispatching plus holding cost for running the system with batch size M . Find an expression for C(M ). Show that the$value of M that minimizes C(M ) is an integer adjacent to the value M ∗ = 2C/hμ. Exercise 44. Continuation. In the context of the preceding exercise, find the average time W that a typical item waits in the system before being dispatched. Find the average waiting time W (i) in the system for the ith arrival in the batch. Exercise 45. Consider an ergodic Markov chain Xn with limiting distribution πi . Prove lim P {Xn = j, Xn+1 = } = πj pj . n→∞

2.20 Exercises

165

One can show that (Xn , Xn+1 ) is a two-dimensional Markov chain that is ergodic with the preceding limiting distribution. However, establish the limit above only with the knowledge that Xn has a limiting distribution. Exercise 46. Items with volumes V1 , V2 , . . . are loaded on a truck one at a time until the addition of an arriving item would exceed the capacity v of the truck. Then the truck leaves to deliver the items. The number of items that can be loaded in the truck before its volume v is exceeded is N (v) = min{n :

n 

Vk > v} − 1.

k=1

Assume the Vn are independent with identical distribution F that has a mean μ and variance σ 2 . Suppose the capacity v is large compared to μ. Specify a single value that would be a good approximation for N (v). What would be a good approximation for E[N (v)]? Specify how to approximate the distribution of N (v) by a normal distribution. Assign specific numerical values for μ, σ 2 , and v, and use the normal distribution to approximate the probability P {a ≤ N (v) ≤ b} for a few values of a and b. Exercise 47. Limiting Distribution of a Cyclic Renewal Process. Consider a cyclic renewal process X(t) on the states 0, 1, . . . , K − 1 as described in Example 8. Its inter-renewal distribution is F = F0 · · · FK−1 , where Fi is distribution of a sojourn time in state i having a finite mean μi . Assume one of the Fi is non-arithmetic. Show that F is non-arithmetic. Prove lim P {X(t) = i} =

t→∞

μi . μ0 + . . . + μK−1

!t Is this limiting distribution the same as limt→∞ t−1 E[ 0 1(X(s) = i)ds], the average expected time spent in state i? State any additional assumptions needed for the existence of this limit. Exercise 48. Consider a G/G/1 system as in Example 60. Let Wn denote the length of time the nth customer waits in the queue prior to obtaining n service. Determine a Little law for the average wait W  = limn→∞ n−1 k=1 Wk . Exercise 49. System Down Time. Consider an alternating renewal process that represents the up and down states of a system. Suppose the up times have a distribution G with mean μ and variance σ 2 , and the down times have a distribution G0 with mean μ0 and variance σ02 . Let D(t) denote the length of time the system is down in the time interval [0, t]. Find the average expected d

down time β = limt→∞ t−1 E[D(t)]. Then show (D(t) − βt)/t1/2 → N (0, γ 2 ) and specify γ. Exercise 50. Congestion in a Running Race. The following model was developed by Georgia Tech undergraduate students to assess the congestion in

166

2 Renewal and Regenerative Processes

the 10-kilometer Atlanta Road Race, which is held every July 4th. After the pack of elite runners begins the race, the rest of the runners start the race a little later as follows. The runners are partitioned into m groups, with rk runners assigned to group k, 1 ≤ k ≤ m, depending on their anticipated completion times (the runners in each group being about equal in ability). The groups are released every τ minutes, with group k starting the race at time kτ (the groups are ordered so that the faster runners go earlier). Although the group sizes rk are random, assume for simplicity that they are not. Typical numbers are 10 groups of 5000 runners in each group. The aim was to design the race so that the congestion did not exceed a critical level that would force runners to walk. To do this, the students developed a model for computing the probability that the congestion would be above the critical level. (They used this model to determine reasonable group sizes and their start times under which the runners would start as soon as possible, with a low probability of runners being forced to walk.) The students assumed the velocity of each runner is the same throughout the race, the velocities of all the runners are independent, and the velocity of each runner in group k has the same distribution Fk . The distributions Fk were based on empirical distributions from samples obtained in prior races. Using pictures of past races, it was determined that if the number of runners in an interval of length  in the road was greater than b, then the runners in that interval would be forced to walk. This was based on observing pictures of congestion in past races where the runners had to slow down to a walk. Under these assumptions, the number of runners in group k that are in an interval [a, a + ] on the road at time t is Zak (t) =

rk 

1(Vkn (t − kτ ) ∈ [a, a + ]),

n=1

where Vk1 , . . . , Vkrk are the independent velocities of the runners in group k that have the distribution Fk . Then the total number of runners that are in [a, a + ] at time t is m  Za (t) = Zak (t). k=1

Specify how one would use the central limit theorem to compute the probability P {Za (t) > b} that the runners in [a, a + ] at time t would be forced to walk. Exercise 51. Confidence Interval. In the context of Example 66, suppose the regenerative-increment process Z(t) is not observed continuously over time, but only observed at its regeneration times Tn . In this case, the information observed up to the nth regeneration time Tn is {Z(t) : t ≤ Tn }. First, find the a.s. limit of Z(Tn )/n, and the limiting distribution of (Z(Tn ) − aTn )/n1/2 .

2.20 Exercises

167

Then find an approximate confidence interval for the mean a analogous to that in Example 66. Exercise 52. Continuation. Use the CLT in Examples 67 and 68 to obtain approximate confidence intervals for a renewal process and a Markov chain comparable to the confidence interval in Example 66. Exercise 53. Consider a delayed renewal process N (t) with initial distribu!x tion Fe (x) = μ1 0 [1 − F (s)]ds. Prove E[N (t)] = U Fe (t) = t/μ by a direct ∞ evaluation of the integral representing the convolution, where U = n=0 F n . Exercise 54. Justify expression (2.49), which in expanded form is 2

σ =

μ−1 i Var[Zν1

− aν1 ] =



πj f˜(j)2 + 2

j∈S



πj f˜(j)

j∈S

∞ 

pnjk f˜(k).

k∈S n=1

First show that E[Zν1 − aν1 )] = 0, and then and use the expansion ν1

  Var[Zν1 − aν1 ] = Ei [ f˜(Xn )]2 n=1

= Ei

ν1



ν1 

  Vn , f˜(Xn )2 + 2Ei

n=1

(2.76)

n=1

ν1 where Vn = f˜(Xn ) =n+1 f˜(X ). Apply Proposition 69 from Chapter 1 to ν1 ν1 −1 the last two expressions in (2.76)(noting that n=1 Vn = n=0 Vn ). Use the fact that ˜ Ei [Vn |Xn = j, ν1 ≥ n] = f(j)h(j), ν1 ˜ where h(j) = Ej [ n=1 f (Xn )] satisfies the equation 

h(j) =

pjk f˜(k) +

k∈S

and hence h(j) =

∞

 k∈S

n=1



pjk h(k),

k∈S

pnjk f˜(k).

Exercise 55. In the context of Theorem 76, show that if the distribution of the residual time B(t) is independent of t, then it is a stationary process. Hint: For any si , xi and t, let Γt = {B(s1 + t) ≤ x1 , . . . , B(sk + t) ≤ xk }. Show that P {Γt |B(t) = x} = P {Γ0 |B(0) = x}, and use this equality to prove P (Γt ) is independent of t.

Chapter 3

Poisson Processes

Poisson processes are used extensively in applied probability models. Their importance is due to their versatility for representing a variety of physical processes, and because a Poisson process is a natural model for a sum of many sparse point processes. The most basic Poisson process, introduced in the preceding chapter, is a renewal process on the time axis with exponential inter-renewal times. This type of process is useful for representing times at which an event occurs, such as the times at which items arrive to a network, machine components fail, emergencies occur, a stock price takes a large jump, etc. The first part of the present chapter continues the discussion of this basic Poisson process by presenting several characterizations of it, including the result that its point locations (i.e., occurrence times) on a finite time interval are equal in distribution to order-statistics from a uniform distribution on the interval. Applications of classical Poisson processes often involve auxiliary marks or random elements associated with the event occurrence times. For instance, if items arrive to a network at times that form a Poisson process, then a typical mark for an arriving item might be a vector denoting its route in the network and its service times at the nodes on the route. A convenient approach for analyzing such marks is to consider them as part of a larger “space-time” marked Poisson process on a multidimensional space. The properties of these processes are similar to those of “spatial” Poisson processes used for modeling locations of discrete items in the plane or a Euclidean space such as cell phone calls, truck delivery points, disease centers, geological formations, particles in space, fish colonies, etc. Following the discussion of classical Poisson processes on the time axis, we describe the structure of contemporary Poisson processes on general spaces, which includes space-time and spatial Poisson processes. The methodology for Poisson processes on general spaces involves the use of counting processes on general spaces and their representation by Laplace functionals. A Poisson process on a general space is characterized in terms of a mixed binomial

R. Serfozo, Basics of Applied Stochastic Processes, Probability and its Applications. c Springer-Verlag Berlin Heidelberg 2009 

169

170

3 Poisson Processes

process. This is a generalization of the uniform order-statistic characterization of a classical Poisson process. Several sections describe summations, partitions, translations and general transformations of Poisson processes. Included are applications of space-time Poisson processes for analyzing particle systems and stochastic networks. Next, we show that many properties of Poisson processes readily extend to several related processes; namely, Cox processes (i.e., Poisson processes with random intensities), compound Poisson processes, and cluster processes. The final results are laws of small numbers (like central limit theorems) for rare events or points, including the Poisson approximation for a binomial distribution. They justify that a Poisson process is a natural limit for a collection (or sum) of many sparse families of random points.

3.1 Poisson Processes on R+ As in the last chapter, we define ∞ a point process N = {N (t) : t ≥ 0} on R+ as a counting process N (t) = n=1 1(Tn ≤ t), where 0 = T0 ≤ T1 ≤ T2 ≤ . . . are random points (or times) such that Tn → ∞ a.s. as n → ∞. The point process N is simple when the points are distinct (T0 < T1 < . . . a.s.). We will also refer to the point process as the set of random variables N = {N (B) : B ∈ B+ }, where N (B) =

∞ 

1(Tn ∈ B),

B ∈ B+ ,

n=1

denotes the number of points in the set B, and B+ denotes the Borel sets in R+ (see Section 6.1 in the Appendix). Note that N (B) is finite when B is bounded since Tn → ∞ a.s. However, N (B) may be infinite when the set B is not bounded, and E[N (B)] may be infinite even though N (B) is finite. In addition, we write N (a, b] = N ((a, b]) = N (b) − N (a),

a ≤ b.

In the last chapter, a renewal process with exponential inter-renewal times was said to be a Poisson process. This characterization, as we show in the next section, is equivalent to the following one. Definition 1. A simple point process N = {N (t) : t ≥ 0} on R+ is a Poisson process with rate λ > 0 if it satisfies the following properties. • It has independent increments: N (s1 , t1 ], . . . , N (sn , tn ] are independent, for s1 < t1 · · · < sn < tn . • N (s, t] has a Poisson distribution with mean λ(t − s), for any s < t. A Poisson process N with rate λ is sometimes called a homogeneous or time-stationary Poisson process, or a classical Poisson process. Under the

3.1 Poisson Processes on R+

171

preceding definition, Theorem 4 below establishes that N is also a renewal process whose inter-renewal times are independent exponentially distributed with rate λ. A number of elementary properties of N follow from this renewal characterization. For instance, we saw in Example 5 that the time Tn of the nth renewal has the distribution P {Tn ≤ t} = P {N (t) ≥ n} = 1 −

n−1 

(λt)k e−λt /k!.

k=0

The derivative of this expression is f (t) = λn+1 tn e−λt /n!, and hence Tn has a gamma distribution with parameters n and λ. Keep in mind that all the properties of renewal processes apply to N . For instance, t−1 N (t) → λ a.s. by Corollary 11 in Chapter 2. Another important fact is that the Poisson process N is also a continuous-time Markov chain as in Chapter 4. Some applications of Poisson processes involve only elementary properties of the processes. Here are two examples. Example 2. Comparing Arrival Times. Two types of items arrive at a station for processing by independent Poisson processes with respective rates λ and λ . Of interest is the probability that n λ-arrivals come before the first λ arrival. This probability can be expressed as P {Tn < T  } =



λ n , λ + λ

where Tn is the time of the nth λ-arrival and T  is the time of the first λ arrival. Indeed, since T  has an exponential distribution with rate λ and it is independent of Tn ,  ∞ P {Tn < T  } = P {T  > t|Tn = t}P {Tn ∈ dt} 0  ∞  λ n  −λ t = e P {Tn ∈ dt} = E[e−λ Tn ] = . λ + λ 0 The last term is the Laplace transform of the gamma random variable Tn with parameters n and λ. More general results on comparing arrival times are given in Exercises 6 and 21. Example 3. Optimal Dispatching. Consider a system in which discrete items arrive to a dispatching station according to a Poisson process N with rate λ during a fixed time interval [0, T ]. The items might represent people to be bussed, ships to be unloaded, computer data or messages to be forwarded, material to be shipped, etc. There is a cost of h dollars per unit time of holding one item in the system. Also, at any time during the period, the items may be dispatched (or processed) at a cost of c dollars, and a dispatch

172

3 Poisson Processes

is automatically done at time T . A dispatch is performed instantaneously and all the items in the system at that time are dispatched. Consider a dispatching policy defined by a vector (n, t1 , . . . , tn ), where n is the number of dispatches to make in the period, and t1 < t2 < · · · < tn = T are the times of the dispatches. The aim is to find a dispatching policy that minimizes the expected cost. We will show that the optimal solution is to have n∗ dispatches at the times t∗i = iT /n∗, where x if xx ≥ x2 ∗ n = (3.1) x otherwise, and x = T (hλ/2c)1/2 . This type of policy is a “static” policy in that it is implemented at the beginning of the time period and remains in effect during the period regardless of how the items actually arrive (e.g., there may be 0 items in a dispatch at a predetermined dispatch time ti ). A static policy might be appropriate when it is not feasible or too costly to monitor the system and do real-time dispatching. An alternative is to use a “dynamic” control policy that involves deciding when to make dispatches based on the observed queue of units. Exercise 11 asks if the policy above is optimal for non-Poisson processes. To solve the problem, we will derive expressions for the total cost and its mean, and then find optimal values of the policy parameters. Under a fixed policy (n, t1 , . . . , tn ), the total cost is Z = cn + h

n 

Wi ,

i=1

where Wi is the amount of time that items wait in the system during the time period (ti−1 , ti ]. Since N (a, b] is the number of arrivals in a time interval (a, b], it follows that  ti

Wi =

N (ti−1 , s] ds. ti−1

Using Fubini’s theorem and E[N (a, b]] = λ(b − a),  E[Wi ] =

ti

E[N (ti−1 , s]] ds ti−1  ti

=λ

(s − ti−1 ) ds = λ(ti − ti−1 )2 /2.

ti−1

Then the expected cost under the policy (n, t1 , . . . , tn ) is hλ  (ti − ti−1 )2 . 2 i=1 n

f (n, t1 , . . . , tn ) = E[Z] = cn +

3.2 Characterizations of Classical Poisson Processes

173

Therefore, the aim is to solve the optimization problem min min f (n, t1 , . . . , tn )], n

t1 ,...,tn

(3.2)

n subject to ti−1 < ti and i=1 (ti − ti−1 ) = T .  minx1 ...,xn ni=1 x2i , under the constraint nIt is well-known that the problem ∗ i=1 xi = T , has the solution xi = T /n. This result follows by dynamic programming (backward induction), or by the use of Lagrange multipliers. Applying this result to the problem (3.2), it follows that for fixed n, the subproblem mint1 ,...,tn f (n, t1 , . . . , tn ) has the solution t∗i − t∗i−1 = T /n, so that t∗i = iT /n. Also, note that g(n) = f (n, t∗1 , . . . , t∗n ) = nc + hλT 2 /2n. Then to solve (3.2), it remains to solve minn g(n). Viewing n as a continuous variable x, the derivative g  (x) = c − hλT 2/(2x2 ) is nondecreasing. Then g(x) is convex and it is minimized at x∗ = T (hλ/2c)1/2 . So the integer that minimizes g(n) is either x∗  or x∗ . Now g(x∗ ) ≤ g(x∗ ) if and only if x∗ x∗  ≥ (x∗ )2 . This yields (3.1).

3.2 Characterizations of Classical Poisson Processes Another way of characterizing a Poisson process is that it has independent increments and satisfies the following infinitesimal properties: the probability of a point in a small interval is directly proportional to the interval length and the probability of having more than one point in such an interval is essentially 0. These properties are the basis of differential equations for probabilities of the process whose solutions are Poisson probabilities. This section consists of the following theorem, which covers the characterization we just mentioned and the renewal characterization as well. Theorem 4. For a simple point process N = {N (t) : t ≥ 0} on R+ and λ > 0, the following statements are equivalent. (a) N is a Poisson process with rate λ. (b) N is a renewal process whose inter-renewal times are exponentially distributed with rate λ. (c) N has independent increments and, for any t and h ↓ 0, P {N (t, t + h] = 1} = λh + o(h),

P {N (t, t + h] ≥ 2} = o(h).

(3.3)

Proof. (a) ⇒ (b). Assertion (b) states that ξn = Tn − Tn−1 , n ≥ 1, are independent and have an exponential distribution with rate λ. That is, P (An ) = e−λ

n

i=1 ti

,

where An = {ξ1 > t1 , . . . , ξn > tn } for ti > 0.

n ≥ 1,

(3.4)

174

3 Poisson Processes

Assuming N is a Poisson process with rate λ, we will prove (3.4) by induction. It is true for n = 1 since P (A1 ) = P {ξ1 > t1 } = P {N (t1 ) = 0} = e−λt1 . Now assume (3.4) is true for some n − 1. Then P (An ) = P (An−1 , ξn > tn ) = P (An−1 )P {ξn > tn |An−1 }.

(3.5)

Using Poisson properties of N , P {ξn > tn |An−1 } = E [P {ξn > tn |An−1 , Tn }] = E [P {N (Tn , Tn + tn ] = 0|An−1 , Tn }] = P {N (tn ) = 0} = e−λtn . Substituting this in (3.5), along with (3.4) for n − 1, yields (3.4) for n. (b) ⇒ (c). Suppose (b) is true. By Corollary 78 in Chapter 2, we know d

that N is stationary. In particular, N (t, t + h] = N (h), for h, t ≥ 0. Now, because the inter-renewal times are exponentially distributed with rate λ, P {N (h) = 0} = P {T1 > h} = e−λh , P {N (h) = 1} = P {T1 ≤ h, T2 > h} =



h

e−λ(h−s) λe−λs ds = λhe−λh .

0

Then using e−λh = 1 − λh + o(h) in these expressions yields (3.3). The proof of (c) will be complete upon showing that N has independent increments: for any s1 < t1 < · · · < sn < tn , N (s1 , t1 ], . . . , N (sn , tn ],

n ≥ 1, are independent.

Proceeding by induction, the statement is obviously true for n = 1. Next, assume it is true for some n. Now, the statement for n+1 will follow by showing that Zn = N (sn+1 , tn+1 ] is independent of Yn = (N (s1 , t1 ], . . . , N (sn , tn ]). Our proof will use the fact from Example 49 in Chapter 2 that the forward recurrence time B(t) = TN (t)+1 − t at time t is exponentially distributed with rate λ, and it is independent of the renewals in [0, t]. Using this property at time tn , it follows that Zn is conditionally independent of Yn given B(tn ), and so P {Yn = k|Zn } = E[P {Yn = k|B(tn ), Zn }|Zn ] = E[P {Yn = k|B(tn )}]. The last term is P {Yn = k}, which does not depend on Zn . Thus, Yn is independent of Zn .

3.2 Characterizations of Classical Poisson Processes

175

(c) ⇒ (a). Assuming (c) is true, (a) will follow by proving N (s, t] has a Poisson distribution with mean λ(t − s), for each s < t. We begin with the case s = 0 and prove pn (t) = P {N (t) = n} = (λt)n e−λt /n!,

n ≥ 0.

We will establish this by deriving differential equations for the functions pn (t) and showing that their solutions are the preceding Poisson probabilities. Under the assumptions in (c), for n ≥ 1, pn (t + h) = P {N (t) = n, N (t, t + h] = 0} +P {N (t) = n − 1, N (t, t + h] = 1} +P {N (t + h) = n, N (t, t + h] ≥ 2}. Since the last probability is ≤ P {N (t, t + h] ≥ 2} = o(h), and N has independent increments, the preceding is pn (t + h) = pn (t)p0 (h) + pn−1 (t)p1 (h) + o(h). In light of this expression, pn (t) is right-continuous on R+ since p0 (h) → 1 and p1 (h) → 0 as h ↓ 0. From (3.3), we have p1 (h) = λh + o(h) and p0 (h) = 1 − p1 (h) − o(h). Substituting these in the preceding expression yields (pn (t + h) − pn (t))/h = −λpn (t) + λpn−1 (t) + o(h)/h. Similar reasoning shows that pn (t) is left-continuous on (0, ∞) and (pn (t) − pn (t − h))/h = −λpn (t − h) + λpn−1 (t − h) + o(h)/h. Then letting h ↓ 0, yields the differential equations pn (t) = −λpn (t) + λpn−1 (t),

n ≥ 1.

In case n = 0, a similar argument yields p0 (t) = −λp0 (t). To solve this family of differential-difference equations, with boundary conditions pn (0) = 1(n = 0), first note that p0 (t) = e−λt satisfies the last differential equation. Then using this function and induction on n ≥ 1 it follows that pn (t) = (λt)n e−λt /n!. This proves that N (t) has a Poisson distribution with mean λt. Furthermore, by the same argument, one can show that N (s, t] has a Poisson distribution with mean λ(t − s) for any s < t (in this case, one uses pn (t) = P {N (s, t] = n}). This completes the proof that (c) implies (a).

176

3 Poisson Processes

3.3 Location of Points Many applications of Poisson processes involve knowledge about the locations of their points. Statement (c) in Theorem 4 above suggests that the points of a Poisson process are located independently in a uniform sense on R+ . This section gives a precise description of this property based on a multinomial characterization of Poisson processes. We begin with an important property of Poisson processes. Remark 5. If N is a Poisson process with rate λ, then the probability that it has a point at any fixed location t is 0 (i.e., P {N ({t}) = 1} = 0). This follows from Proposition 18 in Chapter 2 or from Proposition 30 below. d

This remark implies that N (a, b] = N (I), where I equals (a, b), [a, b], or [a, b), for a < b. Thus, Definition 1 is equivalent to the following one. Definition 6. A simple point process N is a Poisson process with rate λ if and only if N (I1 ), . . . , N (In ) are independent, for disjoint finite intervals I1 , . . . , In , and N (I) has a Poisson distribution with mean λ|I| for any finite interval I. Here |I| denotes the length of I. The next result is a characterization of a Poisson process involving a multinomial distribution (3.6) of the numbers of its points in disjoint intervals. Interestingly, (3.6) is independent of the rate λ. The analogous property for Poisson processes on general spaces is given in Theorem 28 and Example 27. Theorem 7. For a simple point process N = {N (t) : t ≥ 0} on R+ and λ > 0, the following statements are equivalent. (a) N is a Poisson process with rate λ. (b) (Multinomial Property) For any t > 0, the quantity N (t) has a Poisson distribution with mean λt, and, for any disjoint intervals I1 , . . . , Ik in [0, t], and nonnegative integers n1 , . . . , nk , P {N (I1 ) = n1 , . . . , N (Ik ) = nk |N (t) = n} =

n! pn1 · · · pnk k , n1 ! · · · nk ! 1

(3.6)

where n = n1 + · · · + nk and pi = |Ii |/t. Proof. (a) ⇒ (b). Suppose (a) holds. By Definition 6, N has independent Poisson increments over any disjoint intervals. Then letting I0 = [0, t]\∪ki=1 Ii and n0 = 0, the conditional probability in (3.6) is P {N (Ii ) = ni , 0 ≤ i ≤ k} = P {N (t) = n}

k

ni −λ|Ii | /ni ! i=0 (λ|Ii |) e . −λt n e (λt) /n!

k This clearly reduces to the right-hand side of (3.6) since i=0 |Ii | = t. (b) ⇒ (a). Suppose (b) holds. Fix a t and choose any 0 = t0 < t1 < · · · < tk = t and nonnegative integers n1 , . . . , nk such that n = n1 + · · ·+ nk . Define Ii = (ti−1 , ti ] and Ai = {N (Ii ) = ni }. Then under the properties in (b),

3.3 Location of Points

177

P (∩ki=1 Ai ) = P {N (t) = n}P {∩ki=1 Ai |N (t) = n} =

k  [λ(ti − ti−1 )]ni i=1

ni !

e−λ(ti −ti−1 ) =

k 

P (Ai ).

i=1

This proves that N (I1 ), . . . , N (Ik ) are independent, and N (Ii ) has a Poisson distribution with mean λ(ti − ti−1 ). The proof of (a) will be complete upon showing that the increments N (s1 , t1 ], . . . , N (sk , tk ] are independent, for any s1 < t1 < · · · < sk < tk = t. However, this independence follows because these increments are a subset of the increments N (0, s1 ], N (s1 , t1 ], N (t1 , s2 ], . . . , N (sk , tk ] over all the adjacent intervals, which are independent as we just proved. A special case of (3.6) is the binomial property: For I ⊆ [0, t] and k ≤ n,

n P {N (I) = k|N (t) = n} = (|I|/t)k (1 − |I|/t)n−k . k The multinomial property also yields the joint conditional distribution of point locations in [0, t] given N (t) = n, as shown in (3.7) below. Theorem 8. (Order Statistic Property) Suppose N is a Poisson process with rate λ. Then, for any disjoint intervals I1 , . . . , In in [0, t], n  & n!  %  P T1 ∈ I1 , . . . , Tn ∈ In N (t) = n = n |Ii |. t i=1

(3.7)

Hence, the joint conditional density of T1 , . . . , Tn given N (t) = n is   n!   fT1 ,...,Tn t1 , t2 , . . . , tn N (t) = n = n , t

0 < t1 < · · · < tn < t.

(3.8)

The density (3.8) is the density of the order statistics of n independent uniformly distributed random variables on [0, t] (see Proposition 10 below). Proof. Expression (3.7) follows since the conditional probability in it equals P {N (Ii ) = 1, 1 ≤ i ≤ n|N (t) = n}, which in turn equals the right-hand side of (3.7) by the multinomial property (3.6). Next, note that (3.7), for a1 < b1 < · · · < an < bn < t, is  % &   P Ti ∈ (ai , bi ], 1 ≤ i ≤ nN (t) = n =

b1

a1

 ···

bn

an

n! dt1 · · · dtn . tn

Then the integrand n!/tn is the conditional density as asserted in (3.8). Example 9. Marginal Distributions. From (3.7), (3.8) and Exercise 28,

178

3 Poisson Processes

P {Tk ≤ s|N (t) = n} =

n

 n (s/t)j (1 − s/t)n−j , j j=k

fTk (s|N (t) = n) =

n! (s/t)k−1 (1/t)(1 − s/t)n−k , (k − 1)!(n − k)!

0 ≤ s ≤ t.

This conditional distribution of Tk is the same as that of tY , where Y has a beta distribution as shown in the Appendix with parameters a = k and b = n − k + 1, and mean a/(a + b). Taking advantage of this fact, it follows easily that E[Tk |N (t) = n] = E[tY ] = kt/(n + 1). In particular, for a single point, P {T1 ≤ s|N (t) = 1} = s/t,

0 ≤ s ≤ t.

This is a uniform distribution on [0, t]. We referred to (3.8) as the density of n order statistics of independent uniformly distributed random variables on [0, t]. This is justified by the following formula for the density of order statistics of a random sample with a general density. Proposition 10. (Order Statistics) Suppose X1 , . . . , Xn are independent continuous random variables with density f , and let X(1) < · · · < X(n) denote the quantities X1 , . . . , Xn in increasing order. These order statistics X(1) , . . . , X(n) have the joint density f (x1 , . . . , xn ) = n!f (x1 ) · · · f (xn ),

x1 < · · · < xn .

(3.9)

Proof. Choose any a1 < b1 < · · · < an < bn , and let Ii = (ai , bi ], 1 ≤ i ≤ n. Since X(1) , . . . , X(n) is equally likely to be any one of the n! permutations of X 1 , . . . , Xn , P {X(i) ∈ Ii , 1 ≤ i ≤ n} = n!P {Xi ∈ Ii , 1 ≤ i ≤ n} n  = n! P {Xi ∈ Ii } i=1  b1

= n! a1

This proves the density formula (3.9).

 ···

bn

an

f (x1 ) · · · f (xn ) dx1 · · · dxn .

3.4 Functions of Point Locations

179

3.4 Functions of Point Locations Typical quantities of interest for a Poisson process N in a time interval [0, t] are deterministic or random functions of the point locations T1 , . . . , TN (t) . A N (t) classic example is n=1 f (Tn ), where f : R+ → R. This section shows how to analyze such functions in terms of random samples. The following result is an immediate consequence of Theorem 8. Corollary 11. (Order Statistic Tool) Let N be a Poisson process with rate λ, and, for each n ≥ 1, let hn be a function from Rn+ to some Euclidean or more general space S, and let h0 ∈ S. Then, for t > 0, d

hN (t) (T1 , . . . , TN (t) ) = hκ (X(1) , . . . , X(κ) ), where X(1) < · · · < X(n) are the n order statistics associated with independent random variables X1 , . . . , Xn that are uniformly distributed on [0, t] for each n, and κ is a Poisson random variable with mean λt, independent of the Xi . Furthermore, if each hn (x1 , . . . , xn ) is symmetric (meaning it is the same for any permutation of x1 , . . . , xn ), then d

hN (t) (T1 , . . . , TN (t) ) = hκ (X1 , . . . , Xκ ).

(3.10)

These expressions enable one to analyze a function of the random-length, “dependent” variables T1 , . . . , TN (t) by the simpler mixed random sample X1 , . . . , Xκ . The ideas here are related to the characterization of a Poisson process by mixed binomial or sample processes in Theorem 28 below. As an example, for f : R+ → R, N (t)



d

f (Tn ) =

n=1

κ 

f (Xn ).

n=1

n In this case, hn (x1 , . . . , xn ) = i=1 f (xi ) is symmetric. Here is another example involving random functions. Proposition 12. (Random Sums) Suppose N is a Poisson process with rate λ, and define N (t)  Z(t) = f (Tn , Yn ), (3.11) n=1

where Y1 , Y2 . . . are i.i.d. random elements in a space S, independent of N , and f : R+ × S → R. Assume φ(α, t) = E[eαf (t,Y1 ) ] exists for α in a neighborhood of 0 and t ∈ R. Then the moment generating function of Z(t) is E[eαZ(t) ] = e−λt(1−gt (α)) , where gt (α) = t−1

!t 0

φ(α, s) ds. Hence,

(3.12)

180

3 Poisson Processes



t

E[Z(t)] = λ

E[f (s, Y1 )] ds.

0

(3.13)

Proof. First note that

 E[eαZ(t) ] = E E[eαZ(t) |N (s), s ≤ t] =E

(t)

N

(t) 

N  E[eαf (Tn ,Yn ) |N (s), s ≤ t] = E φ(α, Tn ) .

n=1

n=1

n

Applying (3.10) with hn (x1 , . . . , xn ) = E[eαZ(t) ] = E

φ(α, xi ) to the last expression,

i=1

κ



 φ(α, Xn ) .

n=1

Then conditioning on κ, which is independent of the Xn ’s, we have E[eαZ(t) ] = E

κ



 E[φ(α, Xn )] .

n=1

Clearly E[φ(α, Xn )] = gt (α), since Xn is uniformly distributed on [0, t]. Using this along with the independence of the Xn ’s and the well-known Poisson generating function E[β κ ] = e−λt(1−β) , we obtain E[eαZ(t) ] = E[gt (α)κ ] = e−λt(1−gt (α)) . This proves (3.12). In addition, (3.13) follows, since the derivative of (3.12) with respect to α at α = 0 is E[Z(t)] = tλgt (0)e−λt(1−gt (0)) , where gt (0) = 1 !t and gt (0) = φ (0) = t−1 0 E[f (x, Y1 )] dx. Example 13. Discounted Cash Flows. A special case of the sum (3.11) is N (t)

Z(t) =



Yn e−γTn ,

n=1

where γ > 0. This is a standard model for discounted costs or revenues, where γ is a deterministic discount rate. For instance, suppose that sales of a product occur at times that form a Poisson process N with rate λ, and the amount of revenue from the nth sale is a random variable Yn , independent of N . Then the total discounted revenue up to time t is given by Z(t). By (3.12) and (3.13), the moment generating function and mean of Z(t) are E[eαZ(t) ] = exp

%

− λt(1 − t−1



E[Z(t)] = λE[Y1 ](1 − e−γt )/γ.

t

−γx

E[eαe 0

Y1

& ] dx)

3.5 Poisson Processes on General Spaces

181

Example 14. Compound Poisson Process. Another example of (3.11) is Z(t) = N (t) n=1 Yn . This is like the preceding discounted cash flow, but without discounting. Letting γ → 0 in the preceding example, it follows that E[eαZ(t) ] = e−λt(1−E[e

αY1 ])

,

E[Z(t)] = λtE[Y1 ]. This moment generating function of Z(t) is that of a compound Poisson distribution with rate λt and distribution F (y) = P {Y1 ≤ y}, which is P {Z(t) ≤ z} =

∞ 

e−λt (λt)n F n (z)/n!,

z ∈ R.

n=0

The process {Z(t) : t ≥ 0} is called a compound Poisson process; further properties of it are in Section 3.15.

3.5 Poisson Processes on General Spaces Applications of classical Poisson processes on R+ typically require knowledge of Poisson processes on Euclidean and general spaces. In addition, there are many spatial systems of points that can be modeled by Poisson processes. Accordingly, the rest of this chapter will focus on basics of point processes and Poisson processes on general spaces. This section introduces the terminology we will use. A “point process” is a counting process that represents a random set of points in a space. The usual spaces are the real line, the plane, the multidimensional Euclidean space Rd , or, more generally, a complete, separable metric space (a Polish space). Following the standard convention, we will discuss point processes on a polish space S. The exposition will be understandable by thinking of S as an Euclidean space. We let S denote the family of Borel sets of S (see the Appendix), and let Sˆ denote the family of bounded Borel sets (a set is bounded if it is contained in a compact set). We refer to ˜ etc. S simply as a space, and denote other spaces of this type by S  , S, We begin with an informal description of a point process. A random set of points in S is a countable set of S-valued random elements {Xn } such that only a finite number of the points are in any bounded set. Then  δXn (B), B ∈ S, N (B) = n

denotes the number of points in B, where δx (B) = 1(x ∈ B) (a Dirac measure with unit mass at x). This counting measure N , as we will define below, is a point process on S with point locations {Xn }.

182

3 Poisson Processes

Note that N takes values in the set M of all counting measures ν on (S, S) that are locally finite (ν(B) < ∞, for bounded sets B). Each measure in M has the form  δxn (B), B ∈ S, (3.14) ν(B) = n

where {x1 , . . . , xk } is its associated set of points, for 0 ≤ k ≤ ∞. There may be more than one point at a location, and the order of the subscripts on the locations is invariant under permutations. The set M is endowed with the σ-field M generated by the sets {ν ∈ M : ν(B) = n}, for B ∈ S and n ≥ 0. We are now ready for a formal definition. Definition 15. A point process on a space S is a measurable map N from a probability space (Ω, F , P ) to the space (M, M). The quantity N (B) is the number of points in the set B ∈ S. By the formulation (3.14),  δXn (B), B ∈ S, (3.15) N (B) = n

where the Xn denote the locations of the points of N . For the following discussion, assume that N is a point process on the space S. Technical properties of the spaces S and M are not used explicitly in the sequel. One can simply think of N as a counting process on S = Rd that is locally finite (N (B) < ∞ a.s. for bounded sets B). The probability distribution of the point process N (i.e., P {N ∈ ·}) is determined by its finite-dimensional distributions P {N (B1 ) = n1 , . . . , N (Bk ) = nk },

ˆ B1 , . . . , Bk ∈ S.

(3.16)

In other words, two point processes N and N  on S are equal in distribution, d denoted by N = N  , if their finite-dimensional distributions are equal: (N (B1 ) . . . , N (Bk )) = (N  (B1 ) . . . , N  (Bk )), d

ˆ B1 , . . . , Bk ∈ S.

In constructing a point process, it suffices to define the probabilities (3.16) on sets Bi that generate S. When S = Rd , “rectangles” of the form1 (a, b] generate S. The intensity measure (or mean measure) of the point process N is μ(B) = E[N (B)],

B ∈ S.

Note that μ(B) may be infinite, even if B! is bounded. When S = Rd , the intensity is sometimes of the form μ(B) = B λ(x)dx, where λ(x) is the rate of N at the location x and dx denotes the Lebesgue measure. We call λ(x) the location-dependent rate function of N . 1

Here x = (x1 , . . . , xd ) is a typical vector in Rd and (a, b] = {x ∈ Rd : ai < xi ≤ bi , 1 ≤ i ≤ d} is a half-open interval in Rd .

3.6 Integrals and Laplace Functionals of Poisson Processes

183

Our main focus hereafter will be on Poisson point processes. Definition 16. A point process N on a space S is a Poisson process with intensity measure μ that is locally finite if the following conditions are satisfied. • N has independent increments: The quantities N (B1 ), . . . , N (Bn ) are inˆ dependent for disjoint sets B1 , . . . , Bn in S. ˆ • For each B ∈ S, the quantity N (B) is a Poisson random variable with mean μ(B). This definition uses the fact that N (B) = 0 a.s. when μ(B) = 0. Note that the number of points N ({x}) exactly at x has a Poisson distribution with mean μ({x}); so N ({x}) = 0 a.s. when μ({x}) = 0. From the definition, it follows that the finite-dimensional distributions of a Poisson process are uniquely determined by its intensity measure, and vice versa. That is, two Poisson processes N and N  on S are equal in distribution if and only if their intensity measures are equal. Do Poisson processes exist? In other words, does there exist a point process on a probability space that satisfies the properties in Definition 16? We will establish the existence later when we show in Theorem 29 below that a Poisson process can be characterized by independent random elements, which do exist. The family of Poisson processes on R+ contain the classical ones. Example 17. Poisson Processes on R+ . A Poisson process N on R+ (or on R) with intensity measure μ is sometimes called a non-homogeneous Poisson process. We denote its point locations (as we have been doing) by 0 < T1 ≤ T2 ≤ . . . instead of Xn , and call them “times” when appropriate.  Here there may be more than one point at a location. In this setting, N (B) = n δTn (B) has a Poisson distribution with mean μ(B). We also write N (t) = N (0, t], for t > 0, ! and N (a, b] = N ((a, b]) and μ(a, b] = μ((a, b]), for a < b. When μ(B) = B λ(t)dt, we say N is a Poisson process with rate function λ(t). In case μ(t) = E[N (t)] = λt, for some λ > 0, then N is a classical Poisson process with rate λ, consistent with Definition 1. It is sometimes called a homogeneous or stationary Poisson process with rate λ. Results in the preceding sections for homogeneous Poisson processes have obvious analogues for nonhomogeneous processes. For instance, N satisfies the multinomial property (3.6) with pi = μ(Bi )/μ(0, t], and Exercise 29 describes its order statistic property.

3.6 Integrals and Laplace Functionals of Poisson Processes Laplace transforms are useful for identifying distributions of nonnegative random variables and for establishing convergence in distribution of random

184

3 Poisson Processes

variables. The analogous tool for point processes is a Laplace functional. This section covers a few properties of Laplace functionals and related integrals of point processes. These are preliminaries needed to establish the existence of Poisson processes, the topic of the next section. The use of Laplace transforms and functionals for establishing the convergence of random variables and point processes are covered later in Sections 3.16 and 3.17. We begin with a little review. Recall that the Laplace transform of a nonnegative random variable X with distribution F is  e−αx dF (x), α ≥ 0. FˆX (α) = E[e−αX ] = R+

d

This function uniquely determines the distribution of X in that X = Y if and only if FˆX (·) = FˆY (·). For instance, the Laplace transform of a Poisson random variable X with mean λ is FˆX (α) =

∞ 

−α

e−αn e−λ λn /n! = e−λ(1−e

)

.

n=0

Now, if Y is a nonnegative integer-valued random variable with E[e−αY ] = −α e−λ(1−e ) , then Y has a Poisson distribution with mean λ. Here is an example for sums. Example 18. Sums of Independent Poisson Random Variables. Let Y1 , . . . , Yn be independent Poisson random variables with respective means μ1 , . . . , μn . n n Then i=1 Yi has a Poisson distribution with mean μ = i=1 μi (which we assume is finite when n = ∞). To see this result, note that by the independence of the Yi and the form of their Laplace transforms, E[e−α

n

i=1

Yi

]=

n 

−α

E[e−αYi ] = e−μ(1−e

)

.

i=1

We recognize this as  being the Laplace transform of a Poisson distribution n with mean μ, and so i=1 Yi has this distribution. The rest of this section covers analogous properties  for Laplace functionals of point processes. Consider a point process N = n δXn on a space S. The “integral” of a function f : S → R+ with respect to N is simply the sum   Nf = f (x)N (dx) = f (Xn ), S

n

provided it is finite. It is finite when f has a compact support, meaning that {x : f (x) > 0} is contained in a compact set. Similarly, the integral of f : S → R+ with respect to a measure μ will be denoted by

3.6 Integrals and Laplace Functionals of Poisson Processes

185

 μf =

f (x)μ(dx). S

+ (S) of all continuous We will often use integrals of functions f in the set CK functions f : S → R+ with compact support.

Definition 19. The Laplace functional of the point process N is 

 −N f ] = E exp{− f (x)N (dx)} , f : S → R+ . E[e S

The function f is a “variable” of this expectation (just as the parameter α is a variable in a Laplace transform E[e−αX ]). The following result contains the basic property that the Laplace functional of a point process uniquely determines its distribution (the proof is in [61]). It also justifies that a Laplace functional is uniquely defined on the set + CK (S) viewed as “test” functions. Theorem 20. For point processes N and N  on S, each one of the following d statements is equivalent to N = N  . d + (a) N f = N  f, f ∈ CK (S).  + −N f −N f (b) E[e ] = E[e ], f ∈ CK (S). Laplace functionals are often more convenient to use than finite-dimensional distributions in deriving the distribution of a point process constructed as a function of random variables or point processes. A standard approach for establishing that a point process is Poisson is to verify that its Laplace functional has the following form; this also yields its intensity measure. Theorem 21. (Poisson Laplace Functional) For a Poisson process N on S with intensity measure μ, and f : S → R+ ,  −N f E[e ] = exp[− (1 − e−f (x) )μ(dx)]. S

k Proof. First consider the simple function f (x) = i=1 ai 1(x ∈ Bi ), for some ˆ Then nonnegative a1 , . . . , ak and disjoint B1 , . . . , Bk in S. Nf =

k 

 1(x ∈ Bi )N (dx) =

ai S

i=1

k 

ai N (Bi ).

i=1

Using this and the independence of the N (Bi ), we have E[e−N f ] =

k 

E[e−ai N (Bi ) ] = exp[−

i=1



= exp[− S

k  i=1

(1 − e−f (x) )μ(dx)].

μ(Bi )(1 − e−ai )]

186

3 Poisson Processes

Next, for any continuous f : S → R+ , there exist simple functions fn ↑ f (e.g., fn (x) = n ∧ (2n f (x)/2n )). Then by the monotone convergence theorem (see the Appendix) and the first part of this proof,  E[e−N f ] = lim E[e−N fn ] = lim exp[− (1 − e−fn (x) )μ(dx)] n→∞ n→∞ S  −f (x) = exp[− (1 − e )μ(dx)]. S

This completes the proof. Recall that Example 18 uses Laplace transforms to prove that a sum of independent Poisson random variables is Poisson. Here is an analogous result for a sum of Poisson processes. Theorem 22. (Sums of Independent Poisson Processes) Let N1 , . . . , Nn denote independent Poisson processes on S with respective n intensity measures N = μ1 , . . . , μn . Then their sum (or superposition) i=1 Ni is a Poisson n process with intensity measure μ = i=1 μi . This is also true for n = ∞ provided μ is locally finite. Proof. One can prove this, as suggested in Exercise 14, by verifying that N satisfies the defining properties of a Poisson process. Another approach, using Laplace functionals and Theorem 21, is to verify that E[e−N f ] = e−μh ,

+ f ∈ CK (S),

where h(x) = 1 − e−f (x) . But this follows since by the independence of the Ni and the form of their Laplace functionals in Theorem 21, E[e−N f ] =

n  i=1

E[e−Ni f ] =

n 

e−μi h = e−μh .

i=1

Example 23. A company that produces a household cleaning fluid has a bottle-filling production line that occasionally has to stop for repair due to imperfections in the bottles or due to worker errors. There are four types of line stoppages: (1) minor stop (under 30 minutes) due to bottle imperfection, (2) major stop (over 30 minutes) due to bottle imperfection, (3) minor stop due to worker error, and (4) major stop due to worker error. These four types of stoppages occur according to independent Poisson processes with respective rates λ1 , . . . , λ4 . Then by Theorem 22, line stops due to any of these causes occur according to a Poisson process with rate λ1 + · · · + λ4 . Similarly, minor stops occur according to a Poisson process with rate λ1 +λ3 , and major stops occur according to a Poisson process with rate λ2 + λ4 . A model like this was used in a study of a bottling plant by Georgia Tech students.

3.6 Integrals and Laplace Functionals of Poisson Processes

187

 We end this section with more insight on integrals N f = n f (Xn ) with respect to a point process N . Expression (3.17) below says the mean of such an integral equals the corresponding integral with respect to the intensity measure. Theorem 22 in Chapter 2 for renewal processes is a special case. The variance of N f has the nice form (3.18), when N is Poisson.  Theorem 24. Let N = n δXn be a point process on S with intensity measure μ. For any f : S → R,

   E f (Xn ) = f (x)μ(dx), (3.17) S

n

provided the integral exists. That is, E[N f ] = μf . If in addition, N is a Poisson process, then  

 f (Xn ) = f (x)2 μ(dx), (3.18) Var n

S

provided the integral exists. That is, Var[N f ] = μf 2 . Proof. The proof of (3.17) is similar to that of Theorem 22 or Theorem 21. Namely, first one shows E[N f ] = μf is true when f is a simple function, and then monotone convergence yields the equality for general f , which is a monotone limit of simple functions. To prove (3.18), note that by Theorem 21, we have E[e−αN f ] = e−h(α) , where h(α) = α = 0, yields

! S

(3.19)

(1 − e−αf (x) )μ(dx). The derivative of this expression at E[N f ] = h (0)e−h(0) = μf.

Furthermore, taking the second derivative of (3.19) at α = 0, and using h(0) = 1 and h (0) = E[N f ], we obtain

 E[(N f )2 ] = lim (h (α))2 e−h(α) − h (α)e−h(α) α↓0

  = lim f (x)2 e−αf (x) μ(dx) + (h (α))2 α↓0 S  = f (x)2 μ(dx) + (E[N f ])2 . S

This proves (3.18). Example! 25. Suppose N is a Poisson process on R+ with intensity measure μ(B) = B e−at dt, and consider

188

3 Poisson Processes



u

Z=

N (u)



ce−bt N (dt) =

0

ce−bTn .

n=1

Here a, b, c, u are nonnegative constants. By Theorem 24,  u c[1 − e−u(a+b) ] , E[Z] = c e−bt e−at dt = a+b 0  u c2 [1 − e−u(a+2b) ] . Var[Z] = c2 e−2bt e−at dt = a + 2b 0

3.7 Poisson Processes as Sample Processes Section 3.4 showed that a Poisson process on R+ can be characterized by an order-statistic property of its point locations. This section presents an analogous characterization of a Poisson process on a general space in terms of sample processes. Using this result, we establish the existence of Poisson processes. Sample processes are also useful as models by themselves as well as building blocks for identifying Poisson processes. A sample process is a fundamental point process defined as follows. Suppose that X1 , X2 , . . . are i.i.d. random elements in the space S with distribution F (B) = P {X1 ∈ B}. The point process N=

n 

δXi

i=1

on S is a sample process for n samples from F . Clearly, the number of samples N (B) in a set B has a binomial distribution with parameters n and μ(B) (N is also called a binomial point process). In what follows, we consider such a sample process in which the sample size is a random variable that is independent of the samples; the resulting process is called a mixed sample process. We first consider Poisson processes with finite intensities. Theorem 26. Suppose N is a point process on S with an intensity measure μ such that 0 < μ(S) < ∞. The N is a Poisson process if and only if d ˜ N =N =

κ 

δX˜ i ,

i=1

˜ is a mixed sample process for κ samples from the distribution where N μ(·)/μ(S), and κ has a Poisson distribution with mean μ(S).

3.7 Poisson Processes as Sample Processes

189

Proof. To prove the assertion, it suffices to verify that the Laplace functional ˜ is the same as that in Theorem 21 for a Poisson process with intensity of N measure μ. That is,  ˜ + f ∈ CK (S). E[e−Nf ] = exp[− (1 − e−f (x) )μ(dx)], S

But this follows, since using the generating function E[z κ ] = e−μ(S)(1−z) and ˜ 1 has the distribution μ(·)/μ(S), the fact that X



 κ  κ ˜ ˜ ˜ E[e−N f ] = E E[e− i=1 f (Xi ) |κ] = E E[e−f (X1 ) ] ˜

= exp[−μ(S)(1 − E[e−f (X1 ) ])]  = exp[− (1 − e−f (x) )μ(dx)]. S

Example 27. Fires occur in a region S of a city at locations X1 , X2 , . . . that are independent with distribution F . Then n the spatial locations of n fires in S is given by the sample process N = i=1 δXi . In particular, the number of fires in a region B ∈ S has a binomial distribution with parameters n and F (B). Furthermore, the numbers of fires in B1 , . . . , Bk in S that form a partition of S, have the multinomial distribution, for n = n1 + · · · + nk , P {N (B1 ) = n1 , . . . , N (Bk ) = nk } =

n! F (B1 )n1 · · · F (Bk )nk . n1 ! · · · nk !

Next, suppose the number of fires in a year is a Poisson random variable κ with mean λ that is independent of the fire locations. Then the fires in S ˜ = κ δXi , which is Poisson process with intensity are represented by N i=1 ˜ measure λF (·) by Theorem 26. In particular, the number of fires N(B) in B has a Poisson distribution with mean λF (B). Theorem 26 has the following extension that says any Poisson process on a bounded set is equal to a sample process on that set. Theorem 28. A point process N on S with a locally finite intensity measure μ is a Poisson process if and only if, for each B ∈ Sˆ with μ(B) > 0, d

˜ (·), N (· ∩ B) = N ˜ is a mixed sample process of κ samples from the probability measure where N μ(· ∩ B)/μ(B), and κ has a Poisson distribution with mean μ(B). Proof. This follows by applying Theorem 26 to each bounded set B, and using the fact that the distribution of a Poisson process is determined (via Laplace functionals) by its distribution on bounded sets (the supports of the functions in the Laplace functional).

190

3 Poisson Processes

The preceding characterization of a Poisson process yields the following result. Theorem 29. (Existence of Poisson Processes) For any locally finite measure μ on S, there exists a Poisson process N on S with intensity measure μ. Proof. First note that a sample process for a random-sized sample exists since it is a function of an infinite collection of independent random variables, which exist by Corollary 6 in the Appendix (i.e., one can construct a probability space and the independent random variables on it). Then a Poisson process N with a “finite” intensity measure μ exists since it is a sample process by Theorem 26. Next, consider the case when μ is infinite. Choose bounded sets B1 , B2 , . . . in S that partition S such that μ(Bn ) < ∞. By the preceding part of the proof, there exists a Poisson process Nn on S, for each n, with intensity μn (·) = μ(· ∩ Bn ). By Theorem 6 in the Appendix, we can define these Nn on a common probability space so that they are independent. Then define  N . By Theorem 22, N is a Poisson process on S with intensity N = n n  μ = μ. n n We end this section with a criterion for a Poisson process to be simple. A point process N on S is simple if P {N ({x}) ≤ 1, x ∈ S} = 1 (i.e., its point locations are distinct). Proposition 30. A Poisson process N with intensity measure μ is simple if and only if μ({x}) = 0, x ∈ S. Hence any Poisson !process on an Euclidean space is simple if its intensity has the form μ(B) = B λ(x)dx, for some rate function λ(x). Proof. In light of Theorem 28, it suffices to prove this when μ is finite. In this d ˜ ˜ is a mixed sample process as in Theorem 26. Now case, N = N , where N ˜ is simple) = P (N is simple) = P (N

∞ 

P (Dn )P {κ = n},

n=2

˜ 1, . . . , X ˜ n are distinct}. Then N is simple if and only if where Dn = {X P (Dn ) = 1, for each n ≥ 2. The latter statement is true, by Exercise 39, if and only if μ({x})/μ(S) = 0, x ∈ S. Hence N is simple if and only if μ({x}) = 0, x ∈ S.

3.8 Deterministic Transformations of Poisson Processes As we will see, many point processes involving complex phenomena or systems can be represented by functions of Poisson processes. In these settings,

3.8 Deterministic Transformations of Poisson Processes

191

a Poisson process is typically the basic data that defines or initializes a system, and various characteristics of the system are deterministic or random functions of the Poisson process. In particular, if part of a system’s information is a point process (e.g., production completion times, or a random function of the original Poisson points), it is natural to know whether that point process is a Poisson process. A basic issue in this regard is: If the point locations of a Poisson processes are mapped to some space by a deterministic or random mapping, then do the resulting new points also form a Poisson Process? This issue in a variety of contexts is the underlying theme for the next six sections. We begin in this section by considering deterministic maps such as the following one. Example 31. Suppose N is a Poisson process in the nonnegative quadrant S = R2+ of the plane with intensity measure μ. Let N  (r) denote the number of points of N within a distance r from the origin. We can represent N  as a mapping $ of N in which a point (x, y) of N is mapped to its distance g(x, y) = x2 + y 2 from the origin. Then, N  (r) = N ({(x, y) ∈ S : g(x, y) ≤ r}). In this example, N  simply records some characteristic (the distance to the origin) of each point in the so-called data N . By Theorem 32 below, N  is a Poisson process on R+ with mean measure E[N  (r)] = μ({(x, y) ∈ S : g(x, y) ≤ r}). This mean is clearly finite for each r. In case the Poisson process N is homogeneous with a constant rate λ, then E[N  (r)] = λπr2 . We will now show that any general mapping of a Poisson process, such as the one above, results in a new Poisson process. Suppose N is a Poisson process on S with intensity measure μ. Consider a transformation of N in which its points in S are mapped to a space S  (possibly S) by the rule that a point of N located at x ∈ S is mapped to the location g(x) ∈ S  , where g : S → S  . Then the number of points mapped into B ∈ S  is  δg(Xn ) (B), B ∈ S. N  (B) = n

This N  is a point process on S  , provided it is locally finite. In this case, we say that N  is a transformation of N under the map g. A more complete representation of this transformation is given by the point process M on the product space S × S  defined2 by 2

To define a random measure M on a product space, it suffices to define it on product sets  the case at hand, M (A × B) =  A × B (this also highlights the sets separately). For  n δ(Xn ,g(Xn )) (A×B) automatically implies M (C) = n δ(Xn ,g(Xn ))(C) , for C ∈ S ×S . Mean measures E[M (A × B)] are also defined on product sets.

192

3 Poisson Processes

M (A × B) =



A ∈ S, B ∈ S  .

δ(Xn ,g(Xn )) (A × B),

n

This is the number of points of N in A ∈ S that are mapped into B ∈ S  . So M contains the original process N (·) = M (· × S  ) as well as the transformed process N  (·) = M (S × ·). The M is an example of a “marked” point process, where g(Xn ) is a “mark” associated with Xn . General marked point processes are discussed in the next sections. A better understanding of the processes M and N  is provided by the inverse of g, which is g −1 (B) = {x ∈ S : g(x) ∈ B},

B ∈ S.

Namely, using δg(Xn ) (B) = δXn (g −1 (B)), it follows that, for A ∈ S, B ∈ S  , M (A × B) = N (A ∩ g −1 (B)),

N  (B) = N (g −1 (B)).

(3.20)

In other words, M and the transformed process N  behave like the original process N on part of its space. Because of this, M and N  inherit the Poisson property of N as follows. Theorem 32. The marked point process M defined by (3.20) is a Poisson process with E[M (A × B)] = μ(A ∩ g −1 (B)),

A ∈ S, B ∈ S  .

(3.21)

Hence the transformed process N  is a Poisson process on S  with intensity E[N  (B)] = μ(g −1 (B)), B ∈ S  , provided this measure is locally finite. Proof. This result is a special case of Theorem 36 below, which is proved by showing that the Laplace transform of M is the same as that for a Poisson process with intensity (3.21). Here is another proof that illustrates the sample-process characterization of a Poisson process. For the case μ(S) < ∞, we have d

N=

κ  m=1

δX˜ n ,

d

M=

κ 

δ(X˜ n ,g(X˜n )) .

m=1

The first sum is a sample process of κ samples from μ(·)/μ(S) and κ has a Poisson distribution with mean μ(S). The second sum, due to the definition of M and the form of N , is a sample process on S × S  of κ samples from the distribution ˜ 1 ∈ A, g(X ˜ 1 ) ∈ B} = μ(A ∩ g −1 (B))/μ(S). F (A × B) = P {X Therefore, M is a Poisson process with intensity measure given by (3.21). This statement is also true for the case μ(S) = ∞ by Theorem 28, since the preceding argument also applies to each bounded set in S × S  .

3.8 Deterministic Transformations of Poisson Processes

193

In addition, since M is Poisson and N  (·) = M (S × ·) is M on part of its space, it follows that N  is Poisson with intensity E[M (S × B)] = μ(g −1 (B)), when this measure is locally finite. A basic property of a Poisson process N on a product space S1 × S2 is that the projection N (S1 × ·) on S2 is also a Poisson process provided its intensity E[N (S1 × ·)] is locally finite. This follows immediately from the definition of a Poisson process. Here is an extension of this fact.  Example 33. Projections of a Poisson Process. Let N = n δXn denote a Poisson process on S ⊆S1 × · · · Sm , with intensity μ, where Xn = (X1n , . . . , Xmn ). Let Ni = n δXin denote  the projection of N on the subspace Si = {xi : x ∈ S}, and let Mi = n δ(Xn ,Xin ) be the marked point process that describes the points in the domain and range of the mapping of N by the projection map gi (x) = xi . Then Mi is Poisson with intensity μi (A × B) = μ{x ∈ A : xi ∈ B} by Theorem 32. Hence Ni (·) = Mi (S × ·) is a Poisson process with intensity E[Ni (B)] = μ{x ∈ S : xi ∈ B}, provided this intensity is locally finite. For instance, suppose N is a homogeneous Poisson process on Rm + with on the projection. rate λ. Then Ni (0, b] = ∞ a.s., but Mi still gives insights  Next, consider the more general projection NI = n δgI (Xn ) on the space SI = {gI (x) : x ∈ S},where gI (x) = (xi : i ∈ I), for I ⊆ {1, . . . , m}. The related process MI = n δ(Xn ,gI (Xn )) on S × SI is Poisson by Theorem 32. Hence NI (·) = M (S × ·) is Poisson with E[NI (B)] = μ{x ∈ S : gI (x) ∈ B}, provided this is locally finite.  Example 34. Let N = n δ(Xn ,Yn ) denote a Poisson process on the unit disc S in R2 with rate function λ(x, y). Consider the projectionof N on the interval S  = [−1, 1], which is described by the process N  = n δXn on S  . By Example 33, N  is Poisson with 



E[N (a, b]] = a

b



√

1−x2

λ(x, y) dy dx.

√ − 1−x2

More generally, projections on S  = [−1, 1] from points located  in sets like Au = {(x, y) ∈ S : y ≥ u}, u ∈ (0, 1] are described by M = n δ((Xn ,Yn ),Xn ) on S × S  , which is Poisson with 

b



1

E[M (Au × (a, b])] =

λ(x, y) dy dx. a

u

Next, consider the transformation N of N under which a point in the unit disc S is mapped to the closest point on the unit circle C. To define N , we represent a point in S by  its polar coordinates (r, θ), where (x, y) = (r cos θ, r sin θ), and view N = n δ(Rn ,Θn ) as a Poisson process on S = {(r, θ) ∈ [0, 1] × [0, 2π)} with rate function λ(r cos θ, r sin θ). The unit circle can be expressed as C = [0, 2π), since each point on it has the form (1, θ).

194

3 Poisson Processes

The transformation under consideration maps a point  at (r, θ) to (1, θ) (i.e., to θ ∈ C), and so the transformed process is N = n δΘn , which is simply the projection of N on the coordinate set C. Therefore N is Poisson with   1 λ(r cos θ, r sin θ)dr dθ. E[N (B)] = B

0

See Exercise 35 for more details on these processes.

3.9 Marked and Space-Time Poisson Processes The last section showed how a deterministic transformation of a Poisson process can be represented by a marked Poisson process. We now extend this idea to random transformations that are represented by marked Poisson processes. An important class of marked Poisson processes are space-time Poisson processes. The focus of this section is on a transformation of a Poisson process on a space S in which each of its points is independently assigned a random mark in a space S  depending only on the particular point location. The distributions of the marks will be determined by probability kernels. A mark assigned to a point at x ∈ S, will take a value in a set B ∈ S  according to a probability kernel p(x, B) from S → S  . Such a kernel is a function p : S × S  → [0, 1] such that p(· B) is a measurable function on S and p(x, ·) is a probability measure on S  . Our interest will be in modeling the initial points as well as the marks by a marked point process on S × S  . The formal definition is as follows.  Definition 35. on S with intensity  Let N = n δXn be a Poisson process μ. Let M = n δ(Xn ,Yn ) be a point process on S × S  such that P {Yn ∈ B|N } = p(Xn , B),

B ∈ S  , n ≤ N (S),

where p(x, B) is a probability kernel from S to S  . The Yn are p-marks of the Xn , and the point process M of the initial points and their marks is a p-marked Poisson process associated with N . The M is a space-time Poisson process when it is defined on R+ × S  (or R × S  ) and R+ represents the time axis. The mean measure of M is given by (3.22) below. Calling M a Poisson process in this definition is justified by the next result, which is an extension of Theorem 32 for deterministic marks.  Theorem 36. The point process M = n δ(Xn ,Yn ) in Definition 35 is a Poisson process on S × S  with intensity measure μM defined by

3.9 Marked and Space-Time Poisson Processes

195

 μM (A × B) =

p(x, B)μ(dx),

A ∈ S, B ∈ S  .

(3.22)

A

 Hence, the point process of marks N  = n δYn is a Poisson process on S  ! with intensity μ (B) = S p(x, B)μ(dx), B ∈ S  , provided this is locally finite. Proof. It suffices by Theorem 21 to show that the Laplace functional of M is the same as that for a Poisson process on S × S  with intensity measure μM . Since Yn are p-marks of the Xn ,

  E[e−Mf ] = E E[e− n f (Xn ,Yn ) |N ]

  =E e−f (Xn ,y) p(Xn , dy)

n

= E exp

S

%

 log

n

S

& e−f (Xn ,y) p(Xn , dy) .

! Letting h(x) = − log S  e−f (x,y) p(x, dy) in the last line, and using the Laplace functional of N as in Theorem 21, we have E[e−Mf ] = E[e−

!

S



h(x)N (dx)

= exp[−

]

(1 − e−h(x) )μ(dx)]

S = exp[− S×S 

(1 − e−f (x,y))μM (dx, dy)].

The last line by Theorem 21 is the Laplace functional of a Poisson process with mean measure μM and hence M is such a Poisson process. In addition N  (B) = M (S × B) is M on part !of its state space, and so N  is Poisson with intensity μ (B) = μM (S × B) = S p(x, B)μ(dx). Here are some examples. Further applications are in the next sections and in Exercises 47, 49 and 32.  Example 37. Marked Tornados. Suppose N = n δXn is a Poisson process on a region S of a country that represents locations of tornados that might occur in a year. For simplicity, assume its intensity μ(S) is finite. Additional information about the tornados is naturally recorded by marks Yn associated with the tornados at the respective locations Xn . For instance, Yn might record the cost to repair the damage, the number of deaths, or a vector of auxiliary information for tornado Xn . Assume the Yn take values in a space S  and are p-marks of the Xn . Then the tornado information is conveniently represented by the p-marked Poisson processes   δ(Xn ,Yn ) , N = δYn . M= n

n

196

3 Poisson Processes

Example 38. Maxima of Marks. Suppose that  events occur over the time axis R+ according to a Poisson process N = n δTn with intensity measure μ. The event that occurs at time Tn produces a real-valued random variable Yn . Assume the Yn are p-marks of the Tn , where p(t, ·) is the distribution of a mark at time t. Consider the cumulative maxima process t ≥ 0,

V (t) = max Yn , n≤N (t)

where V (t) = 0 when N (t) = 0. For instance, this could be the maxima of heights of solar flares on a region of the sun, where the flares occur at times that form a Poisson process. One can obtain information about this maxima process via the space-time  Poisson process M = n δ(Tn ,Yn ) on R+ × R with intensity  E[M ((0, t] × B)] = p(s, B)μ(ds). (0,t]

For instance, the event {V (t) ≤ y} equals {M ((0, t] × (y, ∞)) = 0}, and so P {V (t) ≤ y} = e−

! (0,t]

p(s,(y,∞))μ(ds)

.

Also, if μ(t) = λt and p(t, ·) = G(·), independent of t, then V (t) is a continuous-time Markov process with transition probabilities P {V (s + t) ≤ y|V (s) = x} = 1(x ≤ y)P {M ((s, s + t] × (y, ∞)) = 0} = 1(x ≤ y)e−λt(1−G(y)) . Example 39. Serial Marking of a Poisson Process. Theorem 36 for a single marking of a Poisson process extends to  a series of markings as follows. Starting with a Poisson process N = n δXn , if Yn are p-marks of Xn ,  then M = n δ(Xn ,Y is a Poisson process. Similarly, if Yn are p -marks of n) (Xn , Yn ), then M  = n (Xn , Yn , Yn ) is again a Poisson process. These marking steps can be continued several times, with the end result being a Poisson process from which one can “read off” many results. In addition to serial markings in applications, they are useful for proving results for compound Poisson processes as discussed in Section 3.15.

3.10 Partitions and Translations of Poisson Processes In this section, we continue our study of transformations of Poisson processes by considering partitioning and translations of the points of a Poisson process. We show how these two types of transformations can be modeled as marked Poisson processes.

3.10 Partitions and Translations of Poisson Processes

197

We begin with a special kind of partitioning. Example 40. Thinning of a Poisson Process. Let N be a Poisson process on S with intensity μ. Suppose the points of N are deleted according to the rule that a point at x is retained with probability p(x), and the point is deleted with probability 1 − p(x). Let N1 and N2 denote the resulting processes of retained and deleted points, respectively, where N = N1 + N2 . By Proposition 41 below, N1 and N2 are independent Poisson processes with respective mean measures   p(x)μ(dx), E[N2 (A)] = (1 − p(x))μ(dx), A ∈ S. E[N1 (A)] = A

A

Interestingly, N1 and N2 are independent even though N = N1 + N2 . As an example, suppose a web site that sells products has visitors arriving to it according to a Poisson process N with rate λ. Suppose p percent of these visitors buy a product, which means that each visitor independently buys a product with probability p. Then from the preceding result, the times of sales form a Poisson process with rate pλ, and the visits without sales occur according to a Poisson process with rate (1 − p)λ. The preceding thinning model is a special case of the following partitioning procedure for decomposing a point process into several subprocesses. Consider a Poisson process N on S with intensity μ. Suppose N is partitioned into a countable family of processes Ni , i ∈ I, on S by the following rule. Partitioning Rule: A  point of N at x is assigned to subprocess Ni with probability p(x, i), where i∈I p(x, i) = 1.  The processes Ni form a partition of N in that N = i∈I Ni . Proposition 41. (Partitioning of a Poisson Process) The subprocesses Ni , i ∈ I, of the Poisson process N are independent Poisson processes with intensities  E[Ni (B)] = p(x, i)μ(dx), B ∈ S, i ∈ I. B

Proof. Let M (B × {i}) denote the number of points of N in B that are assigned to Ni . That is, M (B × {i}) = Ni (B). Clearly, M is a marked Poisson process on S × I associated with N , where the marks have the distribution p(x, i). Since M has independent increments and the subprocesses Ni represent M on the disjoint subsets S × {i}, for i ∈ I, they are independent Poisson processes. Furthermore,  p(x, i)μ(dx). E[Ni (B)] = E[M (B × {i})] = B

The preceding result for partitions is the opposite of the result that a sum of independent Poisson processes is also Poisson (recall Theorem 22). Here is a typical partition model.

198

3 Poisson Processes

Example 42. Suppose telephone calls in a region S of the USA occur according to a space-time Poisson process M on R+ × S, where M ((0, t] × B) denotes the number of calls connected in the subregion B ⊆ S in the time interval (0, t], and E[M ((0, t] × B)] = λtμ(B). There are three types of calls: (1) Long distance calls outside the USA. (2) Long distance calls within the USA. (3) Local calls. The calls are independent, and a call at time t and location x is a type i call with probability p(t, x; i), i = 1, 2, 3. Then by Proposition 41, the number of type i calls occur according to a space-time Poisson process Mi with   t

E[Mi ((0, t] × B)] = λ

p(s, x; i)μ(dx)ds. 0

B

Furthermore, M1 , M2 , M3 are independent and M = M1 + M2 + M3 . The next two examples illustrate indirect uses of thinning. Example  43. Simulating a Non-Homogeneous Poisson Process. Suppose that N = n δXn denotes a !Poisson process on a bounded Euclidean space S with mean measure Λ(A) = A λ(x) dx and rate function λ(x). Let us represent N as a thinning of a homogeneous Poisson process ¯ =  δX¯ on S with rate 1. Accordingly, suppose that Yn are locationN n n ¯ } = p(X ¯ n ) = 1 − P {Yn = 0|N} ¯ ¯ n with P {Yn = 1|N dependent marks of X ¯ where p(x) = λ(x)/Λ(S). These marks form a p(x)-thinning of N and the resulting process is  d Yn δX¯ n = N. n

This thinning representation of N justifies the following procedure from ¯ . Construct a realization of the [79] for simulating N via a realization of N ¯ point locations of N , say x¯1 , . . . , x ¯m in S. Next, independently thin the points xn ) and it is deleted otherwise, such that x¯n is retained with probability p(¯ for 1 ≤ n ≤ m. Then the retained points x1 , . . . , x form a realization of N , because of its representation above. Example 44. Terminating Poisson Process. Suppose that errors in a software package occur (while it is running) at times Tn that form a Poisson process on independent of R+ with rate λ. Each error is detected with probability 1 − p ν everything else. Then the time to detect the first error is Tν = i=1 Xi , where X1 , X2 , . . . are the exponential times between errors and ν is the number of errors until the first one is detected. The ν is a random variable independent of the Xn with the geometric distribution P {ν = n} = (1 − p)pn−1 , n ≥ 1. The Tν can be viewed as the time at which the Poisson process terminates. Exercise 5 shows that Tν is exponentially distributed with rate (1 − p)λ, since it is a geometric sum of exponential variables. An alternate proof is to consider the process of detected errors as a thinning of the Poisson error process, where 1−p is the probability of retaining a point. The resulting thinned process is a Poisson process with rate (1 − p)λ, and Tν

3.10 Partitions and Translations of Poisson Processes

199

is the time to its first point. This proves Tν is exponentially distributed with rate (1 − p)λ. Splitting and merging of flows in a network, as we now show, are typical examples of partitioning and summing of point processes. Example 45. Routing in a Graph. Consider the directed graph shown in Figure 3.1 in which units are routed in the directions of the arrows. Let Nij (t) denote the number of units that are routed on the arc from node i to node j in the time interval (0, t]. Assume that items enter the graph by independent Poisson processes N0j , j = 1, 2, 3 on R+ with respective rates λ0j , j = 1, 2, 3. Upon entering the graph, each item is routed independently through the graph according to the probabilities on the arcs, and there are no delays at the nodes (travel through the graph is instantaneous). For instance, an item entering node 3 is routed to node 5 or node 6 with respective probabilities p35 and p36 , where p35 + p36 = 1. - 2m p24 - 4m >  p12  6p54 p23  λ01- m 5m 1  p13 Z > p35 p56 Z ?  ? λ03 Z~ 3m Z  - 6m λ02

p36

-

-

Fig. 3.1 Partitioning and Merging of Flows

Our results on partitions and sums of Poisson processes yield the following properties. First note that flows N12 and N13 are independent Poisson processes with rates λ12 = p12 λ01 and λ13 = p13 λ01 , since they form a partition of N01 . Next, the flow into node 2 is the sum N02 + N12 (of independent flows) and hence is Poisson with rate λ02 + p12 λ01 . Similar properties extend to the other flows in the graph. Specifically, each flow Njk from j to k is a Poisson process, and one can evaluate their rates λjk in the obvious way. For instance, knowing λ12 and λ13 as mentioned above, λ23 = p23 (λ02 + λ12 ), λ35 = p35 (λ03 + λ13 + λ23 ),

λ36 = p36 (λ03 + λ13 + λ23 ), λ60 = λ36 + p56 λ35 .

Also, some of the flows are independent (denoted by ⊥), while some are not independent (denoted by ⊥). Examples are N12 ⊥ N13 , N36 ⊥ N56 , N36 ⊥ N24 , N13 ⊥ N24 N12 ⊥ N24 , N35 ⊥ N40 , N13 ⊥ N60 , N23 ⊥ N40 .  In addition,  the flow Ni = j Nji through each node i is a Poisson process with intensity j λji . Clearly all the Ni ’s are dependent. If the arc between

200

3 Poisson Processes

5 and 4 did not exist, however, then N4 would be independent of N3 , N5 , and N6 . We end this section with another natural transformation of a Poisson process involving translating its points within the same space. Suppose that N is a Poisson process on S = Rd with intensity measure μ. Assume that a point of N at x is independently translated to another location x+Y by a random vector Y in S that has a distribution Gx (·). That is, x is mapped into a set B ⊆ S by a probability kernel p(x, B) = Gx (B − x), where B − x = {y − x : y ∈ B}. Let M (A × B) denote the number of points of N in A that are translated into B. Then the process N  (B) = M (S × B) denotes the number of points of N translated into B. The definition of marked Poisson processes yields the following result. Proposition 46. (Translation of a Poisson process) The translation processes M and N  defined above are Poisson processes and, for A, B ⊆ Rd+ ,   E[M (A × B)] = Gx (B − x)μ(dx), E[N  (B)] = Gx (B − x)μ(dx). A

S

Example 47. Trees in a Forest. The locations (Xn , Yn ) of a certain type of tree in a forest form a Poisson process with intensity measure μ. Suppose the height of a tree at a location (x, y) has a distribution Gx,y (·).  That is, the height Zn of the tree at location (Xn , Yn ) is a mark, and M = n δ(Xn ,Yn ,Zn ) forms a marked Poisson process with   Gx,y (b)μ(dx dy). E[M (A × B × (0, b])] = A

B

After several years of growth, it is anticipated that the increase in height for a tree has a distribution H(x,y,z) (·), where (x, y) is the location and z is the original height. In other words, the increases Zn are p-marks of (Xn , Yn , Zn ) with p((x, y, z), ·) = H (x,y,z) (·). Then the collection of trees is depicted by the point process M  = n δ(Xn ,Yn ,Zn +Zn ) . By Proposition 46, M  is a Poisson process with     E[M (A × B × (0, b])] = H(x,y,z) (b − z)Gx,y (dz)μ(dx dy). A

B

R+

In the preceding example, a little more realism could be added by considering the possibility that while some trees grow as indicated, other trees may die according to a location-dependent thinning. Then one would have a combined translation–thinning transformation. Similarly, complicated systems might involve transformations involving a combination of translations, thinnings, partitions, deterministic maps, and random transformations.

3.11 Markov/Poisson Processes

201

3.11 Markov/Poisson Processes In this section, we discuss a discrete-time Markov chain whose state at each time is a spatial Poisson process. This Markov/Poisson process is formulated by successive transformations of Poisson processes. This model reveals an interesting property of invariant measures of Markov chains. We will describe a Markov/Poisson process in the context of a particle system. Consider a family of particles that move about in a space S (e.g., Rd , or a countable set) as follows. At time 0, the particles are located in S according to a point process N0 on S. Thereafter, each particle moves independently in S at discrete times as if it were a Markov chain3 on the space S with the one-step transition kernel p(x, B). That is, a particle located at x at time n moves into a set B at time n+1 with probability p(x, B). Then a particle in state x at time 0 will be in a set B at time n with probability pn (x, B). These n-step probabilities are defined by the recursion4  pn (x, B) = pn−1 (y, B)p(x, dy), n ≥ 1. S

As in the setting of countable state space Markov chains, a measure μ on S is an invariant measure of p(x, B) if  p(x, B)μ(dx) = μ(B), B ∈ S. (3.23) S

Consider the point process Nn on S that represents the locations of the points at time n. The sequence Nn is a discrete-time Markov chain that takes values in the set of counting measures on S. This follows since the point locations of Nn+1 depend on N0 , . . . , Nn only through the point locations of Nn and the one-step transition probabilities (which are functions of the p(x, B) that do not depend on n). As above, we will analyze the Markov chain Nn via the marked point processes Mn on S 2 , where Mn (A × B) denotes the number of particles in A at time 0 that are in B at time n. The sequence Mn is a Markov chain for the same reason that Nn is. Theorem 48. Suppose N0 is a Poisson process with intensity measure μ. Then each Mn is a Poisson process on S 2 with  pn (x, B)μ(dx), A, B ∈ S, E[Mn (A × B)] = A 3

The definition of a Markov chain in Chapter 1 extends readily to uncountable state spaces, as is the case here. This example should be understandable by thinking of the probability kernel as a transition probability for a countable state space and interpreting the notions of invariant and stationary distributions as one would for a countable state space. 4

When S is countable, the matrix (pn (x, y)) is the nth product of the matrix (p(x, y)).

202

3 Poisson Processes

! and Nn is a Poisson process with E[Nn (B)] = S pn (x, B)μ(dx). If in addition, the intensity μ for N0 is an invariant measure of p(x, B), then {Nn : n ≥ 0} is a stationary Markov chain, and each Nn is a Poisson process on S with intensity μ. Proof. Think of Mn as a transformation of N0 in which a point of N0 at x is independently mapped into a point of Mn in a set A × B with probability kernel r(x, A × B) = 1(x ∈ A)pn (x, B). Then by Theorem 36, Mn is a rmarked Poisson process associated with N0 and its intensity is given by   r(x, A × B)μ(dx) = pn (x, B)μ(dx). E[Mn (A × B)] = S×S 

A

Furthermore, Nn is also Poisson and its intensity as shown is locally finite since μ is. This proves the first assertion. Next, suppose that μ is an invariant measure for N0 . Then an induction ! argument using (3.23) shows that S pn (x, B)μ(dx) = μ(B), B ∈ S, n ≥ 1. Thus E[Nn (B)] = μ(B), and so the Markov chain Nn is stationary. Consider the stationary Markov/Poisson Particle process Nn described in the preceding theorem. Being a stationary Markov chain, the distribution of N0 (a Poisson process on S with intensity μ) is the stationary and limiting distribution of the chain. This distribution, as we saw in Chapter 1, describes many performance measures of the particle system. Here is an illustration. Example 49. Set with no particles. Let us see what we can say about how likely it is that there are no particles in a set B ∈ S when μ(S) < ∞. From the stationary nature of the Markov chain Nn , it is clear that the portion of time that a set B contains no particles is P {N0 (B) = 0} = e−μ(B) . Next, consider the average duration of time W (B) that the set B is empty; n W (B) = limn→∞ n−1 k=1 Wk (B), where W1 (B), W2 (B), . . . are the successive durations of (discrete) time that B is empty. We will determine W (B) by the Little law in Theorem 57 in Chapter 2 for the artificial queueing process Qn = 1(Nn (B) = 0) (this is 1 when B is empty and 0 otherwise). The average queue length (in discrete time) is L(B) = lim n−1 n→∞

n 

Qk = P {N0 (B) = 0} = e−μ(B)

k=1

Also, the rate at which the queue becomes empty is λ(B) = lim n−1 n→∞

n 

1(Nk−1 (B) > 0, Nk (B) = 0)

k=1

= P {N0 (B) > 0, N1 (B) = 0} a.s. Using the stationarity of Nn ,

a.s.

3.12 Poisson Input-Output Systems

203

λ(B) = P {N1 (B) = 0} − P {N0 (B) = 0, N1 (B) = 0} = e−μ(B) − e−μ(B) P {N1 (B) = 0|N0 (B) = 0}. By the sample process representation of the Poisson process N0 on B c , a typical point in B c is located in a set C ⊆ B c with probability μ(C)/μ(B c ). Then the probability that a typical point in B c does not enter B in the next ! c step is r = B c p(x, B )μ(dx)/μ(B c ). Using this, P {N1 (B) = 0|N0 (B) = 0} = E[P {N1 (B) = 0|N0 (B) = 0, N0 (B c )}] c c = E[rN0 (B ) ] = e−μ(B )(1−r) . Substituting this expression in the preceding display yields λ(B) = e−μ(B) [1 − e−

! Bc

p(x,B)μ(dx)

].

Since this quantity is positive, Theorem 57 in Chapter 2 ensures that the limit W (B) exists and L(B) = λ(B)W (B). Consequently, W (B) = [1 − e−

! Bc

p(x,B)μ(dx) −1

]

.

3.12 Poisson Input-Output Systems This and the next section show how one can use marked Poisson processes to model many processing systems with Poisson input, and arrival-dependent service times, and no queueing. Here we describe Mt /Gt /∞ systems, which are time-dependent versions of a classical M/G/∞ system. The results involve formulating processes of interest as functions of a space-time Poisson input process, and then characterizing several system features by applying the results above for translations, projections and random transformations. Consider a general processing system that operates as follows. Items arrive at times that form a Poisson process on R+ with intensity measure μ. An item that arrives at time t spends a random amount of time in the system that has a distribution Gt (·) and then departs. This sojourn time is independent of the other items in the system and everything else. The items may arrive in batches at any time t for which μ({t}) > 0; the batch size has a Poisson distribution with mean μ({t}). The items in this batch may not depart at the same time since their sojourn times are independent. Let Q(t) denote the quantity of items in the system at time t that arrived after time 0. We are not considering items that may be in the system at time 0. The process {Q(t) : t ≥ 0} is an Mt /Gt /∞ process with time-dependent arrivals and services. The process Q(t) is a typical model for the quantity of items in a service system with a large number of parallel servers (envisioned as infinite servers)

204

3 Poisson Processes

in which there is essentially no queueing prior to service. For instance, Q(t) could be the number of: (1) Computers being used in a wireless network with a high capacity. (2) Groups of people dining in a cafeteria. (3) Vehicles in a parking lot. (4) Patients in a hospital. (5) Calls being processed in a call center. To analyze the process Q(t), the first step is to define it by  the system data. The data is represented by the marked point process M = n δ(Tn ,Vn ) on R2+ , where Tn is the arrival time of the nth item and Vn is its sojourn or service time. The Vn are location-dependent marks of the Tn with the distribution p(t, B) ! = Gt (B), and M is a space-time Poisson process with E[M (A × B)] = A Gs (B)μ(ds). Since the quantity Q(t) is a function of the arrival times Tn and departure times Tn + Vn of the items, let us consider the marked point process of these arrival/departure times, which is  N= δ(Tn ,Tn +Vn ) , on S = {(t, u) ∈ R2+ : u ≥ t}. n

This process N is a transformation of M by the map g(t, v) = (t, t + v). Then N is a space-time Poisson process by Theorem 32. In particular, the quantity N ((a, b] × (c, d]), where b ≤ c, is the number of items that arrive in (a, b] and depart in (c, d], and its mean is  E[N ((a, b] × (c, d])] = [Gs (d − s) − Gs (c − s)]μ(ds). (a,b]

Using the preceding notation, the quantity of items in the system at time t is defined by  1(Tn ≤ t, Tn + Vn > t) = N ((0, t] × (t, ∞]). Q(t) = n

Since N is a Poisson process, it follows that Q(t) has a Poisson distribution with  E[Q(t)] = [1 − Gs (t − s)]μ(ds). (3.24) (0,t]

Although the distribution of each Q(t) is Poisson, the entire process is not. In addition to analyzing the number of items in the system, one may want information about the departure process. This is useful when the departures form an arrival process into another service system. Now, the total number of  departures in (0, t] is D(t) = n 1(Tn + Vn ≤ t). That is, D is the projection of N on its second coordinate, and so D is a Poisson process with  E[D(t)] = E[N (t) − Q(t)] = Gs (t − s)μ(ds). (0,t]

3.12 Poisson Input-Output Systems

205

Example 50. M/G/∞ System. Consider the special case of the preceding system in which the Poisson arrival process is homogeneous with rate λ and the service distribution Gt (·) = G(·) is independent of t. This is a classical M/G/∞ system with arrival rate λ and service distribution G. In this case, Q(t) has a Poisson distribution and from (3.24) (with u = t − s),  E[Q(t)] = λ 0

t

[1 − G(u)]du.

Suppose that G has a mean α. Then it follows by Exercise 32 that the limiting distribution of Q(t) is Poisson with rate λα, as t → ∞. In addition, the total number of departures D(t) in the time interval (0, t] is a Poisson process with  t E[D(t)] = λ G(s)ds. 0

An abstraction of the Mt /Gt /∞ system we just discussed is as follows. Example 51. Poisson Input-Output-Mobility Model. Consider a system in which items enter a space S at times T1 ≤ T2 ≤ . . . that form a Poisson process with intensity measure μ. The nth item that arrives at time Tn moves in the space S for a while and then exists the system (by entering the outside state 0). The movement is determined by a stochastic process Yn = {Yn (t) : t ≥ 0} with state space S ∪ {0}, where the outside 0 is an absorbing state (Yn (t) = 0 for all t > inf{s : Yn (s) = 0}). In particular, the nth item enters S at the location Yn (0), and, at time t > Tn its location is Yn (t − Tn ). Let Y denote a function space that contains the sample paths of Yn (e.g., Y could be a space of real-valued functions that are continuous, or piece-wise constant). Assume the Yn are location-dependent marks of Tn with distribution p(t, ·), which is the conditional distribution of the process Yn starting at time t. For simplicity, assume Yn depends on t only through its initial value Yn (0) (the entry point in S of the nth point), whose distribution is denoted by Ft (·). Then conditioning on Yn (0),  p(t, ·) = P {Yn ∈ ·|Yn (0) = x}Ft (dx). (3.25) S

 In other words, the system data is the process M = n δ(Tn ,Yn ) on R+ × Y, which is a space-time Poisson process by Theorem 32. Now, the number of items in the set B ∈ S at time t is given by   1(Tn ≤ t, Yn (t − Tn ) ∈ B) = δgt (Tn ,Yn ) (B), Nt (B) = n

n

206

3 Poisson Processes

where gt (s, y) = y(t − s) and y ∈ Y. Since Nt is a transformation of the Poisson process M by the map gt , it follows by Theorem 32 that Nt is a Poisson process on S for each fixed t, and from (3.25),   E[Nt (B)] = P t−s (x, B)Fs (dx)μ(ds), (0,t]

S

where P t (x, B) = P {Yn (t) ∈ B|Yn (0) = x}. Next, note that the number of departures from the set B in the time interval (a, b] is  1(h(Tn , Yn ) ∈ (a, b] × B), D((a, b] × B) = n

where h(s, y) = (s, y(t−s)). This D is a transformation of the Poisson process M by the map h, and so by Theorem 32, D is a space-time Poisson process on R+ × S with   E[D((0, t] × B)] = P t−s (x, {0})Fs (dx)μ(ds). (0,t]

B

3.13 Network of Mt/Gt/∞ Stations In this section, we show how the ideas in the preceding section extend to the analysis of flows in a stochastic network of Mt /Gt /∞ stations. The network dynamics are determined by marks of Poisson processes, and the analysis amounts to formulating appropriate Poisson processes that represent parameters of interest, and then specifying their intensity measures. Consider a network of m service stations (or nodes) that operate as follows. Items enter the network at times T1 ≤ T2 ≤ . . . that form a Poisson process with intensity measure μ. The nth item entering the network at time Tn selects, or is assigned, a random route Rn = (Rn1 , . . . , RnLn ) through the network, where Rnk ∈ {1, . . . , m} denotes the kth node the item visits, and the length 1 ≤ Ln ≤ ∞ may be random and depend on the Rnk . After visiting the last node RnLn on its route, the item exits the network and enters node 0 (“outside” the network) and stays there forever. Associated with the nth arrival is a vector of nonnegative sojourn (or visit) times Vn = (Vn1 , . . . , VnLn ), where Vnk is the item’s sojourn time at node Rnk . The time at which the item departs from node Rnk is τnk = Tn +

k 

Vnj ,

k ≤ Ln ,

j=1

where τnLn is the time at which the item exits the network.

3.13 Network of Mt /Gt /∞ Stations

207

The main assumption is that the route and waiting time vectors Yn = (Rn , Vn ) are marks of the arrival times Tn . This implies there are no interactions among the items that affect their waiting times, so each node operates like an Mt /Gt /∞ system. As above, we consider only those items that enter the network “after” time 0. In summary,  the system data is represented by the space-time Poisson process M = n δ(Tn ,Yn ) . Many features of the network are expressible by space-time Poisson processes of the form  Nt = δ(Tn ,gt (Tn ,Yn )) , (3.26) n

 E[Nt ((a, b] × B)] =

(a,b]

P {gt (Tn , Yn )) ∈ B|Tn = s}μ(ds).

(3.27)

One need only define the function gt for the application at hand; in some cases, gt and Nt do not depend on t. Note that each Nt is a Poisson process since it is a deterministic transformation of the Poisson process M . Typical uses of these space-time Poisson processes are as follows. Locations of Items at Time t. The space-time Poisson process describing where the items are located is Nt ((a, b] × B) = # of items that arrive in the time interval (a, b] that are in B ⊆ {0, 1, . . . , m} at time t. The location of the item at time t that arrives at time Tn is 0 if τnLn ≤ t gt (Tn , Yn ) = Rnk if τn(k−1) ≤ t < τnk , for some k ≤ Ln .

(3.28)

Consequently, the quantities Qi (t) = Nt ((0, t] × {i}), 1 ≤ i ≤ m, at the nodes at a fixed time t are independent Poisson random variables with  P {gt (Tn , Yn ) = i|Tn = s}μ(ds). (3.29) E[Qi (t)] = (0,t]

Departure Process. The space-time Poisson process describing the times at which items exit the network is N ((a, b] × B) = # of items arriving in (a, b] whose exit time from the network is in B ⊆ R+ . The item arriving at Tn exits the network a time g(Tn , Yn ) = τnLn . Usually, the mean values (3.27) of the space-time Poisson processes would be determined by the distributions of the routes and sojourn times of the items. The routes depend on the structure of the network and the nature of

208

3 Poisson Processes

the items and services. A standard assumption is that the routes are independent and Markovian, where pjk denotes the probability of an item moving to node k upon departing from node j. Then the probability of a particular route (r1 , . . . , r ) of nonrandom length  is p0r1 · · · pr 0 . Another convention is that there are several types of items and all items of the same type take the same route. In this case, the probability of a route is the probability that the item entering the network is the type that takes that route. The simplest sojourn times at a node are those that are i.i.d., depending on the node and independent of everything else. Then the sums of sojourn times are characterized by convolutions of the distributions. The next level of generality is that the service times are independent at the nodes, but their distributions may depend on the route as well as the node. An example of dependent service times is that an item entering a certain subset of routes is initially assigned a service time according to some distribution and then that time is its service time at “each” node on its route. Here is a typical example of a network. Example 52. Acyclic Network. Consider the stochastic network shown in Figure 3.2 that operates as described above with the following particular properties. Items arrive at the nodes 1, 2 and 3 from outside according to independent Poisson processes with respective rates λ1 , λ2 , λ3 . The sojourn or service times at the nodes are independent random variables, and the sojourn times at node i have the distribution Gi (·). When an item ends its sojourn at node 1, it departs and enters node 2 with probability p12 , or it enters node 3 with probability p13 = 1 − p12 . Analogously, departures from node 2 enter node 3 with probability p23 , or enter node 4 with probability p24 = 1 − p23 . Also, departures from node 3 enter node 5 (p35 = 1), and departures from nodes 4 and 5 exit the network.

p12

 - 1m  Z Z p13

- 2m p24- 4m > 

-

p23

? ZZ ~ 3m -

- 5m

-

Fig. 3.2 Acyclic Network

The times T1 < T2 < . . . at which items enter the network from outside form a Poisson process with rate λ = λ1 + λ2 + λ3 , since this process is the sum of the three independent Poisson processes flowing into nodes 1, 2 and 3. The probability that an arrival at any time Tn enters node i is λi /λ. This is the probability that the exponential time of an arrival at i is smaller than those exponential arrival times at the other nodes; see Exercise 2. The item that arrives at time Tn traverses a route Rn = (Rn1 , . . . , Rnn ) in R (the

3.13 Network of Mt /Gt /∞ Stations

209

set of all routes), and its sojourn times at the n nodes on the route are Vn = (Vn1 , . . . , Vnn ). The joint distribution of these marks Yn = (Rn , Vn ) as functions of the network data λi , Gi and pij is P {Rn = r, Vn ≤ v|Tn } = p(r)

 

Grk (vk ),

k=1

where p(r) = (λr1 /λ)pr1 r2 · · · pr−1 r is the probability of route r = (r1 , . . . , r ). To analyze the quantity of items on the routes as well as at the nodes, let us consider the space-time point process Nt ((a, b] × B) = # of items arriving in (a, b] whose route and node location (r, i) is in B ⊆ R × {0, 1, . . . , m} at time t. As in (3.26), Nt is a Poisson process on R+ × R × {0, 1, . . . , m}, for fixed t, where gt (Tn , Yn ) = (Rn , ht (Tn , Yn )) and ht (Tn , Yn ) is defined by (3.28). The item that enters at Tn is at node ht (Tn , Yn ) at time t. In particular, the quantity of items  Nt ((a, b] × {r} × {i}) Qi (t) = r∈Ri

at node i at time t has a Poisson distribution. The sum here is over all routes in the set of routes Ri that contain node i. Also, for a fixed t, since Nt has independent increments, it follows that Qi (t) is independent of Qj (t) if Ri and Rj are disjoint. For instance, Q4 (t) is independent of Q3 (t) and Q5 (t). The next step is to evaluate the intensity of Nt . Let Pru (i) denote the conditional probability that an item is at node i given that it is on route r for a time u since it entered the network. By the independence of the sojourn times, if i = 0 Gr1 · · · Gr (u) u Pr (i) = Gr1 · · · Grk−1 Grk (u) if rk = i = 0, for some k ≤ . Here G(t) = 1 − G(t). For instance, the conditional probability that an item is at node 3 at time t, given that it enters route r = (1, 2, 3, 5) at time s, is G1 G2 G3 (t − s) = P {τn2 ≤ t < τn3 |Rn = r, Tn = s} = P {ht (Tn , Yn ) = 3|Rn = r, Tn = s}. Then from (3.27), it follows that 

b

E[Nt ((a, b] × {r} × {i})] = λp(r) a

Prt−s (i)ds.

210

3 Poisson Processes

!b The last integral equals a Pru (i)du, under the change-of-variable u = t − s. For example, the number of items arriving in (0, t] that are on route r = (1, 2, 3, 5) and in node 3 at time t has a Poisson distribution with mean  E[Nt ((0, t] × {r} × {3})] = λ1 p12 p23

0

t

G1 G2 G3 (u)du.

The process Nt yields considerable information about numbers of items at nodes and on routes as well. For instance, the quantity Q3 (t) of items at node 3 at time t is the sum of the quantities of items on the routes in R3 = {(3, 5), (2, 3, 5), (1, 3, 5), (1, 2, 3, 5)}, all the routes containing node 3. Then Q3 (t) has a Poisson distribution and its mean is E[Q3 (t)] =

 t λ3 G3 (u) + λ2 p23 G2 G3 (u) + λ1 p13 G1 G3 (u) 0  +λ1 p12 p23 G1 G2 G3 (u) du.

 The term in brackets is λ r∈R3 p(r)Pru (3). Similarly, the quantity of items on route r at time t is Qr (t) = Nt ((0, t] × {r} × {r1 , . . . , r }). This quantity, being part of the Poisson process Nt , has a Poisson distribution whose mean is easy to calculate. For instance,  t E[Q(2,4) (t)] = λ2 p24 [G2 (u) + G2 G4 (u)]du. 0

Let us now consider the departure times of the items from the nodes, which are depicted by the process N ((a, b] × B) = # of items arriving in (a, b] whose departure times from the 5 nodes are in B ⊆ R5+ . The departure times are well-defined since an item cannot visit a node more than once. Now, N is a space-time Poisson process as in (3.26) and (3.27), and the departure times at the 5 nodes are given by g(Tn , Yn ) = (g1 (Tn , Yn ), . . . , g5 (Tn , Yn )), n τnk 1(Rnk = i), the departure time from node i of where gi (Tn , Yn ) = k=1 the item that enters at Tn . In particular, the departure process at node i is Di (t) = N ((0, t] × {(t1 , . . . , t5 ) ∈ R5+ : ti ≤ t}),

t ≥ 0.

3.14 Cox Processes

211

Now, Di is a Poisson process since it is the projection on the ith departuretime coordinate of the Poisson process N . Its mean is  t P {gi (Tn , Yn ) ≤ t|Tn = s}ds. (3.30) E[Di (t)] = λ 0

For instance,  t λ3 G3 (u) + λ2 p23 G2 G3 (u) + λ1 p13 G1 G3 (u) E[D3 (t)] = 0  + λ1 p12 p23 G1 G2 G3 (u) du. Similarly to the independence of quantities at the nodes, processes Di and Dj are independent if Ri and Rj are disjoint. For instance, D4 is independent of D3 and D5 .

3.14 Cox Processes This section describes a Poisson process with a random intensity measure. The random intensity might represent a random environment or field that influences the Poisson locations of points. Because the intensity is random, the resulting process is rather general, but we will show that many of its properties can be characterized by features of the parent Poisson process and the intensity process. Chapter 5 covers similar material for Brownian motion in a random environment. Suppose that N is a point process on a space S and η is a random measure on S that is locally finite a.s. and is defined on the same probability space as N . The N is a Cox process directed by η if, conditioned on η, the N is a Poisson process with conditional intensity measure η. Equivalently, the conditional Laplace functional of N given η is  + E[e−N f | η] = exp{− (1 − e−f (x) )η(dx)}, a.s. f ∈ CK (S). (3.31) S

In particular, P {N (B) = n | η} = e−η(B) η(B)n /n!, and taking expectations,

 P {N (B) = n} = E e−η(B) η(B)n /n! . Note also that E[N (B)] = E[η(B)].

212

3 Poisson Processes

A Cox process is sometimes called a conditional Poisson process, a doubly stochastic Poisson process, or a Poisson process in a randomly changing environment. Several characterizations of Cox processes appear in [61]. An important observation is that a Cox process on R+ can be characterized as a homogeneous Poisson process with a random time parameter. Remark 53. A point process N on R+ is a Cox process directed by η if and d only if N (·) = N1 (η  (·)), where N1 and η  are defined on a common probability d

space such that N1 is a Poisson process with rate 1 and η  = η. This follows by the definition above and consideration of the Laplace functionals of the processes. In case η is strictly increasing, one can show as in Exercise 43 in Chapter 5 that N (·) = N1 (η(·)) a.s., where N1 is defined on the same space as X and η. In some instances, one can formulate a Cox process as a transformation of a Poisson process that has one more layer of randomness than those above. For instance, let N be a Poisson process on R+ with intensity measure μ. Consider a transformation of N in which a point of N at t is mapped to a location γ(t), where γ(t) is a stochastic process on R+ that is independent of N . Then the transformed process N  (B) = M (γ −1 (B)) is a Cox process ! directed by η(B) = R+ 1(γ(t) ∈ B)μ(dt), provided this is a.s. locally finite. This follows since, by Theorem 32, N  is Poisson when γ(t) is deterministic. As a second example, let N be a Poisson process on R+ with intensity measure μ. Assume that N is partitioned into m subprocesses N1 , . . . , Nm by the rule that a point of N at t is assigned to the subprocess with the label α(t), where α(t) is a stochastic process on {1, . . . , m} that is independent of N . Then as in Proposition 41, N1 , . . . , Nm are conditionally independent Poisson processes given α(·). Hence each Ni is a Cox process directed by  η(B) = 1(α(t) = i)μ(dt). B

Of course, the Ni are not independent since they all depend on α. Because Cox processes are essentially Poisson processes with an extra expectation to account for the randomized mean measure, most results for Poisson processes have counterparts for Cox processes. Here is one instance. Example 54. If N1 , . . . ,Nm are Cox processes on S directed by η1 , . . . , ηm , respectively, and (N1 , η1 ), . . . , (Nm , ηm ) are independent, then N = N1 + · · · Nm is a Cox process directed by η = η1 + · · · ηm . Here is an illustration of a Cox input process for a Mt /Gt /∞ system. Example 55. Regenerative-Modulated Poisson Process. In computer and telecommunications systems, a standard model for the occurrences of an event !t in time is a Cox process N on R+ directed by η(t) = 0 f (Y (s)) ds, where Y (t) is a regenerative process on a countable state space S that models a changing environment in which events occur. For instance, a flow of data

3.14 Cox Processes

213

may be Poisson, but dependent on an environment (type or source of the data, congestion in a network, etc.) that is changing according to Y (t). Since the Cox process has the form N (t) = N1 (η(t)), its behavior far out in time is related to the limiting behavior of Y (t). For instance, suppose Y (t) has a limiting distribution p on S. Then the limiting average of N is  f (i)pi , a.s. as t → ∞. t−1 N (t) → λ = i∈S

This follows since by the SLLNs for Poisson processes and regenerative processes, N1 (t)/t → 1 and η(t)/t → λ a.s., and so t−1 N (t) = (η(t)/t)N1 (η(t))/η(t) → λ,

a.s. as t → ∞.

Now, consider a variation of the Mt /G/∞ system in which items arrive for service according to the preceding Cox process N , and G is the  distribution of the independent service times. The system data is M = n δ(Tn ,Vn ) on R2+ , where Tn is the arrival time of the nth item, and Vn is its sojourn or service time. Analogously to marked Poisson processes, M is a space-time Cox process directed by η with  Gs (v)f (Y (s))ds. E[M ([0, t) × [0, v))|η] = (0,t]

Here dη(s) = f (Y (s))ds.  Consider the quantity of items Q(t) = n 1(Tn + Vn > t) in the system at time t. As in (3.24), the “conditional” distribution of Q(t) given η is Poisson with  t [1 − G(t − s)]f (Y (s))ds. E[Q(t)|η] = 0

Furthermore, under the additional assumptions that Y (t) is stationary and G has a mean α, we have  P {Y (0) = i}(f (i)α)n e−f (i)α /n!. (3.32) lim P {Q(t) = n} = t→∞

i∈S

This limit is a conditional Poisson distribution with random mean αf (Y (0)). To prove (3.32), note that

 P {Q(t) = n} = E (E[Q(t)|η])n e−E[Q(t)|η] /n! ,  t d d [1 − G(u)]du → αf (Y (0)). E[Q(t)|η] = f (Y (0)) 0

In light of these properties, (3.32) follows from the dominated convergence theorem for convergence in distribution (see the Appendix).

214

3 Poisson Processes

3.15 Compound Poisson Processes If a Poisson process has real-valued marks at its points, then the cumulative value of the marks in time (or in a space) is a compound Poisson process. This section is a brief description of such processes. We first consider a classical compound Poisson process in time (as mentioned in Example 14). Definition 56. Let N (t) be a homogeneous Poisson process on R+ with rate λ, and let Yn be real-valued random variables that are i.i.d. with distribution F and are independent of N . The stochastic process N (t)

Z(t) =



t ≥ 0,

Yn ,

n=1

is a compound Poisson process with rate λ and distribution F . The name comes from the fact that Z(t) has a compound Poisson distribution with rate λt and distribution F : P {Z(t) ≤ z} =

∞ 

e−λt (λt)n F n (z)/n!,

z ∈ R.

(3.33)

n=0

Indeed, condition on N (t) and use P {Z(t) ≤ z|N (t) = n} = F n (z). Similar conditioning on N (t) yields Var[Z(t)] = λtE[Y12 ],

E[Z(t)] = λtE[Y1 ],

provided these moments exist. Finally, if the moment generating function φ(α) = E[eαY1 ] exists for some α in a neighborhood of 0, then E[eαZ(t) ] = e−λt(1−φ(α)) . The stationary independent increments of the Poisson process N and the i.i.d. property of its marks yield the following result. Theorem 57. A compound Poisson process {Z(t) : t ≥ 0} has stationary, independent increments: Z(t1 ) − Z(s1 ), . . . , Z(tn ) − Z(sn ), for s1 < t1 < · · · sn < tn , are independent; and d

Z(s + t) − Z(s) = Z(t), s, t ≥ 0.  Proof. Using the process M = n δ(Tn ,Yn ) , we can write   Z(t) = Yn 1(Tn ≤ t) = yM ((0, t] × dy). n

R

3.15 Compound Poisson Processes

215

Under the assumptions, M is a space-time Poisson process with E[M ((s, s + t] × B)] = λtF (B). Now, for any s1 < t1 < . . . sn < tn , consider the increments  yM ((si , ti ] × dy), 1 ≤ i ≤ n. Z(ti ) − Z(si ) = R

They are independent since the point processes M ((si , ti ] × ·), for 1 ≤ i ≤ n, on R are independent, because M has independent increments. Next, note that E[M ((s, s+t]×B)] = E[M ((0, t]×B)]. Since M is Poisson and its distribution is uniquely determined by its intensity, it follows that d M ((s, s + t] × ·)] = M ((0, t] × ·) Consequently,   d Z(s + t) − Z(s) = yM ((s, s + t] × dy) = yM ((0, t] × dy) = Z(t). R

R

Hence Z(t) has stationary increments. The classical compound Poisson process described above has several natural extensions. For instance, instead of the Yn being independent of N , suppose Yn are p-marks of Tn . Also, assume N has a general intensity μ. Then N (t) the process Z(t) = n=1 Yn , for t ≥ 0, is a location-dependent compound Poisson process with intensity measure μ and distribution p(t, ·). Many of its  properties follow directly from the fact that M = n δ(Tn ,Yn ) is a Poisson process. For instance, see Exercises 54 and 55. Also, since M is Poisson, the results above for Poisson processes extend to compound Poisson processes by using a p-marking of M , which would be a “second” marking of N as mentioned in Example 39. Exercise 54 illustrates these ideas for partitions of compound Poisson processes. There  are other relatives of compound Poisson processes of the form  M (A) = n Yn δXn (A), where N = n δXn is a Poisson process on a general space, and the marks Yn are random vectors, matrices, or elements of a group with an addition operation. Here is an example when Yn are point processes.  Example 58. Poisson Cluster Processes. Let N = n δXn denote a Poisson process on a general space S with intensity measure μ. Suppose that each point Xn generates a cluster of points in a space S  that are represented by a point process Nn . Assume the Nn are point processes on a space S  that are i.i.d. and independent of N . Then the number of points from the processes Nn in a set B that are generated by points of N in the set A is  Nn (B)δXn (A). M (A × B) = n

216

3 Poisson Processes

This defines a point process M on S × S  called a marked cluster process generated by the Poisson process N ; the M (S × ·), provided it is locally finite, is simply the cluster process on S  . N (A) Since M (A × B) = n=0 Nn (B), it follows by conditioning on N that M (A × B) has the compound Poisson distribution P {M (A × B) ≤ n} =

∞ 

e−μ(A) μ(A)k F k (n; B)/k!,

k=0

where F (n; B) = P {N1 (B) ≤ n}. Also, E[M (A × B)] = E[N (A)]E[N1 (B)] and Var[M (A × B)] = E[N (A)]Var[N1 (B)]. More general cluster processes, where the Nn are marks of Xn , are analyzed in Exercise 57.

3.16 Poisson Law of Rare Events Poisson processes are natural models for rare events in time, or rare points in a space. This is partly due to a law of rarely occurring events in which a sum of thin or rarefied point processes converges in distribution to Poisson process. A classical case for random variables is a Binomial random variable converging to a Poisson random variable as in Example 59 below. In this section, we present a generalization of this result that gives conditions under which a sum of many rare indicator random variables converges to a Poisson random variable. Analogous results under which a sum of point processes converges to a Poisson process are in the next section. Here is a classical example of the Poisson law of rare events. Example 59. Binomial Convergence to Poisson. Suppose Yn1 , . . . , Ynn are independent n Bernoulli random variables with P {Yni = 1} = pn . Then Zn = i=1 Yni has a binomial distribution with parameters n and pn . If d

npn → μ > 0 as n → ∞, then Zn → Z, where Z has a Poisson distribution with mean μ. This is a special case of the following result. Theorem 60. (Poisson Law of Rare Events) Suppose Yn1 , Yn2 , . . ., for n ≥ 1, are a countable number of independent random variables that take values 0 or 1, and satisfy the uniformly null property sup P {Yni = 1} → 0, i

as n → ∞.

Let Z be a Poisson random variable with mean μ. Then as n → ∞, Zn =

 i

d

Yni → Z

if and only if

 i

P {Yni = 1} → μ.

(3.34)

3.16 Poisson Law of Rare Events

217 d

Proof. We will use the property of Laplace transforms that Zn → Z if and only if E[e−αZn ] → E[e−αZ ]. By the independence of the Yni ,   E[e−αYni ] = (1 − cni ), α ≥ 0, E[e−αZn ] = i

i

where cni = E[1 − e−αYni ]. Also, E[e−αZ ] = e−c , where c = μ(1 − e−α ), since Z has a Poisson distribution with mean μ. From these observations, d

Zn → Z

⇐⇒

E[e−αZn ] → e−c

 (1 − cni ) → e−c .

⇐⇒

(3.35)

i

Moreover, under the assumption (3.34) and cni ≤ (1 − e−α ) < 1, it follows by Lemma 61 below for cni = cμ−1 P {Yni = 1}, that    (1 − cni ) → e−c ⇐⇒ cni → c ⇐⇒ P {Yni = 1} → μ. i

i

i

Combining this string of equivalences with (3.35) proves the assertion. The preceding proof of the Poisson convergence boils down to the following result on the convergence of real numbers. Lemma 61. Suppose cn1 , cn2 , . . ., for n ≥ 1, are a countable (possibly finite) number of real numbers in (0, a], where a < 1, that satisfy the uniformly null property as n → ∞. (3.36) sup cni → 0 i

Then, for any c > 0,  (1 − cni ) = e−c lim n→∞

if and only if

i

Proof. The assertion is equivalent to  sn = − log(1 − cni ) → c if and only if i

lim



n→∞

sn =

cni = c.

i



cni → c.

(3.37)

i

Since log(1 − cni ) = −

∞

m m=1 cni /m,

dn = sn − sn =

the difference in these sums is



c2ni

i

∞ 

cm−2 ni /m.

m=2

Using cni ≤ a and αn = supi cni , we have dn ≤

αn sn 1  2 αn sn ≤ . c ≤ 1 − a i ni 1−a 1−a

(3.38)

218

3 Poisson Processes

Now, if sn → c, then from (3.38) and (3.36) we have dn → 0, and hence sn = sn − dn → c. Similarly, if sn → c, then sn = sn + dn → c. These observations prove (3.37).

3.17 Poisson Convergence Theorems* This section contains Poisson convergence theorems for sequences of point processes. These results are extensions of the Poisson law of rare events in Theorem 60 above. The main theorem here is that a sum of many independent sparse point processes converges to a Poisson process. Consequently, certain sums of renewal processes and rare transformations of a point process converge to a Poisson process. Also included are examples justifying that Poisson processes are reasonable approximations for thinnings and partitions of a point process. We will use the following notion of weak convergence, which is reviewed in the Appendix. Suppose μ, μ1 , μ2 , . . . are probability measures on S. The w probabilities μn converge weakly to μ as n → ∞, denoted by μn → μ, if μn f → μf , as ! n → ∞, for each bounded continuous function f : S → R (recall μf = S f (x)μ(dx)). This is equivalent to lim μn (B) = μ(B),

n→∞

B ∈ Sˆμ ,

(3.39)

where Sˆμ = {B ∈ Sˆ : μ(∂B) = 0}, the set of all bounded sets whose boundary has μ-measure 0. A sequence of random elements in a metric space converges in distribution to a random element if their distributions converge weakly. In particular, a sequence of point processes Nn on S converges in distribution to N as w d n → ∞, denoted by Nn → N , if P {Nn ∈ ·} → P {N ∈ ·}. This weak convergence is equivalent to the convergence of the finite-dimensional distributions (condition (ii) in the next theorem). A few points in our analysis use the slightly more general notion of vague convergence of measures. Suppose μ, μ1 , μ2 , . . . are locally finite measures on v S. The measures μn converge vaguely to μ, denoted by μn → μ, if μn f → μf,

+ as n → ∞, for each f ∈ CK (S).

This is equivalent to (3.39) if all the measures are probability measures, so vague convergence, in this case, is the same as weak convergence. The following are several equivalent conditions for point processes to converge in distribution. Here SˆN = {B ∈ Sˆ : N (∂B) = 0 a.s.}. Theorem 62. For point processes N, N1 , N2 , . . . on S, the following statements are equivalent as n → ∞.

3.17 Poisson Convergence Theorems*

219

d

(i) Nn → N . d (ii) (Nn (B1 ), . . . , Nn (Bk )) → (N (B1 ), . . . , N (Bk )),

B1 , . . . , Bk ∈ SˆN .

+ f ∈ CK (S). (iii) Nn f → N f, + −Nn f −N f (iv) E[e ] → E[e ], f ∈ CK (S). d

A proof of this result is in [60]. Condition (ii) says the finite-dimensional distributions of Nn converge to those of N . When S is an Euclidean space, the sets Bi can be replaced by bounded rectangles. Condition (iii) relates the convergence in distribution of integrals with respect to point processes to the convergence of the processes. The convergence (iv) of Laplace functionals is d

a convenient tool for proving Nn → N , when the functionals can be factored conveniently (as in the proof of Theorem 66 below). Here is an elementary but useful fact. It justifies the convergence of a Poisson process when its intensity converges (see Exercise 54 in Chapter 4). Proposition 63. (Convergence of Poisson Processes) For each n ≥ 1, suppose Nn is a Poisson process on a space S with intensity measure μn . If v d μn → μ and μ is locally finite, then Nn → N , where N is a Poisson process with intensity μ. + Proof. From Theorem 21, we know that E[e−Nn f ] = e−μn h , for f ∈ CK (S), v −f (x) where h(x) = 1 − e . By Theorem 62, we have μn h → μh, and so

E[e−Nn f ] = e−μn h → e−μh = E[e−N f ]. d

Thus, Nn → N by Theorem 62. We are now ready to consider the convergence of sums of point processes. Here is a motivating example. n Example 64. Consider a sum N (t) = i=1 Ni (t), for t ≥ 0, where N1 , . . . , Nn are independent renewal processes. Of course, N is generally not a renewal process. However, suppose the times between renewals for each process Ni tend to be large (i.e., Fi (t) is small, where Fi is the inter-renewal distribution). Consequently, each contribution Ni (a, b] to N (a, b] would tend to be 0. In other words, each Ni rarely contributes a point to N on bounded intervals. However, if the number n of these contributions is large, it might be reasonable to approximate N by a Poisson process with intensity E[N (t)] = n E[N (t)]. i i=1 A Poisson convergence theorem justifying such an approximation is as follows. The opposite situation in which Ni (a, b] tends to be large is addressed in Exercise 59.  Theorem 65. (Sums of Renewal Processes) For n ≥ 1, let Nn (t) = i Nni (t) be a point process on R+ , where Nn1 , Nn2 , . . . is a finite or countable number of independent renewal processes and Nni has inter-renewal distribution Fni . Assume the inter-renewal times are uniformly rare in that

220

3 Poisson Processes

t ≥ 0.

lim sup Fni (t) = 0,

n→∞

i

d

Let N be a Poisson process on R+ with intensity measure μ. Then Nn → N , as n → ∞ if and only if, for each t with μ({t}) = 0,  Fni (t) = μ(t). (3.40) lim n→∞

i

Proof. This result follows by Theorem 66 below, since   P {Nni (t) ≥ 2} = Fni (t − s)Fni (ds) i

i

(0,t]

≤ sup Fni (t)



i

Fni (t),

i

and (3.43) is the same as (3.40) because P {Nni (t) ≥ 1} = Fni (t). The next result is a general Poisson convergence theorem for sums of uniformly rare point processes. Suppose that  Nn = Nni , n ≥ 1, i

is a point process on a space S, where Nn1 , Nn2 , . . . is a countable number of independent point processes on S. Assume the point processes Nni are uniformly null, meaning that lim sup P {Nni (B) ≥ 1} = 0,

n→∞

ˆ B ∈ S.

i

(3.41)

Let N be a Poisson process on S with intensity measure μ. d

Theorem 66. (Grigelionis) For the processes defined above, Nn → N , as n → ∞ if and only if  ˆ lim P {Nni (B) ≥ 2} = 0, B ∈ S, (3.42) n→∞

lim

n→∞

i



P {Nni (B) ≥ 1} = μ(B),

B ∈ Sˆμ .

(3.43)

i d

Proof. The convergence Nn → N is equivalent, by Theorem 62, to E[e−Nn f ] → E[e−N f ],

+ f ∈ CK (S).

(3.44)

Using the independence of the Nni and letting cni = E[1 − e−Nni f ], we have   E[e−Nni f ] = (1 − cni ). E[e−Nn f ] = i

i

3.17 Poisson Convergence Theorems*

221

Also, by Theorem 21, E[e−N f ] = e−μh , where h(x) = 1 − e−f (x) . Combining these observations, it follows that (3.44) is equivalent to  + (1 − cni ) → e−μh , f ∈ CK (S). (3.45) i

Keep in mind that cni is a function of f . We will complete the proof by applying Lemma 61 to establish that (3.42) and (3.43) are necessary and sufficient for (3.45). First note that cni = E[1 − e−Nni f ] ≤ P {Nni (Sf ) ≥ 1}, where Sf is the support of f . Then (3.41) implies sup cni ≤ sup P {Nni (Sf ) ≥ 1} → 0. i

i

In light of this property, we can assume cni are in (0, a] for some a < 1. Then Lemma 61 says that (3.45) is equivalent to   + E[1 − e−Nni f ] = cni → μh, f ∈ CK (S). (3.46) i

i

Therefore, it remains to show that (3.42) and (3.43) are necessary and sufficient for (3.46). To prove that (3.42) and (3.43) imply (3.46), consider   cni = E[(1 − e−Nni f )1(Nni (Sf ) = 1)] (3.47) i

i

+



E[(1 − e−Nni f )1(Nni (Sf ) ≥ 2)].

i



The last sum is bounded by i P {Nni (Sf ) ≥ 2)} which converges to 0 by assumption (3.42). The first sum on the right-hand side in (3.47) equals ηn h, where   E[Nni (B ∩ Sf )1(Nni (Sf ) = 1)] = P {Nni (B ∩ Sf ) = 1}. ηn (B) = i

i

The last sum has the same form as the sum in (3.42) minus the one in (3.43), v and so these assumptions imply ηn → μ, which yields ηn h → μh. Using the preceding observations in (3.47) proves that (3.42) and (3.43) are sufficient for (3.46). Conversely, suppose (3.46) is true. Applying this property to the function f (x) = −1(x ∈ B) log s, where B ∈ Sˆ and s ∈ [0, 1], we have  Hn (s) = E[1 − sNni (B) ] → (1 − s)μ(B), (3.48) i

222

3 Poisson Processes

since h(x) = 1 − e−f (x) = (1 − s)1(x ∈ B). Then (3.43) follows since  P {Nni (B) ≥ 1} = Hn (0) → μ(B). (3.49) i

Next, consider the factorization Hn (s) =

 i

[1 −

∞ 

sm P {Nni (B) = m}]

m=0

= (1 − s)Hn (0) +

∞  i

(s − sm )P {Nni (B) = m}.

m=2

This expression along with (3.48) and (3.49) yield  P {Nni (B) ≥ 2} ≤ Hn (s) − (1 − s)Hn (0) → 0. (s − s2 ) i

Thus (3.46) is true. These observations prove that (3.46) implies (3.42) and (3.43), which completes the proof. Theorem 66 justifies Poisson limits for sums of independent renewal processes (Theorem 65 and Exercise 59). Although Theorem 66 is for sums of independent point processes, it also applies to certain sums of conditionally independent point processes. We will consider the convergence of such sums and their application to partitioning and thinning of point processes, following a motivating example. Example 67. A Thinned Process. Let N be a point process on R+ (e.g., a d

renewal process) that satisfies t−1 N (t) → λ as t → ∞, where λ is a positive constant. Suppose each point of N is independently retained with probability p and deleted with probability 1 − p. Let Np (t) denote the number of retained points in (0, t]. When p is very small, the retained points are rare and so it appears that it would be appropriate to approximate the p-thinning Np of N by a Poisson process. Let us see why. As p → 0, clearly Np would converge to 0. However, the thinned process Np (p−1 t) with its time scale magnified by p−1 converges in distribution to a Poisson process with rate λ. This convergence is a special case of Corollary 70 below, which applies to general partitions of a point process. Based on this convergence it follows that it is reasonable to approximate Np by a Poisson process with rate pλ when p is small.  For the next result, suppose that Nn = j δXnj is a sequence of point processes on a space S with intensity measures μn . Let Mn be a marked pn transformation of Nn on S×S  . We will specify conditions for the convergence of Mn to a Poisson process, and then apply this to partitions of a point process.

3.17 Poisson Convergence Theorems*

223

We will use the conditional mean measure of Mn given Nn , which is  pn (Xnj , B)1(Xnj ∈ A) ηn (A × B) = E[Mn (A × B)|Nn ] = j



A ∈ S, B ∈ S  .

pn (x, B)Nn (dx),

= A

The convergence in distribution of these random mean measures ηn would be consistent with the convergence of Mn . For such random measures, the d convergence ηn → η is analogous to convergence in distribution of point processes; equivalent statements for this are in Theorem 62 (with η in place of N ). A natural prerequisite for Mn to converge is that the transformations should be uniformly null. Accordingly, we will use the condition lim sup pn (x, B) = 0,

n→∞ x∈A

ˆ B ∈ Sˆ . A ∈ S,

(3.50)

Theorem 68. (Poisson Limit of Rare Transformations) Suppose the sequence d

Mn of marked pn -transformations of Nn satisfies (3.50). Also, assume ηn → μ as n → ∞, where μ is a (non-random) locally finite measure on S × S  . d

Then Mn → M as n → ∞, where M is a Poisson process on S × S  with intensity measure μ.  Proof. We can write Mn = i Mni , where Mni = δXni ,Yni and Yni are pn marks of the Xni , for i ≥ 1. Although the point processes Mni , i ≥ 1, are not independent, they are conditionally independent given Nn . Clearly P {Mni (B) ≥ 2|Nn } = 0 and, under assumption (3.50), ˆ B ∈ Sˆ . A ∈ S,

sup P {Mni (A × B) ≥ 1|Nn } ≤ sup pn (x, B) → 0, i

x∈A

d Also, using ηn → μ, we have, for A × B ∈ S × S  μ,  P {Mni (A × B) ≥ 1|Nn } = E[Mn (A × B)|Nn ] i d

= ηn (A × B) → μ(A × B). Applying Theorem 66 to the conditional distribution of Mn given Nn , and using Theorem 62, it follows that E[e−Mn f |Nn ] → E[e−Mf ], d

+ f ∈ CK (S).

Taking expectations of this and using the dominated convergence theorem for convergence in distribution (Theorem 17 in the Appendix), we have d

E[e−Mn f ] → E[e−Mf ]. Thus, Mn → M by Theorem 62.

224

3 Poisson Processes

Example 69. Poisson Limits of Partitions. Let N (t) be a point process on R+ . Suppose N is partitioned as in Proposition 41 by the following rule: Each point of N is assigned to subprocess i ∈ I (a  countable set) with probability p(i), independent of everything else, where i∈I p(i) = 1. Then  N = i∈I Ni , where Ni denotes the ith subprocesses in the partition. We address the issue of finding conditions under which the subprocesses {Ni : i ∈ I0 }, for a subset I0 ⊆ I, are approximately independent Poisson processes. A natural condition for this is that the points of N would rarely be assigned to the subprocesses in I0 , but would mostly be assigned to the other subprocesses. The thinning in Example 67 is such a partition consisting of two subprocesses, where I0 = {0} and I = {0, 1}. To ensure that the subprocesses in I0 are sparse, we assume the partitioning probabilities are functions of n such that pn (i) → 0, for i ∈ I0 (I is necessarily infinite when I0 = I). Denote the ith subprocess by Nni (t). Its conditional mean given N is E[Nni (t)|N ] = pn (i)N (t). This mean converges to 0, which would not lead to a non-zero limit of Nni . To obtain a non-zero limit, a normalization of the processes Nni is in order. Accordingly, assume there is a positive constant λ such that t−1 N (t) → λ. d

(3.51)

This ensures that N (t) → ∞ and that the points of N appear at a positive rate out to infinity. Next, assume the partitioning is uniformly rare on I0 : there exist positive constants an → ∞ and r(i), such that lim an pn (i) = r(i),

n→∞

i ∈ I0 .

(3.52)

Under the preceding assumptions, it is natural to consider the convergence of the point process ˆni (t) = Nni (an t), N

i ∈ I0 .

This is a normalization of the partition-processes Nni under a rescaling of ˆni on a “fixed” interval (0, t] time so that an is the new unit of time. The N represents subprocess i on the interval (0, an t], which becomes larger as n → ∞. The choice of an for the time unit is because, as n → ∞, d

ˆni (t)|N ] = an pn (i)(N (an t)/an ) → r(i)λ, E[N

i ∈ I0 .

The following result describes the Poisson limits of the subprocesses. Interˆni for i ∈ I0 are dependent for each n but in the limit estingly, the processes N they are independent. This convergence theorem justifies that the partitionprocesses Nni , i ∈ I0 , for large n are approximately independent Poisson processes on R+ with respective rates an pn (i)λ ≈ r(i)λ, i ∈ I0 . Corollary 70. Under assumptions (3.51) and (3.52),

3.18 Exercises

225 d

ˆni : i ∈ I0 ) → (Ni : i ∈ I0 ), (N

as n → ∞,

(3.53)

where the limiting processes are independent homogeneous Poisson processes with respective rates r(i)λ, i ∈ I0 . Proof. The partition of N we are studying is a special pn -transformation of  the process N (an ·) with pn (t, B) = i∈B pn (i). Specifically, the number of the N (an t) points assigned to subprocess i ∈ I0 is ˆni (t) = Mn ((0, t] × {i}), N where Mn is a marked pn -transformation on R+ × I0 of N (an ·) as in Theorem 68. For each t ≥ 0 and B ⊆ I0 ,  pn (i) → 0. sup pn (s, B) = s≤t

i∈B

Furthermore, under assumptions (3.51) and (3.52), ηn ((0, t] × B) = E[Mn ((0, t] × B)|N (an ·)] = an d

→ η((0, t] × B) =





pn (i)(N (an t)/an )

i∈B

r(i)λt.

i∈B d

Thus, the assumptions of Theorem 68 are satisfied,  and so Mn → M , where M is a Poisson process with E[M ((0, t]×B)] = i∈B r(i)λt. Hence, assertion ˆni (t) = Mn ((0, t] × {i}). (3.53) follows since N

3.18 Exercises The first nine exercises concern properties of exponential random variables that arise naturally in modeling. Each of these exercises has an analogue for geometric distributions (the rate of an exponential distribution is analogous to the probability (or parameter) p in a geometric distribution p(1 − p)n−1 ). Exercise 1. Memoryless Property. Show that an exponential random variable X satisfies P {X > s + t|X > s} = P {X > t},

s, t > 0.

Explain why this is called a memoryless property. Show that a nonnegative continuous random variable has an exponential distribution if and only if it satisfies the memoryless property. Hint: Use the fact that a continuous nonincreasing function f : R+ → R+ satisfies f (s + t) = f (s)f (t), s, t ≥ 0, if and only if f has the form f (t) = e−ct for some c ≥ 0.

226

3 Poisson Processes

Exercise 2. Minima of Exponential Random Variables. Let X1 , . . . , Xm be independent exponential random variables with respective rates λ1 , . . . , λm , and define Y = min Xi , ν = argmin1≤i≤m Xi . 1≤i≤m

Show that Y has an exponential distribution with rate λ =

m i=1

λi , and

P {ν = i} = λi /λ. Show that P {X1 < X2 < · · · < Xm } =

m−1 

λ m k

k=1

j=k

λj

.

Exercise 3. Continuation. In the context of the preceding exercise, prove that ν and Y are independent, and that E[min{X1 , X2 }|X1 ≤ X2 ] = 1/(λ1 + λ2 ),  λj xj ). P {Xj − Y > xj , j = i | ν = i} = exp(− Show that P {ν ∈ I|ν ∈ J} =

 i∈I

λi /

 j∈J

j=i

λj , for I ⊆ J in {1, . . . , m}.

Exercise 4. Prove that a nonnegative distribution F is an exponential dis!t tribution if and only if F (t) = μ1 0 [1 − F (s)]ds for some μ > 0. In that case, μ is the mean of F . Exercise 5. Geometric Sum of Exponential Random Variables. Suppose X1 , X2 , . . . are independent exponentially distributed with rate λ, and ν is a random variable independent of the Xn with the geometric distribution P {N = n} = pn−1 (1 − p), n ≥ 1. Prove that P{

ν 

Xi > t} = e−(1−p)λt

i=1

by using Laplace transforms. Also, prove it by using the representation P{

ν 

Xi > t} = P {N (t) < ν},

i=1

and conditioning on N (t), where N (t) is a Poisson process with rate λ independent of ν. In addition, prove that P {ν > m,

ν−m  i=1

Xi > t} = pm e−(1−p)λt .

3.18 Exercises

227

Exercise 6. Let X1 , . . . , Xn , Y be independent exponentially distributed random variables with respective rates λ1 , . . . , λn , μ. Show that P{

n 

Xi < Y } =

i=1

n 

 λi = P {Xi < Y }. λ + μ i=1 i=1 i n

Show that if N (t) is a Poisson process with rate λ, and T is anindependent n exponential random variable with rate μ, then P {N (T ) > n} = λ/(λ+μ) . Exercise 7. Exponential Series. Let X1 , X2 , . . . be independent exponentially distributed random variables with respective rates λ1 , λ2 , . . . Show that ∞ 

Xn < ∞ a.s.

∞ 

⇐⇒

n=1

λ−1 n < ∞.

n=1

(This property is used to prove Proposition 5 in the next chapter.) Hint: Use Laplace transforms and the property of products that, for an ∈ (0, 1), ∞ 

an > 0

⇐⇒

n=1

∞ 

(1 − an ) < ∞.

n=1

Exercise 8. Show that if X is an exponential random variable with rate λ, then for any h : R+ → R,



X

E[h(X)] = λE

 h(u)du ,

0

provided the expectations exist. (This property is used to prove Theorem 52 in the next chapter.) Exercise 9. Suppose that Fi , i ∈ I, is a finite collection of exponential distributions with respective rates λi , i ∈ I, that are distinct. For J ⊆ I, let FJ denote the convolution of the distributions Fj , j ∈ J. Show that, for any real numbers ai , and subsets Ji of I, for j ∈ I,   ak FJk (x) = ci Fi (x), (3.54) k∈I

where ci = λi

 k∈I

i∈I

ak 1(i ∈ Jk )

 j∈Jk \{i}

λj . λj − λi

This assertion also holds when I is infinite, under the additional assumption that the summations exist. Hint: Use the fact that the Laplace transform of the left-hand side of (3.54) is

228

3 Poisson Processes

L(u) =



ak

k∈I

 j∈Jk

λj . λj + u

A standard partial sum expansion of this sum of products is L(u) =

 i∈I

ci

λi , λi + u

(3.55)

where ci = (λi + u)L(u)|u=−λi . Exercise 10. A space station requires the continual use of two systems whose lifetimes are independent exponentially distributed random variables X1 and X2 with respective rates λ1 and λ2 . When system 2 fails, it is replaced by a spare system whose lifetime X3 is exponentially distributed with rate λ3 , independent of the other systems. Find the distribution of the time Y = min{X1 , X2 + X3 } at which one of the systems becomes inoperative. Find the probability that system 1 will fail before system 2 (with its spare) fails. Exercise 11. Dispatching. In Example 3, what properties of the Poisson process are not needed to obtain the optimal dispatching policy? What is the optimal policy when the arrival process N (t) is a simple, stationary point process with N (t) = λt, such as a stationary renewal process? Exercise 12. Waiting to be Dispatched. Items arrive to a dispatching station according to a Poisson process with rate λ, and all items in the system will be dispatched at a time t. Example 3 shows ! t that the expected time items wait before being dispatched at time t is E[ 0 N (s)ds] = λ2 /2. Suppose there is a cost hw2 for holding an item  in the system for a time w. Then the total holding cost in (0, t] is C = n≥1 h(t − Tn )2 . Find E[C]. Exercise 13. Requests for a product arrive to a storage facility according to a Poisson process with rate λ per hour. Given that n requests are made in a t-hour time interval, find the probability that at least k requests were made in the first hour. Is this conditional probability different if the beginning of the one-hour period is chosen according to a probability density f (s) on the interval [0, t − 1]? Exercise 14. From Theorem 22, we know that the sum N = N1 +· · ·+Nn of independent Poisson processes is Poisson. Prove this statement by verifying that N satisfies the defining properties of a Poisson process. Exercise 15. Meeting of Vehicles. Vehicles enter a one-mile stretch of a twoway highway at both ends by independent Poisson processes and move at 60 miles per hour to the opposite end. Let λ and μ denote the rate of Poisson arrivals at the two ends labeled 0 and 1. Assuming the highway is empty at time 0, show that the probability is (λe−μ + μe−λ )/(λ + μ) that the first vehicle that enters at either end does not encounter another vehicle coming from the other direction during the one-mile stretch.

3.18 Exercises

229

Show that this probability is (λe−μ + μe−(λ+2μ) )/(λ + μ) for such an encounter avoidance for the first vehicle to arrive from end 0. Hint: For this second problem, let N1 (t) denote the Poisson process of arrivals at end 1, and let T denote the time of the first arrival at end 0. Then N1 (T ) denotes the number of arrivals at 1 before the first arrival at 0. Use its distribution from Exercise 6. Exercise 16. Show that a simple point process N on R+ is a Poisson process with rate λ if and only if N has independent increments, E[N (1)] = λ, and, for any t and n, the conditional joint density of T1 , . . . , Tn given N (t) = n is the same as the density of the order statistics of n independent uniformly distributed random variables on [0, t]. Exercise 17. Shot Noise Process. Suppose that shocks (or pulses) to a system occur at times that form a Poisson process with rate λ. The shock at time Tn has a magnitude Yn and this decays exponentially over time with rate γ. Then the cumulative effect of the shocks at time t is N (t)

Z(t) =



Yn e−γ(t−Tn) .

n=1

Assume Y1 , Y2 , . . . are i.i.d. independent of N with mean μ and variance σ 2 . Find expressions for the mean and variance of Z(t) in terms of μ and σ 2 . Exercise 18. Randomly Discounted Cash Flows. In the context of Example 13, consider the generalization N (t)

Z(t) =



Yn e−γn Tn ,

n=1

where γ1 , γ2 , . . . are independent nonnegative discount rates with distribution G that are independent of N and the Yn ’s. Show that  t E[Z(t)] = λE[Y1 ] e−γxG(dγ)dx. 0

R+

Exercise 19. Calls arrive to an operator at a call center at times that form a Poisson process N (t) with rate λ. The time τ devoted to a typical call has an exponential distribution with rate μ, and it is independent of N . Then N (τ ) is the number of calls that arrive while the operator is busy answering a call. Find the Laplace transform and variance of N (τ ). Find P {N (τ ) ≤ 1}. Exercise 20. For a Poisson process N with rate λ, show that E[T − Tk |N (t) = n] = ( − k)/(n + 1), Find an expression for E[t − Tk |N (t) = n].

k <  ≤ n, t > 0.

230

3 Poisson Processes

Exercise 21. Two types of items arrive at a station for processing according to independent Poisson processes with respective rates λ and λ . Let Tm and Tn denote the times of the mth and nth arrivals from the two respective processes. Show that P {Tm < Tn } = P {Y ≥ m}, where Y has a binomial distribution with parameters m + n − 1 and λ/(λ + λ ). Exercise 22. Consider a set of N jobs that are assigned to m workers for processing. Each job is randomly assigned to worker i with probability pi , for i = 1, . . . , m. Let Ni denote the number of items assigned to worker i, so that N = N1 + · · · + Nm . Suppose N has a Poisson distribution with mean λ. Describe the joint distribution of N1 , . . . , Nm . Exercise 23. Patients at an emergency room are categorized into m types. Assume the arrivals of the m types of patients occur at times that form independent homogeneous Poisson processes with respective rates λ1 , . . . ,λm . (a) Find the probability that a type 1 patient arrives before a type 2 patient. (b) What is the probability that the next patient to arrive after a specified time is of type 1? (c) Find the probability that in the next 5 arrivals, there are exactly 3 type 1 patients. (d) Find the probability that 3 type 1 patients arrive before the first type 2 patient. (e) Find the probability that the next patient to arrive is of type 1, 2 or 3. Exercise 24. Dynamic Servicing. Customers randomly request service at a manufacturing facility during an eight-hour day according to a Poisson process with intensity λt . The requested orders are satisfied as soon as possible, but may be delayed due to machine workloads, worker schedules, machine availability, etc. Past history shows that a request at time t will be satisfied either: (1) That day. (2) The next day. (3) Some time later. The request at time t is satisfied under scenario i with probability pi (t), and the expected revenue for such an order is ri , where i = 1, 2, 3. (a) Find the distribution of the number of requests in a day that are satisfied under each scenario i, where i = 1, 2, 3. (b) Find the daily expected revenue for satisfying the customers, and find the variance of this revenue. (This is an actual model of customer requests for orders of paper labels produced by a company.) Exercise 25. Requests for a product (information or service) arrive from m cities at times that form independent Poisson processes with rates λ1 , . . . , λm . Given that there are n requests from the cities in the time interval (0, t], find the conditional probability that n1 are from city 1 and n2 are from city 2.

3.18 Exercises

231

Exercise 26. At the end of a production shift, it is anticipated that there will be N jobs left to be processed, where N has a Poisson distribution with mean μ. Suppose the jobs are processed in parallel and the times to complete them are independent with a distribution G. Let Q(t) denote the number of jobs in the system at time t, and let D(t) denote the number of jobs completed in (0, t]. Find the distributions of Q(t) and D(t). Is D a Poisson process? Answer this question when the jobs are processed serially (one at a time) and G is an exponential distribution with rate λ. Exercise 27. Let X(1) ≤ · · · ≤ X(n) denote the order statistics from a random sample of size n from an exponential distribution with rate λ. Consider the distances between points D1 = X(1) , and Dk = X(k) − X(k−1) , 2 ≤ k ≤ n. Show that these distances are independent, and that Dk has an exponential distribution with rate (n − k + 1)λ. Exercise 28. Let X(1) ≤ · · · ≤ X(n) denote the order statistics of a random sample from a continuous distribution F (x) with density f (x). Show that the distribution and density of X(k) are P {X(k)

n

 n F (x)j (1 − F (x))n−j , ≤ x} = j j=k

fX(k) (x) =

n! F (x)k−1 f (x)(1 − F (x))n−k , (k − 1)!(n − k)!

x ∈ R.

Exercise 29. Point Locations for Non-Homogeneous Poisson Processes. Consider a Poisson process N on R+ with rate function r(x), where r(x) > 0 for each x. Prove the following Order Statistic Property: The conditional density of T1 , . . . , Tn given N (t) = n is     fT1 ,...,Tn t1 , t2 , . . . , tn N (t) = n = n!f (t1 ) · · · f (tn ), for 0 < t1 < · · · < tn < t, where f (s) = r(s)/μ(0, t]. This is the joint density of the order statistics of n i.i.d. random variables on [0, t] with density f (s). Exercise 30. Consider a Poisson process N on R+ with point locations T1 < T2 < . . . and rate function r(t) = 3t2 . (a) Show that Wn = Tn − Tn−1 , n ≥ 1, are dependent. (b) Find the distributions of T1 and Tn . (c) Find the distribution of Wn . Exercise 31. Suppose N is a Poisson process on R2 with rate function λ(x, y) at the location (x, y). For instance, N could represent locations of certain types of animal nests, diseased trees, land mines, auto accidents, houses of certain types of people, flaws on a surface, potholes, etc. Let Dn denote the

232

3 Poisson Processes

distance from the origin to the n-th nearest point of N . (a) Find an expression for the distribution and mean of D1 . (b) Find an expression for the distribution of Dn when λ(x, y) = λ. (c) Are the differences Dn − Dn−1 independent (as they are for Poisson interpoint distances on R)? (d) Suppose there is a point located at (x∗ , y ∗ ). What is the distribution of the distance to the nearest point? (e) Is λ(x, y) = 1/(x2 + y 2 )1/2 a valid rate function for N to be a Poisson process under our definition? (f) Specify a rate function λ(x, y) under which P {N (R2 ) < ∞} = 1. Exercise 32. Let M denote a Poisson process on Rd+ with intensity μ. Show that N (t) = M ((0, t]d ), for t ≥ 0, is a Poisson process on R+ with E[N (t)] = μ((0, t]d ). This fact for d = 2 provides an alternate approach to proving the departure process in an Mt /Gt /∞ system is Poisson; see Section 3.12. Exercise 33. Highway Model. Vehicles enter an infinite highway denoted by R at times that form a Poisson process N on the time axis R+ with intensity measure μ. For simplicity, assume the highway is empty at time 0. The vehicle arriving at time Tn enters at a location Xn on the highway R and moves on it with a velocity Vn for a time τn and then exits the highway. The velocity may be negative, denoting a movement in the negative direction and vehicles may automatically pass one another on the highway with no change in velocity. The Xn are i.i.d. with distribution F and are independent of N . The pairs (Vn , τn ) are independent of N and, they are conditionally independent given the Xn with Gx (v, t) = P {Vn ≤ v, τn ≤ t|Xk , k ≥ 1, Xn = x}, a non-random distribution independent of n.  (a) Justify that M = n δ(Tn ,Xn ,Vn ,τn ) , is a Poisson process on R+ × R2 × R+ and describe its intensity. (b) Consider the departure process D on R × R+ where D(A × (a, b]) is the number of departures from A in the time interval (a, b]. Justify that D is a Poisson process and specify its intensity. Find the expected number of departures in (0, t]. (c) For a fixed t, let Nt (A) denote the number of vehicles in A ⊆ R at time t. Justify that Nt is a Poisson process on R and specify its intensity. (d) Suppose a vehicle is at the location x ∈ R at time t and let X(t) denote the distance to the nearest vehicle. Specify assumptions on μ, F and Gx (v, t) that would guarantee that there is at most one point at any location on the highway. Under these assumptions, find the distribution of X(t). Exercise 34. Continuation. In the preceding highway model, assume there are vehicles on the highway at time 0 at locations that form a Poisson process N0 (·) with rate λ, and this process is independent of the other vehicles.

3.18 Exercises

233

The sojourn times of these vehicles on the highway are like those of the other vehicles vehicles, and they all operate under the distribution Gx (v, t) = G(v)(1 − e−μt ). Solve parts (b)–(d) of the preceding exercise. Exercise 35. In the context of Example 34, suppose N is a homogeneous Poisson process on the unit disc S in R2 with rate λ. For the Poisson processes N  and M related to the projection of N on the line S  = [−1, 1, ], show that √  2 the rate function of N is 2λ 1 − x , and that  E[M (Au × (0, b])] = λ

b 0

$ ( 1 − x2 − u) dx,

√ for Au = {(x, y) ∈ S : y ≥ u} and b ≤ 1 − u2 . Next, consider the transformation of N where a point in the unit disc S is mapped to the closest point  on the unit circle C. Under this map, using polar coordinates, M (A×B) = n δ(Rn ,Θn ) (A)δΘn (B) denotes the number of points of N in A that are mapped into B. Justify that M is a Poisson process on S × C, and give an expression for E[M (A× (0, b])], where A = {(r, θ) ∈ S : 1 θ ∈ (0, b], r ∈ [ sin θ+cos θ , 1]} and b ≤ π/2 (note that x+y = r(sin θ+cos θ) ≥ 1 when (x, y) ∈ A). For a fixed B, consider the process N (r) = M (Ar × B), r ∈ [0, 1], where Ar is a disc in R2 with radius r. Show that N is a Poisson process and specify its rate function λ(r). Exercise 36. Suppose N1 , . . . , Nm are independent Poisson processes on a m space S with respective intensities μ1 , . . . , μm , and let N = i=1 Ni , which is a Poisson process with intensity μ = μ1 + · · · μm . For instance, Ni (B) might be the number of crimes of type i in a region B of a city and N (B) is the total number of crimes. Show that the conditional distribution of N1 (B), . . . , Nm (B) given N (B) = n is a multinomial distribution. Exercise 37. Let N1 , . . . , Nm be independent Poisson processes on R+ with location-dependent rates λ1 (t), . . . , λm (t), respectively. Let τi denote the time of the first occurrence in process Ni . Find the distribution of τ = min1≤i≤m τi . Find P {τi = τ }. Exercise 38. Messages arrive to a web page at times that form a Poisson process with rate λ. The messages are independently of high priority with probability p, and each of these high priority messages is independently of type i with probability pi , ≤ i ≤ m. Let Ni (t) denote the number of high priority messages of type i that arrive in [0, t]. Are N1 , . . . , Nm independent Poisson processes? If so, specify their rates. Exercise 39. Let X1 , X2 , . . . be i.i.d. random variables with a distribution F . Show that X1 , . . . , Xn are distinct with probability one for any n ≥ 2 if and only if F is continuous. (This statement is also true if these are random elements in a space S and “continuous F ”! is replaced by F ({x}) = 0, x ∈ S). Hint: Use induction and P {X1 = X2 } = R (1 − F ({x})F (dx).

234

3 Poisson Processes

Exercise 40. Poisson Processes are Infinitely Divisible. Let N be a Poisson process on S with intensity μ. Using Laplace functionals, show that for each d n, there exist point processes N1 , . . . , Nn that are i.i.d. and N = N1 +· · ·+Nn . This says that N is infinitely divisible.  Exercise 41. As in Example 34, suppose N = n δ(Xn ,Yn ) is a Poisson 2 process on the unit disc S in R with location-dependent rate λ(x, y). The  Poisson process M = n δ((Xn ,Yn ),Xn ) on S × S  represents the number of points of N that are projected onto the x-axis S  = [−1, 1]. Give an expression for the location-dependent intensity of M . Switching to polar coordinates as  in Example 34, describe the process M = n δ((Rn ,Θn ),Θn ) that represents the number of points of N that are mapped onto the unit circle C. Exercise 42. Deposits to a bank account occur at times that form a Poisson process with rate λ and the amounts deposited are independent random variables with distribution F (independent of the times). Also, withdrawals occur at times that form a Poisson process with rate μ and the amounts deposited are independent random variables with distribution G (independent of the times). The deposits and withdrawals are independent. Let X(t) denote the balance of the bank account at time t; the balance may be negative. Find the mean, variance and distribution of X(t) when X(0) = 0.  Exercise 43. Suppose X(t) = minn≤N (t) Yn , for t ≥ 0, where N = n δTn is a Poisson process on R+ and Yn are i.i.d. with distribution F , independent of N . For instance, Yn could be bids on a property and X(t) is the smallest bid up to time t. Find the distribution and mean of X(t). Answer this question for the more general setting in which Yn are p-marks of Tn , where p(t, (0, y]) is the distribution of a typical mark at time t. Exercise 44. E-mail Broadcasting. An official of an organization sends email messages to various subgroups of the organization at times that form a Poisson process with intensity μ. Each message is sent to individual i with probability pi , i = 1, . . . , m, where m is the number of individuals in the organization; and the message is sent at the same time to those selected individuals. Let Ni (t) denote the number of messages individual i receives from the official in the time interval (0, t]. Justify that Ni is a Poisson process and specify its intensity. Consider a subset of individuals I ⊆ {1, . . . , m}.  Specify whether or not the processes Ni , i ∈ I, are independent. Is the sum i∈I Ni Poisson? If so, specify its intensity. Exercise 45. Continuation. In the setting of the preceding exercise, suppose there is a probability ri that individual i will reply to an e-mail from the official, and then send the reply within a time that has a distribution Gi (t). Let Ri (t) denote the number of replies the official receives in (0, t] from all

3.18 Exercises

235

the messages  sent to individual i. Is Ri Poisson? If so, specify its intensity. Is the sum i∈I Ri Poisson? If so, specify its intensity. Suppose individual i receives a message from the official, and is planning to reply to it. Find the probability that before the reply is sent, another message from the official will arrive? Exercise 46. A satellite circles a body in outer space and records a special feature of the body (e.g. rocks, water, low elevations) along a path it monitors. As an idealized model, assume the feature occurs on the polar-angle space S = [0, 2π] at angles Θ1 ≤ Θ2 . . . ≤ 2π that form a Poisson process with intensity μ. We will only consider one orbit of the satellite. Suppose the satellite is moving at a (deterministic) velocity of γ radians per unit time. Upon observing an occurrence at Θn the satellite sends a message to a station that receives it after a time τn . Suppose the transmission times τn are independent with distribution G and are independent of the positions of the occurrences. Consider the point process M on S × R2+ , where M ((α, β] × (a, b] × (c, d]) is the number of occurrences in the radian set (α, β] that are observed in the time set (a, b] and received at the station in the time set (c, d]. Describe the process M and its intensity measure in terms of the system data. Next, let N (t) denote the number of messages received at the station in (0, t] whose transmission time exceeds a certain limit L, where G(L) < 1. Describe the process N and specify its intensity measure. Exercise 47. Multiclass M/G/∞ System. Consider an M/G/∞ system in which items arrive at times that form a Poisson process with rate λ. There are m classes or types of items and pi is the probability that an item is of class i. The processing time of a class i item has a distribution Gi (·). Assume the system is empty at time 0. Let Qi (t) denote the quantity of class i items in the system at time t. Specify its distribution. Determine whether or not Q1 (t), . . . , Qm (t) are independent. Let Di (t) denote the number of departures of class i items in (0, t]. Describe these processes including their independence. Exercise 48. Limiting Behavior of M/G/∞ System. Consider the M/G/∞ system in Section 3.12 with arrival rate λ and service distribution G, which has a mean α. Show that the limiting distribution of the quantity of items in the ! ∞system Q(t) is Poisson with mean λα as t → ∞. Use the fact that α = 0 [1 − G(u)]du. Turning to the departures, consider the point process Dt (B) on R+ that records the numbers of departures in a time set B after time t. In particular, show that the number of departures Dt (0, b] in the interval (t, t + b] has a Poisson distribution with

 E[Dt (0, b]] = λ t



t+b

G(u + b)du −

t+b

 G(u)du .

t

Show that the limiting distribution of Dt (0, b] is Poisson with mean λb. More generally, show that the finite-dimensional distributions of Dt converge to

236

3 Poisson Processes

those of a homogeneous Poisson process D with rate λ. This proves, in light d of Theorem 62, that Dt → D. Exercise 49. Spatial M/G/∞ System. Consider a system in which items enter a space S at times T1 ≤ T2 ≤ . . . that form a Poisson process with intensity measure μ. The nth item that arrives at time Tn enters S at the location Xn and remains there for a time Vn and then exits the system. Suppose Ft (·) is the distribution of the location in S of an item arriving at time t, and G(t,x) (·) is the distribution of the item’s sojourn time at a location x. More precisely, assume (Xn , Vn ) are location-dependent marks of Tn with distribution  p(t, A × (0, v]) = G(t,x) (v)Ft (dx). A

Let Nt (B) denote the number of items in the set B ∈ S at a fixed time t. Show that Nt is a Poisson process on S with   [1 − G(s,x) (t − s)]Fs (dx)μ(ds). E[Nt (B)] = (0,t]

B

Next, let D((a, b] × B) denote the number of departures from the set B in the time interval (a, b]. Show that D is a space-time Poisson process on R+ × S and specify E[D((0, t] × B)]. Exercise 50. For the network in Example 52, justify that the following processes are Poisson and specify their intensity measures. D(t) = # of items that depart from the network in (0, t]. D1 (t) = # of items that enter node 1 and depart from the network in (0, t]. D(2,3,5) (t) = # of items that complete the route (2, 3, 5) in (0, t]. Justify that the following random variables, for a fixed t, have a Poisson distribution and specify their means. Q(t)= # of items that are in the network at time t. Q1 (t)= # of items that are beyond their first node at time t. Q3|2 (t)= # of items in node 3 at time t that came from node 2. Exercise51. Time Transformations and Cox Processes. Suppose that N1 (t) = n 1(Tn ≤ t) is a homogeneous Poisson process on R+ with rate λ, and let η denote a locally finite measure on R+ with η(R+ ) = ∞. Consider the process N (t) = N1 (η(t)), t ≥ 0. Show that N is a Poisson process on R+ with intensity η. Do this by showing that N is a transformation of N1 under a map g : R+ → R+ ; that is, find a g such that

3.18 Exercises

237

N (t) =



1(Tn ≤ η(t)) =

n



1(g(Tn ) ≤ t).

n

This fact implies that if η is a locally finite random measure on R+ with η(R+ ) = ∞ a.s., then N is a Cox process. Exercise 52. Suppose that N is a Cox process on S directed by a locallyfinite random measure η. Show that E[N (B)] = Var[N (B)], for B ∈ S. Exercise 53. Poisson Process Directed by a Cyclic Renewal Process. The state of a system is represented by a continuous-time cyclic renewal process X(t) on states 0, 1, . . . , K −1 as in Example 8. The sojourn times in the states are independent, and the sojourn time in state i has a continuous distribution K−1 Fi with mean μi . By Exercise 47, limt→∞ P {X(t) = i} = μi / k=0 μk . Suppose the system fails occasionally such that, while it is in state i, failures occur according to a Poisson process with rate λi , independent of everything else. Let N (t) denote the number of failures in (0, t]. Show that K−1 k=0 λk μk t−1 N (t) →  , K−1 k=0 μk

a.s. as t → ∞.

Assume the system begins in state 0 and let τ denote the first time it returns to state 0 (the time to complete a cycle). Show that E[N (τ )] =

K−1 

λk μk = Var[N (τ )].

k=0

Exercise 54. Location-Dependent Compound Poisson Process. Suppose that N (t) Z(t) =  n=0 Yn is a location-dependent compound Poisson process, where N = n δTn is a Poisson process on R+ with intensity measure μ, and Yn are p-marks of Tn . Show that the process Z(t) has independent increments (the increments will not be stationary in general), and   yp(s, dy)μ(ds). E[Z(t)] = (0,t]

R

! Suppose the moment generating function φs,t (α) = R eαy Fs,t (dy) exists, where  p(u, (0, y])μ(du)/μ(s, t]. Fs,t (y) = (s,t]

Show that, for s < t, E[eα[Z(t)−Z(s)] ] = e−μ(s,t][1−φs,t (α)] .

(3.56)

(This is the moment generating function of a compound Poisson distribution with rate μ(s, t] and distribution Fs,t .) Use the fact

238

3 Poisson Processes

E[eα[Z(t)−Z(s)] ] = E[e

! R

yM((s,t]×dy)

where hs,t (u, y) = −αy1(u ∈ (s, t]) and M =



] = E[e−Mhs,t ],

n δ(Tn ,Yn ) .

Exercise 55. Suppose Z1 (t), . . . , Zm (t), are independent compound Poisson processes with respective rates λ1 , . . . , λm and distributions F1 , . . . , Fm m . nShow that Z(t) = i=1 Zi (t) is acompound Poisson process with rate λ = i=1 λi n and distribution F = λ−1 i=1 λi Fi . Exercise 56. Partition of a Compound Poisson Process. Suppose Z(t) = N (t) n=1 Yn , for t ≥ 0, is a location-dependent compound Poisson process with intensity measure μ and distribution p(t, ·). Suppose the quantity Y n at time   , . . . , Ynm ) so that Yn = m Y . Tn is partitioned into m pieces Yn = (Yn1 N (t) i=1 ni These pieces are assigned to m processes  defined by Zi (t) = n=1 Yni . They m form a partition of Z(t) in that Z(t) = i=1 Zi (t). Assume the Yn are p marks of (Tn , Yn ), where p ((t, y), B1 ×· · ·×Bm ) is the conditional distribution of a typical vector Yn given (Tn , Yn ) = (t, y). Prove that Zi (t) is a compound Poisson process with intensity μ and distribution p (t, ·), and specify p (t, ·). Use the idea that the Yn are second marks  of the Poisson process N as discussed in Example 39, resulting in M  = n δ(Tn ,Yn ,Yn ) . Exercise 57. Origin-Dependent Cluster Processes. The cluster process in    N (B)δ Example 58 has the form M (A × B) = Xn (A), where Nn are n n  point processes on a space S generated by the points Xn in S. Instead of assuming the Nn are independent of N , consider the more general setting in which the Nn are p-marks of Xn . Let Nx be a point process on S  such that p(x, C) = P {Nx ∈ C}. Show (by conditioning on N ) that the Laplace functional of M is  E[e−Mf ] = exp[− (1 − g(x))μ(dx)], S

where g(x) = E[e−

!

S

f (x,x



)Nx (dx )

].

Exercise 58. The moments of a point process N on S are given by E[N (A1 )n1 · · · N (Ak )nk ] ∂ n1 +···+nk E[e−N f ]|t1 =...=tk =0 , = (−1)n1 +···+nk n1 ∂t1 · · · ∂tnk k k where f (x) = i=1 ti 1(x ∈ Ai ). Prove this for k = 1 and k = 2. Use this fact to find expressions for the first two moments of the cluster process quantity M (A × B) in Exercise 57. Exercise 59. Sums of Identically Distributed Renewal Processes. Suppose ˜2 . . . are independent renewal processes with inter-renewal dis˜1 , N that N tribution F . By the strong law of large numbers, we know that the sum

3.18 Exercises

239

n

˜i (t) converges to ∞ a.s. as n → ∞. (The discussion prior to ExamN ple 65 addressed the opposite case where the sum tends to 0.) To normalize this sum (analogously to that in a central limit theorem) so that it converges to a non-degenerate  limit, it is natural to rescale the time axis and consider ˜i (t/n). This is the sum with 1/n as the new unit the process Nn (t) = ni=1 N of time. Assume the derivative λ = F  (0) exists and is positive. Show that i=1

d

Nn → N , where N is a Poisson process with rate λ. Exercise 60. Poisson Limit of Thinned Processes. Let Nn be a sequence of point processes on S. Suppose Nn is subject to a pn (x) thinning: A point of Nn at x is retained with probability pn (x) and is deleted with probability 1 − pn (x). Let Nn denote the resulting thinned process on S. Assume the thinning is uniformly null in that lim sup pn (x) = 0,

n→∞ x d

ˆ B ∈ S.

Show that Nn → N  , a Poisson process on S with intensity measure μ, if  d pn (x)Nn (dx) → μ(B), B ∈ Sˆμ , as n → ∞. B

Chapter 4

Continuous-Time Markov Chains

A continuous-time Markov chain (CTMC) is a discrete-time Markov chain with the modification that, instead of spending one time unit in a state, it remains in a state for an exponentially distributed time whose rate depends on the state. The methodology of CTMCs is based on properties of renewal and Poisson processes as well as discrete-time chains. CTMCs are natural candidates for modeling systems in real time such as production and inventory systems, computer and telecommunications networks, and miscellaneous input-output systems. Many continuous-time processes have discretetime analogues; for instance, birth-death and Brownian motion processes are continuous-time analogues of discrete-time random walks. One’s choice of a continuous- or discrete-time model for a system typically depends on how realistic it is, its ease in addressing the issues at hand, or in computing quantities of interest. A CTMC is a continuous-time Markov process on a countable state space whose sample paths are right-continuous and piecewise-constant with finite lengths, and the number of jumps in any finite time is finite. This type of Markov process is represented by the sequence of states it visits and the sojourn times at the visits. Our study of these processes begins with Kolmogorov’s backward and forward differential equations for the transition probabilities of a CTMC, and a characterization of its infinitesimal transition rates. Next is a description of properties of sample paths of CTMCs. Included is a “uniformization” property that a CTMC with bounded transition rates can be represented as a Markov chain that takes jumps at times that form a Poisson process. Several sections are devoted to describing the equilibrium behavior of CTMCs, including ergodic theorems for functions of CTMCs and L´evy-type expressions for expectations. Then we give detailed descriptions of reversible CTMCs, Jackson network processes, and multiclass networks. Next, we show how Palm Probabilities are used to describe a CTMC conditioned on the occurrence of a certain type of transition. For instance, in a production system with queueing, one may be interested in the number of R. Serfozo, Basics of Applied Stochastic Processes, Probability and its Applications. c Springer-Verlag Berlin Heidelberg 2009 

241

242

4 Continuous-Time Markov Chains

items in the system conditioned that an arrival occurs. In this and analogous situations, we present PASTA properties that “Poisson arrivals see time averages”, or “Palm actions see time averages”. The chapter ends with a description of M/G/1 and G/M/1 queueing processes, and an introduction to Markov renewal processes, which are relatives of a CTMC.

4.1 Introduction In this section, we introduce CTMCs and describe some of their features. A continuous-time stochastic process {X(t) : t ≥ 0} on a countable state space S is a Markov process if it satisfies the Markov property: for each i, j ∈ S and t, u ≥ 0, P {X(t + u) = j|X(t) = i, X(s), s < t} = P {X(u) = j|X(0) = i}. This is a time-homogeneous process because the last probability does not depend on t. Our interest is in such processes with nicely behaved sample paths as follows. Definition 1. A Markov process {X(t) : t ≥ 0} on a countable state space S is a continuous-time Markov chain (CTMC) if its sample paths are rightcontinuous and piecewise constant with finite lengths a.s., and the number of transitions in any finite time interval is finite a.s. That is, X(t) = Xn

if t ∈ [Tn , Tn+1 ) for some n,

(4.1)

where 0 = T0 < T1 < T2 < · · · are the jump times of the process, and Xn is the state visited at time Tn . The Tn → ∞ a.s. and Xn = Xn+1 , for each n. The sojourn time of the process in state Xn is χn = Tn+1 − Tn .1 Any process X(t) of the form (4.1) is called a jump process on S with embedded process (Xn , ξn ). Our first observation is a characterization of a CTMC in terms of elementary properties of its embedded process. Theorem 2. A jump process X(t) on S with embedded process (Xn , ξn ) is a CTMC if and only if (i) Xn is a discrete-time Markov chain on S with transition probabilities P = {pij }, where pii = 0, for each i. (ii) For nonnegative t0 , . . . , tm , 1

This differs from the renewal process notation, where ξn = Tn − Tn−1 is the nth interrenewal time.

4.1 Introduction

P {ξ0 ≤ t0 , . . . , ξm ≤ tm |Xn , n ≥ 0} =

243 m 

P {ξn ≤ tn |Xn },

(4.2)

n=0

and there are positive qi , i ∈ S, such that, for each n ≥ 0, P {ξn ≤ t|Xn = i} = 1 − e−qi t ,

t ≥ 0, i ∈ S.

(4.3)

Proof. This follows by Propositions 12 and 13 in Section 4.3. In this theorem, the embedded Markov chain Xn may be transient, irreducible, ergodic, etc. Subsequent sections show how X(t) inherits these properties of Xn . Note that the process X(t) does not have absorbing states, since its sojourn times in the states it visits are finite. For particular applications, one can describe Markov processes with the jump-like behavior described above, but with absorbing states, by the techniques in Chapter 1 and this chapter. For instance, one can determine probabilities of absorbtion or times to absorbtion by the results in Section 1.7 in Chapter 1 and in Section 4.6 below. Our study will not cover such Markov processes. Similar statements apply to Markov processes that may have an infinite number of jumps in a finite time, resulting in finite lifetimes. Condition (4.2) says that the sojourn times ξn are conditionally independent given the Xn , and condition (4.3) says that such a sojourn time in a state i is exponentially distributed with rate qi . Another way of stating Theorem 2 is as follows. This is a consequence of the Markov property of discrete-time Markov chains on general state spaces. Remark 3. A jump process X(t) with embedded process (Xn , ξn ) is a CTMC if and only if (Xn , ξn ) is a discrete-time Markov chain on2 S × R+ with transition probabilities P {Xn+1 = j, ξn+1 ≤ t|Xn = i, ξn } = pij (1 − e−qi t ),

i, j ∈ S, t ≥ 0,

where qi are positive constants and P = {pij } is a Markov transition matrix with each pii = 0. Hereafter, we will adopt the notation of Theorem 2 for describing CTMCs. In particular, we say that the defining parameters of the CTMC X(t) are (αi , pij , qi ), where αi = P {X(0) = i} is the initial distribution. We refer to pij and qi as the main defining parameters. Theorem 22 below shows that there exists a CTMC X(t) for any set of defining parameters (αi , pij , qi ). These parameters uniquely determine the distribution of a CTMC, and vice versa, as follows. Proposition 4. Two CTMCs have the same distribution if and only if their defining parameters are equal. 2

This state space is not countable.

244

4 Continuous-Time Markov Chains

Proof. It suffices to show that the following statements are equivalent. (a) The two CTMCs have the same distribution. (b) The embedded Markov chains for the CTMCs have the same distribution. (c) The defining parameters for the CTMCs are equal. Clearly (a) and (b) are equivalent by the construction of a CTMC. Also, (b) and (c) are equivalent by the property from Chapter 1 that two Markov chains have the same distribution if and only if their initial distributions and transition probabilities are equal. By its definition, a CTMC satisfies the regularity condition that Tn → ∞ a.s., which ensures that the CTMC is defined on the “entire” time axis. This condition has the following characterization in terms of the defining parameters of a CTMC. Proposition 5. Suppose that conditions (i) and (ii) in Theorem 2 hold. Then Tn → ∞ a.s. if and only if the qi are P-regular in the sense that ∞ 

−1 qX =∞ n

a.s.

(4.4)

n=0

In particular, Tn → ∞ a.s. if supi∈S qi < ∞ (which is true for finite S), or if the Markov chain Xn is recurrent. ∞ Proof. Clearly Tn → ∞ a.s. means Z = n=0 ξn = ∞ a.s. Also, by condition −1 , we have (ii) and E[ξn |Xn ] = qX n E[Z|Xn , n ≥ 0] =

∞ 

E[ξn |Xn ] =

n=0

∞ 

−1 qX . n

n=0

In light of these observations, the assertion to be proved is that E[Z|Xn , n ≥ 0] = ∞ a.s.

⇐⇒

P {Z = ∞} = 1.

(4.5)

Now, Exercise 7 in Chapter 3 shows that an infinite sum of independent exponential random variables is infinite a.s. if and only if the sum of their means is infinite. Applying this to the exponential sojourn times conditioned on the Xn , we have P {Z = ∞|Xn , n ≥ 0} = 1 a.s.

⇐⇒

E[Z|Xn , n ≥ 0] = ∞ a.s.

On the other hand, using P {Z = ∞} = E[P {Z = ∞|Xn , n ≥ 0}], we have P {Z = ∞|Xn , n ≥ 0} = 1 a.s.

⇐⇒

P {Z = ∞} = 1.

These two equivalences prove (4.5). −1 Next, if b = supi∈S qi < ∞, then qX ≥ b−1 , which clearly implies (4.4). n Finally, if Xn is recurrent, it visits some state i infinitely often, so qi−1 appears an infinite number of times in the sum in (4.4), and hence that sum is infinite.

4.2 Examples

245

4.2 Examples To justify that a jump process is a CTMC, one typically verifies the conditions in Theorem 2 that the sequence of states it visits is a discrete-time Markov chain, and that its sojourn times are exponentially distributed; one must also verify the P-regularity property. Here are some standard examples. Example 6. Poisson Process. A Poisson process N (t) with rate λ is a jump process whose state increases by unit jumps, and its sojourn times are independent and exponentially distributed with rate λ. Therefore, N (t) is a CTMC with one-step transition probabilities pi,i+1 = 1, and exponential sojourn rates qi = λ, which are obviously P-regular. Example 7. Pure Birth Process. Suppose that X(t) represents the number of occurrences of a certain event in the time interval [0, t]. Assume the interoccurrence times are independent, and the time between the ith and i + 1st occurrence is exponentially distributed with rate qi . Therefore, the state increases by unit jumps, and the sojourn ∞ times are exponentially distributed. The qi are P-regular if and only if i=0 qi−1 = ∞. So assuming this is true, X(t) is a CTMC on Z+ with parameters pi,i+1 = 1 and qi . For instance, the X(t) might denote the number of times a system is repaired. In this case, the qi would typically be increasing for a machine that wears out, but it would be decreasing for a software package that is perfected as flaws are fixed. This process is called a pure birth process because of the classical model in which X(t) is the size of a population, and whenever the population size is i, the time to the next birth is exponentially distributed with rate qi . ∞ Note that if i=0 qi−1 < ∞, the jump process X(t) could still be defined, but only on the time interval [0, supn Tn ), which is finite with a positive probability. There would be an infinite number of births in this time interval. Since a CTMC is essentially a Markov chain whose unit sojourn times in the states are replaced by exponential times, any discrete-time Markov chain has an analogous CTMC version. For instance, here is a continuous-time analogue of Exercise 49 in Chapter 1. Example 8. Exhaustive Parallel Processing. A set of m jobs are processed in parallel until they are all completed. At the completion time, another m jobs instantaneously enter the system and are processed similarly. This is repeated indefinitely. Assume the times to process jobs are independent exponentially distributed with rate λ. Let X(t) denote the number of jobs being processed at time t. This is a jump process on S = {1, . . . , m} and its embedded process (Xn , ξn ) satisfies, for 2 ≤ i ≤ m, P {Xn+1 = i − 1, ξn+1 > t|(Xm , ξm ), m ≤ n, Xn = i} = P {min{Y1 , . . . , Yi } > t} = e−iλt ,

246

4 Continuous-Time Markov Chains

where Y1 , . . . , Yi are independent exponentially distributed with rate λ. Here we use the property that the minimum of independent exponential random variables is again exponential with rate that is the sum of the rates of the variables. The other transition probability is P {Xn+1 = m, ξn+1 > t|(Xm , ξm ), m ≤ n, Xn = 1} = P {Y1 > t} = e−λt . Then by Remark 3, X(t) is a CTMC with parameters pi,i−1 = 1,

2 ≤ i ≤ m,

p1,m = 1,

and qi = iλ. The P-regularity is due to the finite state space. The following example describes a general framework for formulating a CTMC by clock times. Example 9. Clock Times and Transition Rates. Suppose X(t) is a jump process on S with embedded process (Xn , ξn ) whose dynamics are as follows. Whenever the process enters a state i, a set of independent clock times τij , j ∈ Si are started, where Si is the subset of states in S\{i} that can be reached from state i in one step. The times τij are exponentially distributed with rates qij . Then the sojourn time in state i is the minimum τi = minj∈Si τij , and at the end of the sojourn, the process jumps to the state j ∈ Si for which τij = τi . Think of τij as the time to the next “potential” transition from i to j ∈ Si with transition rate qij , and the clock time j that is the smallest of these times “triggers” a transition from i to j. Under these assumptions, P {Xn+1 = j, ξn+1 > t|(Xm , ξm ), m ≤ n, Xn = i} = P {τij = τi , τi > t}. By the properties of exponential random variables in Exercise 2of Chapter 3, the sojourn time τi is exponentially distributed with rate qi = j∈Si qij , and it is independent of the event τij = τi , where P {τij = τi } = qij /qi . Therefore, (Xn , ξn ) is a Markov chain with transition probabilities P {X1 = j, ξ1 > t|X0 = i, ξ0 } =

qij −qi t e . qi

 Then X(t) is a CTMC with parameters pij = qij /qi and qi = j∈Si qij , provided the qi are P-regular. More insights on such transition rates are in Section 4.3. See Exercise 4 for analogous discrete-time geometric clock times, and see Exercise 5 for clocks associated with multiple sources that trigger transitions. Example 10. Birth-Death Process. Suppose that X(t) represents the number of discrete items in a population at time t, where births and deaths (or

4.3 Markov Properties

247

additions and departures) in the population occur as follows. Whenever the population state is 0, the time to the next birth is exponentially distributed with rate λ0 . Also, whenever there are i ≥ 1 items in the population, the time to the next (potential) birth is exponentially distributed with rate λi , and the time to the next (potential) death is exponentially distributed with rate μi . These times are independent and independent of the rest of the process. Assume the birth and death rates are bounded. Under these conditions, it follows as in the preceding example that X(t) is a CTMC on Z+ with transition rates qi,i+1 = λi and qi,i−1 = μi . The X(t) is called a birth-death process with birth rates λi and death rates μi . Its exponential sojourn rates are qi = λi + μi , where μ0 = 0, and its one-step transition probabilities are pi,i+1 = λi /(λi + μi ),

pi,i−1 = μi /(λi + μi ).

There are a variety of queueing processes and general input-output processes that are birth-death processes. Here is a classic example. Example 11. M/M/s Queueing Process. Consider a processing system in which items arrive according to a Poisson process with rate λ. There are s servers who process the items one at a time, where 1 ≤ s ≤ ∞. The processing times are independent exponential random variables with rate μ, independent of everything else. An item that arrives when all the s servers are busy waits in a queue for processing; otherwise it goes to any available server for processing. Whenever there are i items in the system, min{i, s} items are being processed independently at rate μ, and so the time for a potential departure (considered as a death) has an exponential distribution with rate μi = μ min{i, s}. This is μi = iμ when s = ∞. Let X(t) denote the number of items in the system at time t. Clearly, X(t) is a birth-death CTMC with birth rate λ in each state, and death rate μi = μ min{i, s} in state i ≥ 1. This process is called an M/M/s queueing process. Because X(t) counts the quantity of items, which are indistinguishable, several service disciplines are possible (e.g., first-come-first-served, service in random or arbitrary order, or last-come-last-served).

4.3 Markov Properties This section proves Theorem 2 that characterizes a CTMC by its embedded process. Included are an integral equation and the Chapman-Kolmogorov equations for its transition probabilities. For this discussion, X(t) will denote a continuous-time jump process on S with embedded process (Xn , ξn ). As in Chapter 1, we adopt the convention that Pi {·} = P { · |X(0) = i}, and let Ei [·] denote the associated conditional

248

4 Continuous-Time Markov Chains

expectation. We denote the transition probabilities of X(t) by pij (t) = Pi {X(t) = j},

i, j ∈ S, t ≥ 0.

These probabilities will play a similar role for CTMCs that the n-step probabilities play for a discrete-time Markov chain. We begin with two results that prove Theorem 2. The integral equation (4.6) yields the Kolmogorov differential equations in the next section that determine transition rates and probabilities for CTMCs. Proposition 12. If the jump process X(t) on S is such that its embedded process satisfies conditions (i) and (ii) in Theorem 2, then X(t) is a CTMC and its transition probabilities pij (t) satisfy, for i, j ∈ S and t > 0, pij (t) = e

−qi t

1(i = j) +

 t 0 k=i

pkj (t − v)qi pik e−qi v dv.

(4.6)

Proof. To prove X(t) is a CTMC, it suffices to verify the Markov property that, for each i, j ∈ S and t, u ≥ 0, P {X(t + u) = j|X(t) = i, X(s), s < t} = pij (u).

(4.7)

∞ Consider the point process N (t) = n=1 1(Tn ≤ t), t ≥ 0, which denotes the number of transitions of the X(t) up to time t. For fixed states i and j, conditioning on N (t), we have P {X(t + u) = j|X(t) = i, X(s), s < t} (4.8) ∞  [am (t, u) + bm (t, u)]P {N (t) = m|X(t) = i, X(s), s < t}, = m=0

where am (t, u) = P {X(t + u) = j, Tm+1 > t + u|Fm (i, t)} bm (t, u) = P {X(t + u) = j, Tm+1 ≤ t + u|Fm (i, t)} Fm (i, t) = {N (t) = m, X(t) = i, X(s), s < t}. By condition (ii) on the exponentially distributed sojourn times and the memoryless property of the exponential sojourn time in state i, the conditional distribution of the residual sojourn time Tm+1 − t at time t is exponentially distributed with rate qi . Then am (t, u) = 1(i = j)P {Tm+1 − t > u|Fm (i, t)} = 1(i = j)e−qi u . Also, conditioning on Xm+1 and Tm+1 and using Remark 3,

4.3 Markov Properties

bm (t, u) =

249



 k=i

P {X(t + u) = j|Tm+1 = t + v, Xm+1 = k, Fm (i, t)}

0

k=i

=

u

×P {Xm+1 = k, Tm+1 − t ∈ dv|Fm (i, t)}

0

u

pkj (u − v)qi e−qi v pik dv.

Substituting these expressions for am (i, t) and bm (i, t) in (4.8), and noting that they are independent of m and t, we have P {X(t + u) = j|X(t) = i, X(s), s < t} = 1(i = j)e−qi u  u pkj (u − v)qi e−qi v pik dv. + k=i

(4.9)

0

Since this expression is true for all t ≥ 0, by setting t = 0 on the left-hand side, the right-hand side must equal pij (u). This proves (4.7), and it proves (4.6) as well. Proposition 13. If X(t) is a CTMC, then its embedded process (Xn , ξn ) satisfies conditions (i) and (ii) in Theorem 2. Proof. First, consider the function gi (t) = Pi {ξ1 > t}. For any s, t ≥ 0, gi (s + t) = Pi {ξ1 > s}Pi {ξ1 > s + t|ξ1 > s} = gi (s)gi (t). The last equality uses the memory less property of the exponential distribution. It is well known that any nonnegative, decreasing function satisfying such an equation has the form gi (t) = e−qi t , for some qi . The qi is a positive number since gi (0) = 1, and so Pi {ξ1 > t} = e−qi t . Now, to prove the proposition, it suffices by Theorem 2 and Remark 3 to show that, for i, j ∈ S, sk , t > 0, n ≥ 0, P {Xn+1 = j, ξn+1 > t|Xk , ξk = sk , k ≤ n, Xn = i} = pij e−qi t ,

(4.10)

for some qi > 0 and Markov transition probabilities pij , with pii = 0, i ∈ S. We will show this is true for the qi above and the Markov probabilities p ij = Pi {X(ξ1 ) = j}. Using the Markov property of X(t) at the time tn =  n k=0 sk , the probability on the left-hand side of (4.10) equals P {X(tn + ξn+1 ) = j, ξn+1 > t|X(tn ) = i} = P {X(tn + ξn+1 ) = j|X(tn ) = i} ×P {ξn+1 > t|X(tn ) = i, X(tn + ξn+1 ) = j}. Then (4.10) is true, since the last two probabilities equal Pi {X(ξ1 ) = j} = pij and

250

4 Continuous-Time Markov Chains

P {X(u) = i, u ∈ [tn , tn + t]|X(v) = i, v ∈ [tn , t), X(tn + ξn+1 ) = j} = P {X(u) = i, u ∈ [0, t]|X(v) = i, v ∈ [0, t), X(ξ1 ) = j} = Pi {ξ1 > t} = e−qi t . Stopping times and the strong Markov property for discrete-time Markov chains have natural analogues in continuous time. Definition 14. A random variable τ in [0, ∞] is a stopping time for a continuous-time stochastic process {X(t) : t ∈ R+ } if, for each t ∈ R+ , the event {T ≤ t} is a function of the history {X(s) : s ≤ t}. Remark 15. Strong Markov Property. A CTMC satisfies the Strong Markov Property, which is (4.7) with a stopping time in place of the time parameter t. A proof is in [61]. Similarly to discrete-time Markov chains, the finite-dimensional distributions of X(t) are determined by the transition probabilities and the initial distribution αi . Namely, for each i1 , . . . , in in S and 0 = t0 < t1 < · · · < tn , P {X(t1 ) = i1 , . . . , X(tn ) = in } n   αi0 pik−1 ,ik (tik − tik−1 ). = i0 ∈S

(4.11)

k=1

This follows by induction on n using the Markov property. Remark 16. A consequence of (4.11) is that two CTMCs are equal in distribution if and only if their initial distributions and transition probabilities are equal. The final observation of this section is that the transition probabilities satisfy the Chapman-Kolmogorov equations  pij (s + t) = pik (s)pkj (t), i, j ∈ S, s, t ≥ 0. k∈S

This follows by applying (4.11) to  Pi {X(s) = k, X(s + t) = j}. pij (s + t) = k∈S

Using matrix notation P(t) = {pij (t)}, the Chapman-Kolmogorov equations are P(s + t) = P(s)P(t), which means that the family of matrices {P(t) : t ≥ 0} forms a semigroup. A CTMC is sometimes defined via this semigroup associated with a process that satisfies the Markov property; additional technical conditions are needed to ensure that the chains are regular and do not have “instantaneous” states. This approach, which will not be covered here, yields the same type of CTMC we are considering.

4.4 Transition Probabilities and Transition Rates

251

4.4 Transition Probabilities and Transition Rates This section continues the discussion of the transition probabilities pij (t) for a CTMC X(t) with defining parameters (αi , pij , qi ). We introduce the notion of transition rates for the CTMC, and show how they are related to its transition probabilities. The main result describes Kolmogorov differential equations for the transition probabilities. We first introduce another important family of parameters for a CTMC. Definition 17. The transition rates of the CTMC X(t) are j = i.

qij = qi pij ,

Expression (4.15) below verifies that these qij are indeed “infinitesimal” transition rates in that, for i = j, pij (t) = qij t + o(t)

as t → 0.

These rates are similar to the rates qijof the exponential clock-times in Example 9, where pij = qij /qi and qi = j=i qij . We will now describe differential equations for the transition functions. Here we simplify summations by using the negative rate  qij . qii = −qi = − j=i

Theorem 18. For each i, j ∈ S, the derivative pij (t) exists and is continuous in t, and (4.12) lim pij (t) = 1(i = j). t→0

The pij (t) satisfy the Kolmogorov differential equations  pij (t) = qik pkj (t), “Backward Equation”

(4.13)

k

pij (t) =



pik (t)qkj .

“Forward Equation”

(4.14)

k

In particular, pij (0)

=

qij −qi

if j = i if j = i.

(4.15)

Proof. Consider (4.6), which is (with the change of variable u = t − v) ⎤ ⎡  t  eqi u qik pkj (u) du⎦ . (4.16) pij (t) = e−qi t ⎣1(i = j) + 0

k=i

252

4 Continuous-Time Markov Chains

The integral is continuous in t since its integrand is bounded on finite intervals, and so pij (t) is continuous. Then the integrand is continuous, and so the derivative of the integral exists, which in turn implies that pij (t) is differentiable in t. In addition, note that the limit (4.12) follows from (4.16). Next, taking the derivative of (4.16) and using some algebra, we have  qik pkj (t). pij (t) = −qi pij (t) + k=i

This proves that pij (t) is continuous in t, and that (4.13) holds. Letting t → 0 in this equation and using (4.12) proves (4.15). To prove (4.14), consider the Chapman-Kolmogorov equation  pik (t)pkj (s). pij (t + s) = k

Taking the derivative of this with respect to s yields  pik (t)pkj (s). pij (t + s) = k

The derivative of the sum follows by the bounded  convergence theorem in the Appendix. Letting s → 0, we have pij (t) = k pik (t)pkj (0). Applying (4.15) to the last term yields (4.14). Using matrix notation, the differential equations (4.13) and (4.14) are P (t) = QP(t),

P (t) = P(t)Q.

The matrix Q = {qij } is the infinitesimal generator of the semigroup P(t). The unique solution to either equation (with the condition P(0) = I) is P(t) = etQ =

∞ 

tn Qn /n!.

n=0

This is the matrix version of the well-known solution of the equations if P(t) were simply a real-valued function. There are only a few CTMCs for which the preceding infinite series expression for P(t) simplifies to a tractable formula. An example is the M/M/1 queueing process, but even this case is complicated [47] and is omitted. However, using another approach, the next section shows that the large class of CTMCs with bounded sojourn rates do indeed have tractable transition probabilities that are expectations of Poisson distributions. We end this section with several useful facts. Remark 19. Suppose two CTMCs with the same state space have the same initial distribution. Then the CTMCs are equal in distribution if and only if

4.5 Existence of CTMCs

253

their transition rates are equal. This follows since the CTMCs are equal in distribution if and only if their transition probabilities are equal (Remark 16); and two transition probability functions are equal if their associated transition rates are equal (Theorem 18). We will see later in Example 50 that the mean measure of the point process N (B) = ∞ n=1 1(Tn ∈ B), B ∈ B, of transition times is  E[N (B)] = E[qX(t) ] dt, B ∈ B. B

This mean measure tells us something about jump times. Remark 20. The probability is 0 that a CTMC X(t) has a jump at any fixed time t (i.e., X(t) = X(t−), or N ({t}) = 1) This follows since E[N ({t})] = 0 by the preceding expression for the mean measure. Here is another fact about sample paths. Remark 21. A CTMC X(t) is continuous in distribution in that lim P {X(s) = j} = P {X(t) = j},

s→t

j ∈ S.

 This follows since P {X(s) = j} = i P {X(0) = i}pij (s), and Theorem 18 ensures that pij (s) → pij (t) as s → t.

4.5 Existence of CTMCs We will now establish that there exists a CTMC for any set of defining parameters. We also show how to define a CTMC by a jump process that may have fictitious jumps from a state back to itself. Theorem 22. There exists a CTMC for any set of defining parameters. Proof. Consider a set of defining parameters (αi , pij , qi ), where the qi are necessarily P-regular by Proposition 4.4. Let Un , Vn , for n ≥ 0, denote independent random variables on a common probability space such that each Un is uniformly distributed on [0, 1] and each Vn is exponentially distributed with rate 1. The existence of these random variables follows from Corollary 6 in the Appendix. Now, let f be a function as in Theorem 14 in Chapter 1 such that Xn = f (Xn−1 , Un ) is a Markov chain with transition matrix P and initial distribution α. Next, let ξn = Vn /qXn . Then define X(t) to be the jump process on S with embedded process (Xn , ξn ). The constructed process X(t) will be a CTMC with the desired properties provided that the ξn satisfy conditions (4.2) and (4.3) in Theorem 2. But this follows since, by the independence of the sequences Xn and Vn ,

254

4 Continuous-Time Markov Chains

P {ξ0 ≤ t0 , . . . , ξm ≤ tm |Xn , n ≥ 0} = =

m 

P {Vn ≤ tn qXn |Xn }

n=0 m 

(1 − e−tn qXn ).

n=0

Next, we reconsider the characterization of a CTMC in Theorem 2 and show that the condition pii = 0, for each i, can be relaxed. Suppose (Xn , ξn ) is a Markov chain that satisfies assumptions (i)–(ii) in Theorem 2, with the exception that 0 ≤ pii < 1 is allowed. Associated with this chain, define a jump process X(t) by (4.1). For any state i where pii > 0, the chain Xn may jump from i back to i several times in a row, but these jumps are not seen in the process X(t) — they are “ fictitious” jumps. Proposition 23. The process X(t) described above is a CTMC with one-step transition probabilities p∗ii = 0, p∗ij = pij /(1 − pii ),

j = i,

and exponential sojourn rates qi∗ = (1 − pii )qi . Proof. The sequence of “distinct” states that X(t) visits is Xn∗ = Xνn , where ν0 = 0 and νn+1 = min{m > νn : Xm = Xνn }, n ≥ 0.  n+1 The sojourn time in state Xn∗ is ξn∗ = νm=ν ξm . Then X(t) is a jump n +1 ∗ ∗ process with embedded process (Xn , ξn ). To prove the assertion, it suffices by Theorem 2 and Remark 3 to show that (Xn∗ , ξn∗ ) is a Markov chain with transition probabilities ∗

P {X1∗ = j, ξ1∗ > t|X0∗ = i, ξ0 } = p∗ij e−qi t ,

(4.17)

and that the qi∗ are P∗ -regular. Exercise 26 shows that νn are stopping times of the Markov chain (Xn , ξn ), and so by the strong Markov property at time νn , ∗ ∗ ∗ P {Xn+1 = j, ξn∗ > t|Xn∗ = i, Xm , ξm , m < n} = Pi {Xν1 = j,

ν1 

ξm > t}

k=1

= Pi {Xν1 = j}Pi {

ν1 

ξm > t|Xν1 = j}.

k=1

By standard Markovian reasoning Pi {Xν1 = j} =

∞  =1

∗ p−1 ii pij = pij .

(4.18)

4.6 Uniformization, Travel Times and Transition Probabilities

255

Next, recall that Exercise 5 in Chapter 3 showed that a geometric sum of i.i.d. exponential random variables is again exponential, and note that ν1 conditioned on X(0) = i has a geometric distribution with mean 1/(1 − pii ). Therefore ν1  Pi { ξm > t} = e−qi (1−pii )t . k=1

Applying the last two displays to (4.18) and simplifying, it follows that (Xn∗ , ξn∗ ) is a Markov chain with transition probabilities (4.17). Next, using the P-regularity of the qi and E[νn+1 |Xn∗ ] = 1/(1 − pXn∗ ,Xn∗ ), it follows, upon conditioning on Xn∗ , that ∞ = E[

∞ 

1/qXn ] = E[

n=0

∞ 

νn+1 /qXn∗ ] = E[

n=0

∞ 

∗ 1/qX ∗ ]. n

n=0

Thus the qi∗ are P∗ -regular.

4.6 Uniformization, Travel Times and Transition Probabilities This section gives more insight into the transient behavior of CTMCs. We begin by describing a special CTMC whose exponential sojourn rates are all the same and consequently it has tractable transition probabilities. Remarkably, any CTMC with bounded sojourn rates is equal in distribution to a CTMC with uniform rates. The rest of the section shows how to derive travel time distributions and transition probabilities based on sums of exponential random variables. Example 24. CTMC with Identical Sojourn Rates. Suppose that Xn is a Markov chain with transition probabilities pij , with pii < 1. Assume that the sojourn times ξn in the respective states Xn are independent exponentially distributed with rate λ. Then the state of the chain at time t can be expressed as X(t) = XN (t) , t ≥ 0,  where N (t) = n 1(Tn ≤ t) is a Poisson process on R+ with rate λ that is independent of the Xn . The jump process X(t) is sometimes called the Markov chain Xn subordinated to the Poisson process N (t) with parameters pij and λ. Furthermore, by Proposition 23, this subordinated Markov chain X(t) is a CTMC, and its main defining parameters are p∗ij = pij /(1 − pii ),

j = i,

qi∗ = λ(1 − pii ).

256

4 Continuous-Time Markov Chains

A striking feature of the process X(t) is that its transition probabilities have a tractable form. Indeed, conditioning on N (t), N (t)

pij (t) = E[Pi {XN (t) = j|N (t)}] = E[pij Therefore,

∞ 

pij (t) =

].

pnij e−λt (λt)n /n!.

(4.19)

n=0

One can compute these probabilities by truncating the series. Next, we establish the important result that any CTMC with bounded exponential sojourn rates can be represented as a subordinated Markov chain. Proposition 25. (Uniformization of a CTMC) Suppose X(t) is a CTMC on S with defining parameters (αi , pij , qi ), where the qi are bounded. Then the ˆ process X(t) is equal in distribution to a subordinated Markov chain X(t) with parameters α ˆ i = αi , pˆij = qij /λ,

j = i,

pˆii = 1 − qi /λ,

qˆi = λ,

for any fixed λ ≥ supi qi . ˆ Proof. By Proposition 23, the subordinated chain X(t) is also a CTMC with parameters pˆij (αi , , qˆi (1 − pˆii )) = (αi , pij , qi ). 1 − pˆii Since the latter are the parameters for X(t), it follows by Proposition 4 that ˆ the processes X(t) and X(t) have the same distribution. One consequence of this uniformization principle is that any CTMC with bounded sojourn rates has transition probabilities of the form (4.19). This is true for any birth-death process with bounded rates, such as an M/M/s system with finite s. In addition, uniformization is a key tool in Markov decision theory (e.g., [90, 54, 109]), where a continuous-time Markov decision process is usually formulated by a simpler discrete-time Markov decision process. We now turn to another approach for deriving transition probabilities that involves evaluating sums of independent exponential sojourn times associated with a path of states. A tool for this is the following proposition, which is of general interest; Exercise 9 in Chapter 3 is a related result. Proposition 26. If Yk , k ∈ I, are independent exponentially distributed random  variables with rates qk that are distinct, then the distribution of Z = k∈I Yk is the mixture of exponential distributions

4.6 Uniformization, Travel Times and Transition Probabilities

FZ (t) =





(1 − e−qk t )

∈I,=k

k∈I

q , q − qk

Proof. Consider the Laplace transform   L(α) = E[e−αZ ] = E[e−αYk ] = k∈I

k∈I

257

t ≥ 0.

(4.20)

qk . α + qk

This is a ratio of polynomials in α, in which the denominator has distinct roots −qk , k ∈ I. Therefore, its partial-sum expansion is  qk ck , (4.21) L(α) = α + qk k∈I

where

  ck = (α + qk )L(α)

= α=−qk

 ∈I,=k

q . q − qk

Then the distribution with Laplace transform (4.21) is the mixture3 of exponential distributions given by (4.20). Example 27. Travel Times on Paths. Let ZI denote the travel time of a CTMC X(t) on a path of states I = (i0 , i1 , . . . , im ) whose exponential sojourn rates are distinct. Then the distribution of ZI is given by (4.20). Moreover, the travel time Z in a set P of such paths has the distribution  P {Z ≤ t} = pI FZI (t), I∈P

where pI = P {X0 = i0 }pi0 ,i1 · · · pim−1 ,im is the probability of traversing I. Example 28. Machine Deterioration Model. Consider a CTMC X(t) that represents the state of deterioration of a machine at time t, where the set of states is S = {0, 1, . . . , }. Assume that its deterioration is nondecreasing (pij = 0, j ≤ i ≤ ), and that p,0 = 1, so after its stay in state , it is replaced by a new machine. Then the lifetime of a machine is its travel time Z in the set P of all nondecreasing paths from 0 to . Thus the machine life-time distribution is as in Example 27. Example 29. If X(t) is a pure birth process as in Example 7 with distinct sojourn rates qi , then its transition probabilities are pij (t) =

j−1  k=i

3

[e−qj t − e−qk t ]

j−1 qk  q . qk − qj =i q − qk =k

Note that the coefficients ck may be negative and do not necessarily sum to 1.

258

4 Continuous-Time Markov Chains

 To see this, consider the travel time Z = k∈I Yk through the states I = (i, i + 1, . . . , j − 1), where Yi are the independent exponential sojourn times in the states. Then  t pij (t) = P {Z ≤ t < Z + Yj } = P {Yj > t − s}FZ (ds). 0

Substituting the exponential distribution for Yj and the distribution (4.20) in this integral and integrating proves the assertion. Sometimes transition probabilities can be determined by exploiting properties of Poisson processes as follows. Example 30. M/M/∞ System. Suppose X(t) is an M/M/∞ process with Poisson arrival rate λ and exponential service rate μ. To derive its transition probabilities at time t, we will consider two independent populations of items: those present at time 0, and those arrivals in (0, t) that are still in the system at time t. Now, each item present at time 0 is still in the system at time t with the probability e−μt that the residual service time exceeds t (the exponential distribution is memoryless). Then if there are i items present at time 0, the number of these still present at time t has the Binomial distribution with parameters i and 1 − e−μt . Next, note that since this process is a special case of the M/G/∞ process described in Chapter 3, we know that the number of the Poisson arrivals in (0, t) that are still present at time t has a Poisson distribution with mean 

t

η(t) = 0

e−μ(t−s) λds = (λ/μ)(1 − e−μt ).

Then the number of items in the system at time t is the sum of these independent binomial and Poisson random variables. Consequently, the transition probabilities are i

 η(t)j−k −η(t) i −kμt pij (t) = e e (1 − e−μt )i−k . (j − k)! k k=0

4.7 Stationary and Limiting Distributions The classification of states of a CTMC and its equilibrium behavior are closely related to those properties of its embedded Markov chain. We will now characterize stationary and limiting distributions for CTMCs by applying results for discrete-time Markov chains in Chapter 1. For this discussion X(t) will denote a CTMC on S with embedded chain (Xn , ξn ) and defining parameters (αi , pij , qi ). Also, qij and pij (t) will denote its transition rates and transition probabilities.

4.7 Stationary and Limiting Distributions

259

Our first concern is a classification of states. This mirrors the classification for the embedded chain Xn . A state i ∈ S is recurrent (or transient) for X(t) if i is recurrent (or transient) for Xn . The process X(t) is irreducible if Xn is. Recall the convention that Xn does not have any absorbing states; consequently, neither does X(t). CTMCs do not have periodic states, because their sojourn times in states are continuous random variables. However, their embedded chains may be periodic in discrete time. To describe positive recurrence, we will use the discrete- and continuoustime first passage times νi = min{n ≥ 1 : Xn = i} and τi = inf{t > ξ0 |X(t) = i} =

ν i −1

ξn .

n=0

A recurrent state i for X(t) is positive recurrent or null recurrent according as the mean Ei [τi ] is finite or infinite. Finally, X(t) is ergodic if it is irreducible and all of its states are positive recurrent. In relating X(t) to its embedded chain Xn , keep in mind that X(t) is irreducible and recurrent if and only if Xn is irreducible and recurrent. On the other hand, Exercise 28 shows that X(t) may be positive recurrent while Xn is not, and vice versa. Of course, if the transition rates are bounded away from 0 and ∞, then X(t) is ergodic if and only if Xn is. Our last preliminary is a formula for relating mean cycle values of X(t) to those of Xn . Lemma 31. If X(t) is irreducible and recurrent, then, for f : S → R,  Ei [

ν i −1

τi

0

f (X(t)) dt] = Ei [

f (Xn )/qXn ],

(4.22)

n=0

provided these means exist. Proof. Using the pull-through property, the left side of (4.22) is ν i −1

Ei [

f (Xn )ξn ] = Ei

n=0

∞



 1(νi > n)f (Xn )Ei [ξn |Xk , k ≤ n]

n=0

and the last expression equals the right side of (4.22). We will now describe stationary and invariant measures for CTMCs. Definition 32. A probability measure p on S is a stationary distribution for X(t) if i

 η(t)j−k −η(t) i e (1 − e−μt )k e−(i−k)μt . pij (t) = (j − k)! k k=0

260

4 Continuous-Time Markov Chains

This equation in matrix notation is p = pP(t), where p = (pi : i ∈ S) is a row vector. More generally, any measure γ on S that satisfies γ = γP(t), t ≥ 0, is an invariant measure for X(t). As in discrete time, a stationary distribution is sometimes called an equilibrium distribution, and it is related to the notion of a stationary process. The proof of the following result parallels that of Proposition 52 in Chapter 1. Recall that a continuous-time process {X(t) : t ≥ 0} on a space S is stationary if, for any s1 < · · · < sm , d

(X(s1 + t), . . . , X(sm + t)) = (X(s1 ), . . . , X(sm )),

t ≥ 0.

Proposition 33. For the CTMC X(t), which need not be irreducible or recurrent, the following statements are equivalent. (a) X(t) is a stationary process. d

(b) X(t) = X(0), t > 0. (c) The distribution of X(0) is a stationary distribution. We will now establish the existence of an invariant measure for X(t). Recall the transition-rate notation Q = {qij }, where qij = qi pij (j = i), and  qii = −qi = qij . j=i

Theorem 34. For an irreducible, recurrent CTMC X(t) and a nonnegative measure γ on S, the following statements are equivalent. (a) γ is an invariant measure for X(t). (b) γi qi is an invariant measure for Xn . (c) γ satisfies the balance equations γQ = 0, or equivalently,   qij = γj qji , i ∈ S. (4.23) γi j=i

j=i

Furthermore, for a fixed i, the measure γ defined by   τi  γj = Ei 1(X(t) = j) dt , j ∈ S,

(4.24)

0

is a positive invariant measure for X(t), and this measure is unique up to a multiplication by a constant.

4.7 Stationary and Limiting Distributions

261

Proof. (a) ⇔ (c). For any measure γ on S, using the representation P(t) = etQ , we have γP(t) = γetQ = γ +

∞ k  t k=1

k!

γQk .

Thus γ is an invariant measure for X(t) (i.e., γP(t) = γ) if and only if γQk = 0 for each k. But the latter (by induction) is equivalent to γQ = 0. (b) ⇔ (c). Clearly (b) implies (c) since using pij = qij /qi ,   γi qi = γj qj pji = γj qji . j=i

j=i

Reversing the last two sums proves that (c) implies (b). Theorem 35. For an irreducible, recurrent CTMC X(t) and a positive distribution measure p on S, the following statements are equivalent. (a) X(t) is ergodic with stationary distribution p. (b) p satisfies the equations pQ = 0. (c) For a fixed i, Ei [τi ] is finite, and   τi  1 Ei pj = 1(X(t) = j) dt , j ∈ S. (4.25) Ei [τi ] 0 Proof. The equivalence of (a) and (b) follows by Theorem 34 (statements (a) and (c)). Next, suppose X(t) is ergodic with stationary distribution p. Then each Ei [τi ] is finite and p is a positive distribution. Also, by Theorem 34, p must = ci γj (ci as well be a multiple of γ in (4.24), and so, for a fixed i, we have pj  as γj depends on i). Then summing this on j yields 1 = ci j γj = ci Ei [τi ]. These observations imply pj = γj /Ei [τi ], which is equivalent to (4.25). This proves that (a) implies (c). Finally, suppose (c) holds. The p given by (4.25) is positive and it is a multiple of the invariant measure γ in (4.24). In addition, it clearly sums to 1. Therefore it is a stationary distribution for X(t). This proves that (c) implies (a). Remark 36. Another representation of the distribution in (4.25) is pj =

1 qj Ej [τj ]

This follows from (4.25) by setting i = j.

j ∈ S.

262

4 Continuous-Time Markov Chains

Recall that the preceding results do not require that Xn be ergodic. If it is, however, the stationary distributions of X(t) and Xn are related as follows. Proposition 37. If X(t) and Xn are ergodic with respective stationary distribution p and π, then πj /qj pj =  , k πk /qk

j ∈ S.

Proof. Remark 36 tell us that pj = 1/(qj Ej [τj ]). Also, by Lemma 31 with f (·) = 1 and Proposition 69 in Chapter 1, Ej [τj ] = Ej

j −1

ν

  −1 = πj−1 qX πk /qk . n

n=0

k

Combining these two observations proves the assertion. Before getting into applications of stationary distributions, we will relate them to limiting distributions. We begin by describing the regenerative property of CTMCs, which is comparable to Proposition 67 of Chapter 1 for discrete-time chains. The proof, like that in discrete time, is by an application of the strong Markov property in continuous time. Proposition 38. An irreducible, recurrent CTMC X(t) is a delayed regenerative process over the times at which the process enters a fixed state. We end this section by establishing that the stationary distribution of an ergodic CTMC is also its limiting distribution. The limiting distribution of the CTMC X(t) is defined by pj = lim pij (t), t→∞

i ∈ S,

provided the limits exist and do not depend on i. Theorem 39. (Limiting Distribution) If the CTMC X(t) is ergodic, then its stationary distribution given by (4.25) is its limiting distribution. Proof. By Proposition 38, X(t) is a delayed regenerative process over the hitting times of a state i. Also, the i.i.d. times between hitting i have a nonarithmetic distribution due to the exponential sojourn times in the states. Then by the characterization of limiting distributions for regenerative processes in Corollary 46 in Chapter 2, it follows that the limiting distribution of X(t) is the same as the stationary distribution given by (4.25).

4.8 Regenerative Property and Cycle Costs

263

4.8 Regenerative Property and Cycle Costs One use of stationary distributions is in evaluating means of functionals of a CTMC in between successive visits to a fixed state. This section contains formulas for two such “cycle” means. The formulas are of interest by themselves and are used to evaluate limiting averages of functionals, as we will see in the next section. An analogous result for Markov chains is Proposition 69 in Chapter 1. The first formula is for a cycle cost or utility that is an integral of a rate function. Proposition 40. If X(t) is an ergodic CTMC with stationary distribution p, then, for f : S → R,

 τi  1  Ei f (X(t)) dt = f (j)pj . (4.26) pi qi j 0 provided the sum is absolutely convergent. Proof. The left-hand side of (4.26) equals



j f (j)Ei

!

τi 0

 1(X(t) = j) dt ,

and this equals the right-hand side of (4.26) by (4.25) and Remark 36. The next formula is for a cycle cost for a process with regenerative increments; see Section 2.10. Proposition 41. Associated with as ergodic CTMC X(t) with stationary distribution p, suppose that {Z(t) : t ≥ 0} is a real-valued stochastic process with delayed regenerative increments over the times at which X(t) enters a fixed state i, and Z(0) = 0. Assume that 

  Ei Z(Tn+1 ) − Z(Tn )Xm , m ≤ n = hi (Xn ), i ∈ S, n ≥ 0, for some hi : S → R. Then Ei [Z(τi )] =

1  hi (j)pj qj , pi qi j

(4.27)

provided the sum is absolutely convergent. Proof. Since {νi > n} is a function of X0 , . . . , Xn , using the pull-through property for conditional probabilities and the hypotheses,

264

4 Continuous-Time Markov Chains

Ei [Z(τi )] = Ei

i −1

ν  [Z(Tn+1 ) − Z(Tn )]

n=0

= Ei

∞





  1(νi > n)Ei [Z(Tn+1 ) − Z(Tn )]X0 , . . . , Xn

n=0

= Ei

i −1

ν

 hi (Xn ) .

n=0

Applying Lemma 31 to the last term and then using Proposition 40, we obtain

 τi  1  Ei [Z(τi )] = Ei hi (X(t))qX(t) dt = hi (j)pj qj . pi qi j 0

4.9 Ergodic Theorems In this section, we present several SLLNs for CTMCs that apply to a variety of functionals associated with cost and performance parameters. These results follow from the general SLLN for processes with regenerative increments covered in Chapter 2. Included are insights on the rate of convergence of the SLLNs, based on the refined SLLN in Chapter 2. For this development, X(t) will be an ergodic CTMC with stationary distribution p, and τi is the hitting time of state i. Theorem 42. Suppose that {Z(t) : t ≥ 0} is a real-valued stochastic process that has delayed regenerative increments over the times at which X(t) enters a fixed state i, and Z(0) = 0. Assume Ei [supt≤τi |Z(t)| ] is finite, and 

  n ≥ 0, Ei Z(Tn+1 ) − Z(Tn )Xm , m ≤ n = h(Xn ), for some h : S → R. Then assuming the sum is absolutely convergent,  h(j)qj pj a.s. lim t−1 Z(t) = t→∞

j

Proof. By the SLLN for processes with regenerative increments (Theorem 54 in Chapter 2), we have lim t−1 Z(t) = Ei [Z(τi )]/Ei [τi ]

t→∞

But this limit is finite and equals

 j

a.s.

h(j)qj pj by (4.27) and Remark 36.

The process Z(t) in the preceding theorem is a general model for various functionals of the CTMC X(t). Here are some illustrations.

4.9 Ergodic Theorems

265

∞ Example 43. Jump Times. Consider the process N (t) = n=1 1(Tn ≤ t), which records the number of jump times Tn in (0, t]. This point process is of interest by itself and it is used in functionals based on jump times. The average number of jumps per unit time (or rate of N (t)) is  lim t−1 N (t) = pj qj a.s., t→∞

j

provided the sum is finite. This follows by Theorem 42 with Z(t) = N (t) and Z(Tn+1 ) − Z(Tn ) = 1. In addition, knowing the rate of N (t), Theorem 10 in Chapter 2 tells us that the average sojourn time over all the states determined by the Xn is lim n−1

n→∞

n−1 

ξm = 1/

m=0



pj qj

a.s.

j

!t Example 44. Integral Functionals. Suppose Z(t) = 0 V (s) ds, where V (t) is a delayed regenerative process over the times at which X(t) enters a fixed state. Then the integral process Z(t) has delayed regenerative increments over these times. The V (t) might be a cost or utility rate at time t associated with a CTMC and auxiliary environmental information, such as V (t) = g(X(t), Y (t)), for g : S × S  → R. The limit statement in Theorem 42 will be true for a particular application upon verifying the other assumptions in the theorem. The preceding example sets the stage for the following classical result. Theorem 45. (SLLN for CTMCs). For the ergodic CTMC X(t) with stationary distribution p, and a function f : S → R,  t  f (X(s)) ds = f (j)pj a.s., (4.28) lim t−1 t→∞

0

j

provided the sum is absolutely convergent. Proof. First note that, for n ≥ 0,



Tn+1

Ei Tn

   −1 f (X(t)) dtXm , m ≤ n = Ei [f (Xn )ξn |Xn ] = f (Xn )qX . (4.29) n

This also holds with |f (j)| in place of f (j). Then using (4.27),

 τi  1  Ei [sup |Z(t)| ] ≤ Ei |f (X(t))| dt = |f (j)|pj . pi qi j t≤τi 0

(4.30)

This quantity is finite by assumption. Thus (4.28) follows by Theorem 42.

266

4 Continuous-Time Markov Chains

The preceding SLLN has the following rate of convergence. Example 46. Refined SLLN. In the context of the preceding theorem, assume the time between entrances to a fixed state i has a finite variance σi2 , and let μi = Ei [τi ] = 1/qi pi . Then

 E 0

t

  f (X(s)) ds = [t + (σi2 + μ2i )/2μi ] f (j)pj −



j

f (j)pj /qj + o(1),

as t → ∞. (4.31)

j

In particular, the expected time spent in state i has the asymptotic behavior 

 t 1(X(s) = i) ds = tpi + pi [(σ 2 + μ2i )/2μi − 1/qi ] + o(1), as t → ∞. E 0

Applying Theorem 85 in Chapter 2 to Z(t) = the required constants are (see Corollary 40)  a = Ei [Z(τi )] = μi f (j)pj , c=−

1 Ei μi

 0

!t 0

f (X(s)) ds yields (4.31);

j

τi

  f (X(t)) dt = − f (j)pj /qj . j

Example 44 and Theorem 45 concerned integrals of cost or utility “rates”. Here is another type of functional associated with transition times. N (t) Example 47. Functionals of Embedded Processes. Suppose Z(t) = n=0 Vn , where Vn is a cost or parameter associated with the jump time Tn . As an example, suppose that g(i, j, t, y) is a cost incurred at the beginning of a sojourn time of length t in state i that ends with a jump to state j, and y is an auxiliary variable. Then Vn = g(Xn , Xn+1 , ξn , Yn ),

n ≥ 0,

is the cost incurred at time Tn , where Yn are auxiliary random variables and T0 = 0. Assume the Yn are i.i.d. and independent of the CTMC. By the exponential sojourn time and jump probabilities pjk , the expected cost incurred at time Tn in state Xn = j is h(j) = E[Vn |Xm , m ≤ n − 1, Xn = j]  ∞  pjk E[g(j, k, t, Y1 )]qj e−qj t dt. = k=j

0

Assume this exists and is finite when g is replaced by |g|.

4.9 Ergodic Theorems

267

Under these assumptions, the average cost is N (t)

lim t

−1

t→∞



Vn =

n=0



h(j)qj pj

j

=



pj

j



a.s. 

qjk

k=j

∞

0

 E[g(j, k, t, Y1 )]qj e−qj t dt , (4.32)

provided the sum is absolutely convergent. In particular, N (t)

lim t−1

t→∞



g(Xn , Xn+1 ) =

n=0



pj

j



qjk g(j, k) a.s.,

(4.33)

k=j

Expression (4.32) will follow by Theorem 42 upon verifying its assumptions. Clearly, Ei [Z(Tn+1 ) − Z(Tn )|Xm , m ≤ n] = Ei [Vn |Xn ] = h(Xn ), Also, Ei [ sup |Z(t)| ] ≤ Ei t≤τi

(τi )

N

 |Vn | ,

n=0

and, similarly to (4.30), the last term is finite when the right-hand side of (4.32) is absolutely convergent. Thus Theorem 42 yields (4.32). A typical application of the SLLN (4.32) involves defining Vn by an appropriate function g, evaluating  the mean function h(j), and verifying the absolute convergence of the sum j h(j)qj pj . Here is an illustration. Example 48. Consulting Company. Potential projects arrive to a consulting company according to a Poisson process with rate λ. The times to do the projects are independent exponentially distributed with rate μ. The company is small and can only handle one project at a time. Therefore, when it is working on a project, any potential projects that arrive are rejected.4 Suppose the revenues from completing the projects are i.i.d. with mean α and the revenue from the rejected projects are i.i.d. with mean α ¯ (which may be higher than α due to a tarnished image of the company). Of interest are the average revenue from completing the projects and the average revenue lost from the rejected projects. To derive these averages, let X(t) denote the state of the company at time t, where the states are 1 if a job is in service and 0 otherwise. Clearly X(t) is a CTMC with transition rates q10 = μ (the service rate) and q01 = λ (the rate of the Poisson arrival process, which has stationary independent increments). 4

This model, which is sometimes called an M/M/1/1 system, might also be appropriate for any service station or computer that can only work on one job at a time.

268

4 Continuous-Time Markov Chains

It is easily seen that the stationary distribution of this two-state CTMC is5 p0 = λ−1 /(λ−1 + μ−1 ) = μ/(λ + μ),

p1 = 1 − p0 = λ/(λ + μ).

The revenue received at time Tn for completing a project when the company is in state Xn is Vn = Yn 1(Xn = 0) , where Y1 , Y2 , . . . are i.i.d. project revenues, independent of the CTMC, with mean α. Although this revenue may be received any time during the engagement of a project, we assume, with no loss in generality, that it is received at the beginning of a project. Then it follows from (4.33) that the average revenue from completing the projects is N (t)  Vn = αp0 a.s. lim t−1 t→∞

n=0

To formulate the lost revenue from rejected projects, note that when the company is busy on a project, the lost revenue can be represented by a compound Poisson process M (t) with rate λ and the distribution of an increment is that of a single lost-project with mean α ¯ . Then the lost-project revenue at time Tn is Vn = Mn (ξn )1(Xn = 1), where Mn are i.i.d. compound Poisd son processes with Mn = M that are independent of the (Xn , ξn ). Although the arrival times of rejected projects occur during a sojourn time in state 1, we assume, with no loss in generality, that their lost-project revenues are incurred at time Tn . Note that Vn has the form Vn = g(Xn , ξn , Mn ) as in (4.32) and h(j) = E[Vn |Xn = j] = E[Mn (ξn )|Xn = j]¯ α1(j = 1) = λμ−1 α1(j ¯ = 1). Therefore, by (4.32), the average lost-revenue is N (t)

lim t

t→∞

−1



Vn = λμ−1 α ¯ p1

a.s.

n=0

One could use the preceding information to maximize profit. For instance, suppose the service rate μ could be varied by changing the number of workers or amount of overtime, and the cost per unit time of having a rate μ is C(μ). Then the average revenue would be αp0 − λμ−1 αp ¯ 1 − C(μ). One could then select μ to maximize this average revenue. Another quantity of interest is the average number of projects that take longer than d days to complete. As a slightly different application of (4.32), the average number of projects that take longer than d days to complete is N (t)

lim t−1

t→∞ 5



1(Xn = 1, ξn > d) = p1 e−dμ

a.s.

n=0

The two-state Markov chain was covered in Exercise 10 in Chapter 1.

4.10 Expectations of Cost and Utility Functions

269

Our next example concerns rates of several types of transitions associated with the ergodic CTMC X(t) with stationary distribution p. Analogous results for discrete-time are in Section 1.13. Example 49. Movement Between Sets. The average number of transitions the CTMC makes from a set of states A to a set of states B per unit time is N (t)

λ(A, B) = lim t−1 t→∞



1(Xn ∈ A, Xn+1 ∈ B).

n=0

By (4.33) this limit exists and equals   pi qij . λ(A, B) = i∈A

j∈B

In particular, the rate at which the CTMC moves from i to j is λ(i, j) = pi qij . Recall that the total balance equations for the CTMC are   pi qij = pj qji , i ∈ S. j=i

j

This is equivalent to λ(i, S) = λ(S, i). That is, the rate at which the CTMC moves out of state i is equal to the rate at which it moves into i. More generally, it follows as in Chapter 1 that λ(A, Ac ) = λ(Ac , A), the rate at which the chain moves out of a set A is the rate at which it moves into A. The number of entrances the CTMC makes into a set A ⊂ S up to time t is defined by N (t)  NA (t) = 1(Xn ∈ A, Xn+1 ∈ A). n=0

Then as above, the average number of entrances per unit time into A is  pi qij a.s. λ(A) = lim t−1 NA (t) = t→∞

i∈Ac ,j∈A

This is the same as the rate λ(Ac , A) at which the chain enters A. A related quantity is the nth time the chain enters the set A, which is τA (n) = min{t : NA (t) = n}. By Theorem 10 in Chapter 3, the limiting average of these times is limt→∞ n−1 τA (n) = 1/λ(A) a.s.

4.10 Expectations of Cost and Utility Functions We have been discussing limiting averages for functionals of CTMCs. We now switch back to finite time and present an important formula for means

270

4 Continuous-Time Markov Chains

of functionals of CTMCs. In addition to being useful for cost functions, the formula is a key tool for our analysis of Palm probabilities later in Sections 4.15 and 4.16. Here X(t) will denote a CTMC with the usual notation, but with no other assumptions about it being ergodic, etc. We begin with two examples of the upcoming main result. Example50. Mean Measure of Transition Times. Consider the point process ∞ N (t) = n=1 1(Tn ≤ t) of transition times of X(t). Its mean by (4.35) is 

t

E[N (t)] = 0

t ≥ 0,

E[qX(s) ] ds,

 provided the integral exists. This mean is finite if i qi < ∞. In particular, if X(t) is stationary with distribution p, then  pi qi . E[N (t)] = tE[N (1)] = t i

Example 51. Discounted Rewards. Suppose X(t) is in equilibrium with stationary measure p, and a discounted reward r(i)e−αt is received whenever it enters state i at time t. Then the expected total discounted reward over the infinite horizon is ∞

   −αTn E = r(Xn )e E[qX(t) r(X(t))]e−αt dt R+

n=1

= α−1



pi qi r(i).

i

The preceding are examples of the following main result, which is formula (4.35) for the mean of various functions of X(t). This result is a generalization of the L´evy formula in Example 53 below. Theorem 52. Suppose {Vn : n ≥ 1} are real-valued random variables associated with the CTMC X(t). Assume that the conditional expectations h(ζn ) = E[Vn |ζn ],

n ≥ 1,

(4.34)

exist, where ζn = (Tn , Xn−1 , Xn ) and h : R+ × S 2 → R is independent of n. Then ∞  



  E Vn = E qX(t),j h(t, X(t), j) dt, (4.35) n=1

R+

j=X(t)

provided the last integral is finite with |h| in place of h. Proof. By Exercise 8 in Chapter 3, we know that if ξ is an exponential random !ξ variable with rate λ, then E[h(ξ)] = λE[ 0 h(u) du]. Using an analogous

4.10 Expectations of Cost and Utility Functions

271

expression for conditional expectations applied to ξn−1 , along with (4.34) and Markov properties of (Xn , ξn ), we have

 E[Vn ] = E E[Vn |ζn ] = E[h(ζn )]

 = E E[h(Tn−1 + ξn−1 , Xn−1 , Xn )|Tn−1 , Xn−1 , Xn ]  ξn−1 

h(Tn−1 + u, Xn−1 , Xn ) du = E qXn−1 0



 h(t, Xn−1 , Xn ) dt .

Tn

= E qXn−1 Tn−1

The change-of-variable t = Tn−1 + u is used for the last line. Under further conditioning, E[Vn ] = E



E qXn−1



   h(t, Xn−1 , j) dtXn−1 , Tn−1 , Xn = j

Tn

Tn−1

j=Xn−1

×P {Xn = j|Tn−1 , Xn−1 }  Tn 

 qXn−1 ,j h(t, Xn−1 , j) dt . =E



Tn−1

j=Xn−1

The last line uses the fact that the conditional probability in the second line is qXn−1 ,j /qXn−1 . Finally, using Xn−1 = X(t) for t ∈ [Tn−1 , Tn ) E

∞



 Vn = E

n=1



∞

 

 =

E R+

h(t, Xn−1 , j) dt Tn−1

n=1 j=Xn−1





Tn

qXn−1 ,j

 qX(t),j h(t, X(t), j) dt.

j=X(t)

Here are two examples. Example 53. L´evy Formula. For f : S 2 → R and t ≥ 0, E

(t)

N 

  t   f (Xn−1 , Xn ) = E qX(s),j f (X(s), j) ds, 0

n=1

j=X(s)

provided the last integral is finite with |f | in place of f . This formula is a special case of Theorem 52, since N (t)



n=1

f (Xn−1 , Xn ) =

∞  n=1

f (Xn−1 , Xn )1(Tn ≤ t).

272

4 Continuous-Time Markov Chains

Example 54. For f : R+ × S → R, E

∞



  f (Tn , Xn ) =

n=1

R+

E[qX(t) f (t, X(t))] dt,

provided integral is finite with |f | in place of f . Remark 55. The results above also apply with obvious modifications to the process {X(t) : t ∈ R} defined on the entire time axis R, which is natural when considering stationary processes. In particular, the transition ∞ times < T < T ≤ 0 < T < T · · · , and would be labeled · · · < T −2 −1 1 2 n=1 and !0 !  would be replaced by and . n∈Z R+ R

4.11 Reversibility The focus in this and the next few sections will be on describing reversible CTMCs. Recall that Chapter 1 introduced the notion of reversibility for discrete-time Markov chains. We now describe an analogous reversibility for CTMCs The results includes a canonical formula for the stationary distribution of reversible processes and a characterization of a CTMC in reverse time. The next section describes several tools for formulating reversible CTMCs and further examples. As usual, X(t) will denote a CTMC with embedded process (Xn , ξn ) and the standard notation. We begin with terminology on reversibility. Definition 56. A CTMC X(t) (or its transition rate matrix qij ) is reversible with respect to a measure γ on S if γ satisfies the detailed balance equations γi qij = γj qji

i = j ∈ S.

(4.36)

Our first observation is that if X(t) is reversible with respect to γ, then γ is an invariant measure for the chain. This follows since summing the detailed balance equations on j yields the total balance equations γQ = 0 or   qij = γj qji . γi j=i

j=i

Of course, when γ is finite, it can be normalized to be the stationary distribution of the process. Another observation is that the reversibility of X(t) is equivalent to the reversibility of its embedded chain Xn . In other words, X(t) is reversible with respect to γ if and only if Xn is reversible with respect to πi = γi qi , i ∈ S. This follows by the definition of reversibility and qij = qi pij . The definition of reversibility for CTMCs is essentially the same as that for discrete-time Markov chains in Chapter 1; the only difference is that

4.11 Reversibility

273

transition rates are used instead of transition probabilities. However, reversibility in continuous time has more interesting applications as we will see. The discrete-time results in Chapter 1 also apply (with transition rates in place of transition probabilities) to CTMCs, and so we can exploit these results here. For instance, if X(t) is reversible, then it has the two-way communication property: for each i = j in S, the transition rates qij and qji are both positive or both equal to 0. Also, X(t) is reversible with respect to γ if and only if   γi qij = γj qji , A, B ⊂ S. i∈A,j=i∈B

j∈B,i=j∈A,

This says that the rate of transitions between any two sets is equal to the rate of the reverse transitions, when the chain is ergodic. The important characterization of invariant measures in Theorem 95 in Chapter 1 has the following continuous-time analogue. Theorem 57. If a CTMC X(t) is reversible, then an invariant measure for it is γi0 = 1 and   qik−1 ,ik γi = , i ∈ S\{i0 }, qik ,ik−1 k=1

where i0 is a fixed state and i0 , i1 , . . . , i = i is any path from i0 to i. Here is the companion result. Kolmogorov Criterion. A CTMC X(t) with the two-way communication property is reversible if and only if, for each n ≥ 3 and i1 , . . . , in in S with in = i1 , and ik = ik+1 , 1 ≤ k ≤ n − 1, n−1  k=1

qik ,ik+1 =

n−1 

qik+1 ,ik .

i=1

Quintessential examples of reversible processes are birth-death processes, including M/M/s queueing processes. These are special cases of the following general models. Example 58. Up-Down Process. Let X(t) = (X1 (t), . . . , Xm (t)) be a CTMC on S = Zm + , where a typical state is denoted by x = (x1 , . . . , xm ). The CTMC might represent quantities of jobs at m processing stations, stock levels of m products in a warehouse, outstanding orders at m part suppliers, or simply locations of items moving in the plane. Consider the general case in which the only movements may be up or down according to the transition rates qxy =

v(x) u(y) 1(x < y) + 1(x > y), u(x) v(y)

where u(·) and v(·) are positive functions on S.

274

4 Continuous-Time Markov Chains

Viewing u˜(x) = −log u(x) as a potential function, u(y)/u(x) = e−(˜u(y)−˜u(x)) reflects the change in potential by component “increases” from x to y. Similarly v(x)/v(y) reflects a “decrease” in potential. An easy check shows that X(t) is reversible with respect to γx = u(x)/v(x). This model has an obvious extension to a partially-ordered state space S (e.g., one can model excursions on partially-ordered graphs). Example 59. Batch Birth-Death Processes. A special case of the preceding process is a CTMC X(t) = (X1 (t), . . . , Xm (t)) that describes the numbers of items in m populations, where Xi (t) is the number of items in the ith population. The population state x may increase to x+ a or decrease to x− a, where a is a positive m-vector in a fixed subset A of S of allowable increments. For simplicity, assume the set A contains the unit vectors e1 , . . . , em , where ei is the vector with 1 in position i and 0 elsewhere. Assume that λi (xi ) is a single-unit birth rate in population i when xi are present, and that the birth rate of a batch of size ai is λi (xi )λi (xi + 1) · · · λi (xi + ai − 1). This is like a compound transition xi → xi + 1 → · · · → xi + ai occurring instantaneously under the single birth rate function λi (·). Single death rates μi (xi ) and batch death rates are defined similarly. In other words, the transition rates of the CTMC are 'm a i    qxy = λi (xi + k − 1)1(y = x + a) a∈A

i=1 k=1

+

ai m  

( μi (xi − k + 1)1(y = x − a) .

i=1 k=1

Note that these transition rates are the same as those in the preceding example with u(x) =

xi m  

λi (k − 1),

i=1 k=1

v(x) =

xi m  

μi (k).

i=1 k=1

Therefore, an invariant measure for this process is γx =

xi m  

λi (k − 1)/μi (k).

i=1 k=1

Interestingly, because of the multiplicative nature of the batch rates, this measure does not depend on the form of the set A of batch sizes as long as it contains the m unit vectors. In particular, any process has the same invariant

4.11 Reversibility

275

measure as the birth-death process with “single” births and deaths in which A = {e1 , . . . , em }. Example 60. Classical Birth-Death Processes and Queues. From the preceding example, it follows that a classical birth-death process X(t) as in Example 10 with birth and death rates λi and μi , respectively, is reversible with respect to the measure γ0 = 1 and γi =

λ0 · · · λi−1 , μ1 · · · μi

i ≥ 1.

Then the process is ergodic if and only if the sum of these γi is finite. In that  γ case, its stationary distribution is pi = cγi , i ≥ 0, where c−1 = ∞ i=0 i . In particular, the classical M/M/s queueing process in Example 11 with Poisson arrival rate λ and service rate μ is reversible. The process with s = ∞ servers is automatically ergodic and the process with 1 ≤ s < ∞ servers is ergodic if and only if λ < sμ. Their stationary distributions are as follows. M/M/1 system with λ < μ:

pi = (1 − λ/μ)(λ/μ)i ,

i ≥ 0.

M/M/s system with λ < sμ and s < ∞:  0≤i≤s c(λ/μ)i /i! pi = i−s ps (λ/sμ) i > s. M/M/∞ system:

pi = e−λ/μ (λ/μ)i /i!

i ≥ 0.

We end this section by showing how the reversibility of the CTMC X(t) via detailed balance equations is related to X(t) viewed backward in time. For a fixed τ > 0, consider the process X τ (t) = X(τ − t),

0 ≤ t ≤ τ.

Each sample path of this process corresponds to a sample path of X(t) in reverse time starting at τ (like viewing a video tape in reverse). The process {X τ (t) : 0 ≤ t ≤ τ } is called the time reversal of X(t) at τ .6 Lemma 61. The process X τ (t) is a non-time-homogeneous CTMC with time-dependent transition probabilities, for 0 ≤ s ≤ t ≤ τ , P {X τ (t) = j|X τ (s) = i} =

6

P {X(τ − t) = j} pji (t − s). P {X(τ − s) = i}

(4.37)

Note that X τ (t) has “left-continuous” paths instead of the conventional right-continuous paths. The right-continuous version X((τ − t)−) is a little cumbersome.

276

4 Continuous-Time Markov Chains

If in addition X(t) is stationary with distribution p, then X τ (t) is a timehomogeneous CTMC that is stationary, and its transition probabilities (4.37) reduce to t ∈ [0, τ ]. (4.38) p−1 i pj pji (t), Hence its transition rates are p−1 i pj qji . Proof. Consider the probability P {X τ (t) = j|X τ (s) = i, A} =

P {X(τ − t) = j, X(τ − s) = i, A} , P {X(τ − s) = i, A}

for any event A that is a function of {X τ (u) : 0 ≤ u < s}, and 0 < s ≤ t < τ . To prove the first assertion, it suffices to show that this fraction equals the right side of (4.37). But this equality follows since the denominator equals P {X(τ − s) = i}P (A|X(τ − s) = i) and the numerator equals P {X(τ − t) = j}P {X(τ − s) = i|X(τ − t) = j}P (A|X(τ − s) = i), because A is a function of {X(u) : τ < u < τ + s} and X(t) is Markovian. When X(t) is stationary with distribution p, then clearly the transition probabilities (4.37) (for s = 0) reduce to (4.38), and the associated transition rates, by Theorem 18, are p−1 i pj qji . Therefore, X(t) is a time-homogeneous CTMC. In addition, X τ (t) = X(τ −t) has the distribution p for each t ∈ [0, τ ], and hence X τ (t) is stationary with distribution p by Proposition 33. The notion of a time-reversal process on a finite time interval has a natural extension to the entire time axis. Definition 62. The time-reversal of an ergodic CTMC X(t), with transi¯ with initial tion rates qij and stationary distribution p, is a CTMC X(t) distribution p and transition rates q¯ij = p−1 i pj qji . ¯ The finite-dimensional distributions of X(t) are the reverse-time of those for X(t) as depicted by property (4.39) below. Proposition 63. If the CTMC X(t) is stationary with distribution p, then ¯ its time-reversal X(t) is stationary with distribution p and d ¯ 1 ), . . . , X(t ¯ n )) = (X(t (X(τ − t1 ), . . . , X(τ − tn )),

t1 < . . . < tn ≤ τ. (4.39)

Proof. By Remark 19, two CTMCs are equal in distribution if and only if their initial distributions and transition rates are equal. Then by the descrip¯ tion of X τ (t) in Lemma 61 when X(t) is stationary, it follows that X(t) and

4.12 Modeling of Reversible Phenomena

277

¯ X τ (t) on [0, τ ] are equal in distribution. This proves (4.39). Also, X(t) being stationary on [0, τ ], for each τ , implies that it is stationary on [0, ∞). The CTMC X(t) is defined to be reversible in time if it is equal in distri¯ bution to its time-reversal process X(t); that is (in light of (4.39)), d

(X(t1 ), . . . , X(tn )) = (X(τ − t1 ), . . . , X(τ − tn )),

t1 < . . . < tn ≤ τ. (4.40)

How is this time-reversibility related to the (algebraic) reversibility we have been discussing? Proposition 64. The CTMC X(t) is reversible in time if and only if it is stationary and reversible. Proof. We will use the property that two CTMCs are equal in distribution if and only if their initial distributions and transition rates are equal. Suppose ¯ X(t) is reversible in time. Then the transition rates q¯ij = p−1 i pj qji of X(t) are equal to the rates qij of X(t), which implies pi qij = pj qji . Thus, X(t) is d

reversible. In addition, (4.40) implies X(0) = X(τ ) for each τ , and so X(t) is stationary by Proposition 33. Conversely, suppose X(t) is stationary and reversible and its distribution ¯ is p. Then by Proposition 63, X(t) is a stationary CTMC with distribution p. Also, the reversibility supposition implies q¯ij = p−1 i pj qji = qij . ¯ Thus, X(t) and X(t) are equal in distribution, and hence X(t) is reversible in time. Remark 65. For a CTMC {X(t) : t ∈ R} on the entire time axis R, the time-reversibility definition (4.40) is equivalent to d

(X(t1 ), . . . , X(tn )) = (X(tn ), . . . , X(t1 )),

t1 < . . . < tn in R.

4.12 Modeling of Reversible Phenomena There are a surprising number of complex systems that can be modeled by functions of reversible processes or by reversible transition rates. This section illustrates this with a few key principles of reversibility that help one identify or construct reversible processes. For this discussion, X(t) will denote a CTMC on S. Since many interesting models are in multi-dimensional spaces, we adopt slightly different notation and let q(x, y), for x, y ∈ S, denote its transition rates, and let p(x, y), q(x) and γ(x) denote the usual transition probabilities, sojourn rates and invariant

278

4 Continuous-Time Markov Chains

measure. For simplicity, assume that all the transition rates in this section are irreducible and recurrent, and hence they have positive invariant measures. The examples in this section are based on elementary properties of reversibility that follow directly from the definition. We begin with two properties that are often applied together. Proposition 66. (State Space Truncation) Let q(x, y) be a transition rate function on S˜ that is reversible with respect to γ. If X(t) is a CTMC on a subset S ⊂ S˜ with transition rates q(x, y) for x, y ∈ S, then X(t) is reversible with respect to γ restricted to S. In other words, a reversible process restricted to a subset of its state space is also reversible. The next observation is that a vector-valued CTMC of independent reversible components is reversible. The first assertion uses the fact that two independent CTMCs cannot have a common jump time; this is due in part to Remark 20 that says the probability is 0 that a CTMC has a jump at a fixed time. Proposition 67. (Juxtaposition of Processes) If X1 (t), . . . , Xm (t) are independent CTMCs on spaces S1 , . . . , Sm , then X(t) = (X1 (t), . . . , Xm (t)) on S = S1 × · · · × Sm is a CTMC with transition rates q(x, y) = qi (xi , yi )

if xj = yj , j = i, for some i.

where x = (x1 , . . . , xm ) and qi (xi , yi ) are the transition rates for Xi (t). If in addition each Xi (t) is reversible with respect to γi , then X(t) is reversible with respect to γ(x) = γ1 (x1 ) · · · γm (xm ). Reversible processes arise in many contexts where several independent reversible processes are linked together by certain interactions or constraints. This is exemplified by the following application of the preceding results. Example 68. Birth-Death and Queueing Processes: Dynamic Population Constraints. Consider a birth-death process (e.g., an M/M/s queueing process) with invariant measure γ(x) as in Example 60. We will discuss two variations of this process in which the quantity of items allowed in the system is bounded. These apply naturally to M/M/s queues in which the waiting space for items is limited. First, assume the allowable number of items in the population is a fixed constant m, so that there are no births when the system is full (e.g., arrivals to a queue are turned away). Let X(t) denote the quantity of items in the population at time t. By Proposition 66, it follows that X(t) is a reversible CTMC on S = {0, 1, . . . , m} with respect to the invariant measure γ of the unconstrained process restricted to S. In particular, if m = s, there is no queueing, and the limiting probability of a full system (the Erlang Loss Probability) is (λ/μ)s /s! ps = s . i i=0 (λ/μ) /i!

4.12 Modeling of Reversible Phenomena

279

Next, consider the more complicated situation in which the number of items in the system at time t, denoted by X(t), cannot exceed a value Y (t). Assume that Y (t) operates as a reversible Markov process on a subset of Z+ with transition rates qY (y, y  ) and stationary distribution pY (y), but it is constrained by the inequality X(t) ≤ Y (t). That is, whenever X(t) = Y (t), there are no births; also, transitions of Y (t) below X(t) are not allowed. More precisely, assume that (X(t), Y (t)) is an irreducible CTMC on the space S = {(x, y) ∈ Z2+ : x ≤ y}, and its transition rates are q((x, y), (x , y  )) = qX (x, x )1(y  = y, x ≤ y) +qY (y, y  )1(x = x, y  ≥ x).

(4.41)

Here qX (x, x ) is the transition rate function for the unrestricted birth-death process on the nonnegative integers. Since qX (x, x ) and qY (y, y  ) are reversible, the transition rates (4.41) without the inequality constraints x ≤ y are as in Proposition 67. Consequently, (4.41) without the inequality constraints is reversible with respect to the product measure γ(x)pY (y). Thus, by Proposition 66, the (X(t), Y (t)) is reversible with respect to p(x, y) = γ(x)pY (y). The next observation is that a transition rate function is reversible if it is a multiplication or compounding of reversible transition functions. This also follows directly from the definition of reversibility. Proposition 69. (Compound Transition Rates) Suppose the transition rates of a CTMC X(t) are of the form q(x, y) = q1 (x, y) · · · qm (x, y),

x = y ∈ S,

where qi (x, y) is a transition rate function on S, for 1 ≤ i ≤ m. If each qi (x, y) is reversible with invariant measure γi , then X(t) is invariant with respect to γ(x) = γ1 (x1 ) · · · γm (xm ). Using this property, the truncations of states in Proposition 66 has the following extension to modifications as well as truncations of transitions. Example 70. Transition Modifications. Consider a CTMC on S˜ whose transition rates q˜(x, y) are modified by multiplying it by a transition rate function ˜ The resulting process X(t) is a CTMC on S with r(x, y) on a subset S ⊆ S. transition rates q(x, y) = r(x, y)˜ q (x, y), x, y ∈ S. Suppose q˜(x, y) is reversible on S˜ with respect to a positive γ˜ . Then X(t) is reversible if and only if r(x, y) is reversible. In this case, an invariant measure for X(t) is γ(x) = ρ(x)˜ γ (x), where ρ is an invariant measure for r(x, y). This follows from Proposition 69 and the definition of reversibility. For instance, the function r(x, y) might simply be symmetric (r(x, y) = r(y, x), x, y ∈ S).

280

4 Continuous-Time Markov Chains

There are many rate-modification functions r(x, y) on a space S˜ ⊆ Rm +, such as r(x, y) = 1(x ≤ y) + 1(x ≥ y) (transitions are either up or down), r(x, y) = 1(max{|yj − xj |} ≤ b) (component differences are ≤ b). j

Here are three more illustrations. Example 71. Jumps Affected by a Random Environment. Suppose the transition rate function q˜(x, y) represents a CTMC on S˜ that is subject to a random environment that affects its jumps as follows. Whenever the process is in state x, a jump to a new state y is allowed with probability r(x, y) and is not allowed with positive probability 1 − r(x, y). This jump modification is independent of everything else. The resulting system process X(t) is a CTMC as in Example 70. Example 72. Resource Constraints. Consider Example 70, where q˜(x, y) represents a system on S˜ ⊆ Rm + that requires certain resources to sustain it. In particular, assume that whenever it is in a state x = (x1 , . . . , xm ), it requires a quantity aij xj of a resource i ∈ I for each component j, and bi is themaximum of the resource i that is available. Then setting r(x, y) = 1( j aij yj ≤ bi , i ∈ I) constrains the system to new states y that do not exceed the resources. Example 73. Communication Network with Capacity Constraints and Blocking. Consider a communication network, as introduced in [21], that services m types of items. The items arrive to the network according to independent Poisson processes with respective rates λ1 , . . . , λm . For its communication across the network, each type j unit requires the simultaneous use of aij channels on link i for each i in the set I of links of the network. Some of the aij may be 0. If these quantities of channels are available, they are assigned to the item, and the item holds the channels for a time that is exponentially distributed with rate μj . At the end of this time, the item releases the channels and exits the network. The total number of channels available on link i ∈ I is bi . If an item arrives and its required channel quantities are not available, then it cannot enter the network (it is blocked from entering or lost). Let X(t) = (X1 (t), . . . , Xm (t)) denote the numbers of the m types of items in the network at time t. WhenX(t) is in state x = (x1 , . . . , xm ), the number of channels in use on link i is j aij xj . Then the state space of X(t) is S = {x : 0 ≤



aij xj ≤ bi , i ∈ I}.

j

Note that if the state of the process is x, then a type j item cannot enter the network when

4.12 Modeling of Reversible Phenomena

x ∈ Bj = {x :



281

aik xk > bi − aij , for some i ∈ I}.

k

Without these constraints, one can view the Xi (t) as independent birthdeath processes (Example 59) with respective birth and death rates λi and μi , for 1 ≤ i ≤ m. Under these constraints, by Propositions 66 and 67, X(t) is a multivariate birth–death process with single-unit movements that is constrained to be in S, and its stationary distribution is m 

p(x) = c

(λj−1 /μj )xj ,

x ∈ S,

j=1

where c is the normalization constant. Many network performance parameters can be expressed in terms of this distribution. Of prime importance is the probability that an item is blocked from entering the network.  The probability that a type j arrival is blocked in equilibrium is p(Bj ) = x∈Bj p(x). Ideally, the channel capacities bi would be sized such that this blocking probability would be less than some small amount such as .01. One may also be interested in which links cause the blocking. For instance, theprobability that  a type j item is blocked because the load on link i is full is x∈Bj p(x)1( k aik xk > bi ). What is the average number of! type j items blocked per unit time? To t determine this, consider τj (t) = 0 1(X(s) ∈ Bj ) ds, which is the amount of time in [0, t] that type j items are blocked. Now, the number of type j items blocked in [0, t] can be expressed as Nj (τj (t)), where Nj (t) is a Poisson process with rate λj that is independent of X(t). Thus, by the strong law of large numbers for Nj (t) and for τj (t), the number of type j items blocked per unit time is lim t−1 Nj (τj (t)) = lim τj (t)−1 Nj (τj (t))τj (t)/t = λj p(Bj ) a.s..

t→∞

t→∞

A process forassessing loads on the network is Y (t) = (Y i (t) : i ∈ I), where Y i (t) = j aij X j (t) is the number of channels on link i that are in use at time t. Although this process Y (t) is not Markovian, its stationary distribution, as a function of p, is   p(x)1( aij xj = yi , i ∈ I). pY (y) = x∈S

j

This distribution can be used to determine various performance parameters such as the percent of time that link i is idle or the stationary probability that link i has more channels in use than link k. Another parameter of interest is   the average number of channels in use on link i, which is j aij x∈S xj p(x).

282

4 Continuous-Time Markov Chains

4.13 Jackson Network Processes In Chapter 1, we introduced Jackson network processes in discrete time as examples of Markov chains. These models are appropriate when arrivals to the network and service completions occur exactly at discrete times. On the other hand, in many communications and industrial processing networks, arrivals typically occur at times that form Poisson processes or point processes in continuous time, such as those that can be approximated by Poisson processes as in Chapter 3. In addition, the processing times may be more realistic as continuous random variables. In these cases, it is appropriate to use continuous-time models. An important family of such models are Jackson network processes that are CTMCs. This section describes the equilibrium behavior of these Jackson network processes, and the next section describes a related family of network processes. The network terminology here will be somewhat like that in Section 1.15. Consider an m-node network in which discrete items move among the nodes. The state of the network at time t will be represented by a CTMC X(t) = (X1 (t), . . . , Xm (t)), where Xi (t) denotes the number of items at node i. The state space S is a set of vectors x = (x1 , . . . , xm ) with nonnegativeinteger m entries, and q(x, y) denotes the transition rates. We also use |x| = i=1 xi . Typical nodes will be labeled i, j, k, . . . The network may be any one of the following types. • Closed network with ν items and S = {x : |x| = ν}. • Open network with unlimited capacity and S = {x : |x| < ∞}. • Open network with finite capacity ν and S = {x : |x| ≤ ν}. Think of the items as moving in the node set  {1, . . . , m} if the network is closed M= {0, 1, . . . , m} if the network is open. A typical transition is triggered by the movement of a single item. When the network is in state x and one item moves from some i to j in M, the network has a transition x → Tij x = x − ei + ej . Here ei is the m-dimensional vector with 1 in position i and 0 elsewhere, and e0 is the zero vector. Most of the following development applies to any of the preceding types of networks; we designate the network type when the distinction is needed. The movement of items and the resulting transition rates of X(t) are conveniently described by clock times as in Example 9. Specifically, assume that whenever the network is in state x, the time to the next movement of a single item from node i to node j, resulting in a transition x → Tij x, is exponentially distributed with rate λij φi (xi ), where φ0 (·) = 1. The λij are

4.13 Jackson Network Processes

283

nonnegative with each λii = 0 (this is for convenience — Exercise 39 shows how it can be relaxed). The φi (xi ) is positive except that φi (xi ) = 0 if xi = 0 and i = 0. Under this assumption of independent exponential times to item movements, the network process X(t) is a CTMC with transition rates  λij φi (xi ) if y = Tij x for some j = i in M q(x, y) = 0 otherwise. We call X(t) a Jackson network process. Before describing its behavior, a few comments are in order. The exponential sojourn time of X(t) in state x has the rate   q(x) = φi (xi ) λij , i

j

and the one-step transition probability is p(x, y) = q(x, y)/q(x). Whenever the process is in state x and an item moves out of node i, the probability that the item moves to node j is   p(x, Tik x) = λij / λik , i, j ∈ M. pij = p(x, Tij x)/ k

k

Since this probability is independent of x, one can view the items as being independently routed at each transition via the Markov chain routing probabilities pij . We follow  the standard convention that λij may either be a routing probability (with k∈M λik = 1) or a nonnegative intensity of selecting the nodes i and j, and call it the i-to-j routing rate. For an open, unlimited-capacity network, the routing assumption implies that items enter the network at the nodes according to independent Poisson processes with respective rates λ0j (this is proved in Example 95); and λ0j = 0 means that node j does not have arrivals from outside. Think of λij as the transition rates of a CTMC that determine the movement of a single item in the node set M (which includes 0 when the network is open), and pij are its one-step transition probabilities. The φi (xi ) is the service rate or departure intensity at node i when the network state is x. Since this rate does not depend on xk for k = i, the nodes are said to “operate” independently. For instance, if node i operates like an M/M/s system with s independent servers that serve at the rate μi , then φi (xi ) = μi min{xi , s}1(xi ≥ 1). Another interpretation is that φj (xi ) represents an egalitarian processorsharing scheme in which at any instant, the time to the next “potential” departure of any particular item at the node is exponentially distributed with rate φi (xi )/xi , independent of the other items. That is, node i works on the each item with this rate, and all the items receive service simultaneously.

284

4 Continuous-Time Markov Chains

These service and routing rates may have other interpretations for general movements of items or particles. In a transition x → Tij x, we refer to a “single item” moving from i to j. However, more than one item may actually move in the transition, as long as the node populations before and after the transition are x and Tij x, respectively. For instance, in a manufacturing network, a part exiting a certain node i may be considered as a completed part that actually exits the network and triggers another item outside the network to take its place and enter node j. We are now ready to describe the equilibrium behavior of the three types of Jackson processes we have been discussing. With no loss in generality, we assume the routing rates λij are irreducible. This is equivalent to X(t) being irreducible; see Exercise 37. Let wi , i ∈ M, denote a positive invariant measure that satisfies the routing balance equations or traffic equations   λij = wj λji , i ∈ M. (4.42) wi j∈M

j∈M

To simplify some expressions, we assume w0 = 1 when the network is open.7 Theorem 74. If X(t) is a closed Jackson process with ν items, then it is ergodic and its stationary distribution is p(x) = cf1 (x1 ) · · · fm (xm ), x ∈ S = {x : |x| = ν}, (4.43)  i where fi (xi ) = wixi xk=1 φi (k)−1 and the wi satisfy (4.42). The normalization constant is  −1 c= f1 (x1 ) · · · fm (xm ) . x∈S

Theorem 75. If X(t) is an open Jackson process with finite capacity ν, then the assertions of Theorem 74 apply to this process with S = {x : |x| ≤ ν}. Theorem 76. If X(t) is an open Jackson process with unlimited capacity, then it has an invariant measure of the form (4.43) with S = {x : |x| < ∞}. Hence, the process is ergodic if and only if = c−1 i

∞ 

fi (k) < ∞,

1 ≤ i ≤ m.

k=0

In this case, its stationary distribution is p(x) = p1 (x1 ) · · · pm (xm ), where pi (xi ) = ci fi (xi ). 7

Recall the convention that

0

k=1 (·)

= 1.

x ∈ S,

4.13 Jackson Network Processes

285

Proof. To prove these theorems, it suffices to show that p given by (4.43) satisfies the balance equations, which in this case are   q(x, Tij x) = p(Tji x)q(Tji x, x), x ∈ S. (4.44) p(x) i,j∈M

i,j∈M

This proof by substitution is comparable to the proof of Theorem 86 in Chapter 1 for the closed network in discrete time. The substitution is the same for each type of state space. The other details in Theorem 76 are obvious. Remark 77. Product-Form Distributions. The stationary distributions in the three preceding results have a product form, but only the last one for an open unlimited-capacity network represents a product of probabilities for independent random variables. In this case, if one were to take a snapshot of the node quantities X1 (t), . . . , Xm (t) at a fixed time t, they would be independent random variables with distributions as in Theorem 76. Of course, these node quantities at different times are not independent. Remark 78. Partial Balance. The stationary distribution p in the preceding results, which satisfies the total balance equations (4.44), also satisfies the partial-balance equations   p(x) q(x, Tij x) = p(Tji x)q(Tji x, x), i ∈ S, x ∈ S. j∈M

j∈M

Summing these equations on i yields the total balance equations. From the law of large numbers for CTMCs, the partial balance equations say that the average number of items departing from node i per unit time when X(t) is in state x equals the average number of items entering node i per unit time that land X(t) in state x. Or, loosely speaking, the equilibrium flow of items out of node i, for any state x, equals the flow into i. Because of this, the equations are also called station balance equations. Remark 79. Traffic Equations. Although the traffic equations (4.42) precede the theorems and are ostensively an assumption, they are also a consequence of the results. Namely, upon substituting the measure p given by (4.43) in the preceding partial balance equations, the service-rate functions cancel and the traffic equations are what is left. In other words, the traffic equations are a necessary and sufficient condition for p to satisfy the partial balance equations. The modeling of a network by a Jackson process typically involves specifying the nodes and service actions, verifying the assumption of an exponential time to a transition and identifying the routing and service rates. The next step is to solve the routing equations for the parameters wi , the only unknown parameters for the stationary distribution. With the stationary distribution in hand, one can then derive various network performance parameters.

286

4 Continuous-Time Markov Chains

The following model of a maintenance network is similar to those in industrial and military settings for maintaining expensive equipment to produce goods or services or to perform a mission. Example 80. Production–Maintenance Network. Consider a system shown in Figure 4.1 in which ν machines (subsystems, trucks or electronic equipment) are available for use at a facility or location called node 1. At most s1 machines can be in use at node 1 at any time for producing goods or services. Therefore, if x1 machines are present then min{x1 , s1 } of these will be in use. After a machine is put into use, it operates continuously until it fails or degrades to a point that it requires a repair. The total operating time is exponentially distributed with rate μ1 . At the end of this time, the machine is transported to a repair facility. The transportation system (which may involve initial processing and rail or air travel) is called node 2, and the machine’s time at this node is exponentially distributed with rate μ2 ; there is no queueing for the transportation. The repair facility consists of nodes 3, 4, 5, which are single-server nodes with respective rates μ3 , μ4 , μ5 . Depending on the nature of the repair, the machine goes to one of these nodes with respective probabilities p23 , p24 , p25 . After its repair, the machine goes to another transportation system, called node 6, for an exponentially distributed time with rate μ6 . And then it enters node 1 to begin another production/repair cycle.

6 6

- 1m

- 2m ?

- 3m

-

- 4m

? - 6m

- 5m

6 -

-



?

Fig. 4.1 Production–Maintenance Network

Let X(t) denote the process representing the numbers of machines at the six nodes at time t. Under the preceding assumptions, X(t) is a closed Jackson process in which each node i is an si -server node, where s2 = s6 = ∞,

s3 = s4 = s5 = 1.

The service rate of each server at node i is μi . The routing intensities are the routing probabilities λij = 1,

(i, j) ∈ {(1, 2), (3, 6), (4, 6), (5, 6), (6, 1)},

λ2j = p2j ,

j = 3, 4, 5.

the other λij ’s are 0. A solution of the traffic equations (4.42) for these routing probabilities is wi = 1 for i = 1, 2, 6 and wi = p2i for i = 3, 4, 5. Then by Theorem 74, the stationary distribution of X(t) is

4.14 Multiclass Networks

p(x) = c

287

x1 6  1  1 (wi /μi )xi , x2 !x6 ! min{k, s1 } i=1

x ∈ S,

k=1

where c is the normalization constant. The quality of this maintenance system is measured by the number of machines in productive use at node 1. Suppose the aim is to find the number of machines ν ∗ to provision for the network such that the probability of having less than x ¯1 machines in use at node 1 in equilibrium is below β (e.g., .10). From the stationary distribution above, it follows that the equilibrium probability of having less than x ¯1 machines at node 1 (as a function of ν) is  p(x)1(xi < x¯1 ) α(ν) = =c

x x ¯ 1 −1 n=0

6 6   1  1 1( x = ν − n) (wj /μj )xj . j μn1 n! x ,...,x x !x ! 2 6 j=2 j=2 2

6

Then the provisioning quantity ν ∗ = min{ν : h(ν) ≤ β} is obtained by computing α(ν) for ν = x¯1 , x ¯1 + 1 . . . until it falls below β.

4.14 Multiclass Networks This section shows how the Jackson network models with homogeneous items extend to networks with multiple types of items, where the routing and services may depend on an item’s type. The difference is that we now keep track of the number of items of each type at a node and envision each item as moving within a set of class-node indices. We will describe the equilibrium behavior of two types of multiclass networks. The first model is for an open network in which each item chooses a particular route through the network, and the item’s class label at any instant is designated by the route it is traversing and its stage on the route. Consider an open m-node network that processes items that travel through it as follows. A typical route of an item is a finite sequence r = (r1 , . . . , r ) of nodes inside the network, where rs is the node the item visits at stage s of its route, 1 ≤ s ≤ ; the length of the route  = (r) depends on the route. Upon leaving the last node r , the item exits the network. A node may appear more than once on a route, and the set of all relevant routes, for simplicity, is finite. Items that traverse a route r arrive to the network according to a Poisson process with rate λ(r), and these arrival processes are independent for all the routes.  Then the total arrival stream to the network is a Poisson process with rate r λ(r). The preceding description applies to several scenarios. One is that a deterministic route r is an attribute of an item and all items that traverse

288

4 Continuous-Time Markov Chains

a given route are in the same class. A second scenario is that each item carries a permanent class label that determines its route. A third possibility is that deterministic routes are obtained by random routes as follows. The items arrive to the network by a Poisson process with rate λ, and each item independently selects or is assigned a route r with probability p(r). In this case, λ(r) = p(r)λ. For instance, a route may be selected by Markov probabilities pjk such that p0r1 pr1 r2 · · · pr−1 r is the probability of the route r = (r1 , . . . , r ). Combinations of the preceding scenarios yield further possibilities. To represent the network by a multiclass network process, we assign a class label to each item to denote its routing status at any time in the network. Namely, if an item is traversing route r and is at stage s in this route, we call it a rs-item (which of course resides at node rs ). Let M denote the set of all route-stage labels rs, including the outside node 0 as well. Then the possible states of the network are given by the state space S = {x = (xrs : rs ∈ M\{0})}, where xrs denotes the number of rs-items in the network at node rs . Like a Jackson network, assume that whenever the network is in state x, the time to the next departure of an rs-item from its current node location rs is exponentially distributed with rate φrs (x), independent of everything else. The departing item goes immediately to its next node rs+1 and becomes an r(s + 1)-item. In case s = , the r+1 = 0, which means that the route is complete and the item exits the network. It is informative to think of the items as moving in the route-stage set M as well as among the nodes 0, 1, . . . , m. Let {X(t) : t ≥ 0} denote the stochastic process representing the state of the network in the space S. Under the preceding assumptions, X(t) is a CTMC with transition rates ⎧ ⎪ if y = x + er1 for some r ⎨λ(r) q(x, y) = φrs (x) if y = x − ers + er(s+1) for some rs ∈ M ⎪ ⎩ 0 otherwise. Here ers is the unit vector  with 1 in component rs and 0 elsewhere, and e0 = 0. Note that xi = rs xrs 1(rs = i) is the number of items at node i Assume that each node is a processor-sharing node with service rates φrs (x) =

xrs μi (xi ), xi

where i = rs . Here μi (xi ) > 0 is a load-dependent service rate for the xi items at node i that is apportioned equally among the xrs rs-items at the node. The CTMC X(t) is a multiclass Kelly network process [67] with processor-sharing nodes [67].

4.14 Multiclass Networks

289

Theorem 81. The Kelly network process has an invariant measure  λ(r)νr frs (x), x ∈ S, p(x) = rs∈M

where νr =

 s=1

xrs (the number of items on route r) and frs (x) =

m 1  xi ! . xrs ! i=1 μi (1) · · · μi (xi )

Proof. The rates at which the items move in the set M are λ0,r1 = λ(r),

λrs,r(s+1) = 1,

rs = 0.

The traffic equations for these routing rates are simply w0 = 1 and, for each route r, wr1 = λ(r), wrs = wr(s−1) , s = 2, . . . , . A solution to these equations is wrs = λ(r) for each rs = 0. Then like an open Jackson network with unlimited capacity, X(t) is irreducible and one can show by substitution that p(x) satisfies the balance equations and hence it is an invariant measure for the process. We will now consider a network model in which each item carries an attribute or class label from a finite set. An item’s class is a distinguishing characteristic that determines its routing or service rates; the route-stage class label in the preceding model is an example. The class label may be permanent, or temporary and subject to change as the item moves. Examples of permanent labels are: • The size of an item when it is a batch of subunits such as data packets, orders to be filled, or capacity of a circuit. • The type of part or tool in a manufacturing network. • The origin or destination of a item. • The general direction in which an item moves through the network (e.g., north to south). Examples of temporary labels are: • The status of a part as it is being produced. • The number of nodes an item has visited. • The number of times an item has been fed back to the node where it resides. • The phase of service that an item is undergoing, when it has a phase-type distribution. Our focus will be on a network in which an item’s service rate at a node is a compounding of two intensities — one intensity is a function of the total number of items at the node, and the other intensity is a function of the number of items in the same class as the one being served. The other operating rules of the network are similar to those for Jackson networks.

290

4 Continuous-Time Markov Chains

In particular, consider an open m-node network that processes discrete items whose class at any instant is designated by an abstract label α, and xαi denotes the number of α-items at node i. Here αi is in a set M of class node pairs, including α0 for the outside node 0. Envision the items moving within the set M. The evolution of the network is represented by a CTMC {X(t) : t ≥ 0} with state space S = {x = (xαi : αi ∈ M, i = 0)}.  The number of items at node i is xi = α xαi . Whenever the process is in a state x, a typical transition consists of an α-item departing from node i and moving instantaneously into a node j and entering there as a β-item. We denote this transition by x → x − eαi + eβj , where eαi denotes the unit vector with a 1 in component αi and 0 elsewhere, and eα0 = 0. It is allowable that i = j or α = β, provided αi = βj. Assume the transition rates of the process X(t) are of the form  λαi,βj φαi (x) if y = x − eαi + eβj for some αi = βj in M q(x, y) = 0 otherwise. The φαi (·) are service rate functions or intensities, and λαi,βj are routing rates or intensities. We will consider service rates of the form φα0 (x) = 1 and φαi (x) = gαi (xi )hαi (xαj ),

i = 0,

(4.45)

where gαi (xi ) is the node intensity and hαi (xαi ) is the class intensity. The routing rates λαj,βk may be reducible, but they do not contain transient states. It is clear that the structure for this multiclass network X(t) is basically the same as that of a Jackson network. Consequently, the theory of Jackson networks also applies to this multiclass analogue —one just replaces the single-node subscripts i with a double subscript αi. In particular, an invariant measure for it is as follows. Theorem 82. For the multiclass network process X(t) described above, an invariant measure is  xαi p(x) = wαi fαi (x), x ∈ S, αi∈M

where fα0 (x) = 1, fαi (x) =

xi  k=1

−1

gαi (k)

xαi 

hαi (k  )−1 ,

i = 0,

k =1

and wαi are positive numbers, with wα0 = 1, that satisfy the traffic equations

4.15 Poisson Transition Times

wαi



291

λαi,βj =

βj∈M



wβj λβj,αi ,

αi ∈ M.

(4.46)

βj∈M

Note that a Kelly network process (with rs replaced by αi) is a special case of this multiclass network. Several other examples are discussed in [101, 111]; here is another one. Example 83. The multiclass network described above is a BCMP network, introduced by Baskett, Chandy, Muntz, and Palacios in [9], if each of its nodes is one of the following four types. • First-Come, First-Served node with service rates φαi (x) = μi (xi ). Each item (as in a Jackson network) has an exponential service time with the same load-dependent service rate μi (xi ). • Processor-Sharing node with service rates φαi (x) = xαi x−1 i μαi (xi ). The μαi (xαi ) is a customer-load-dependent service rate, which is apportioned equally among the xαi α-items at the node. • Last-Come, First-Served with Preemption node with service rates as in the preceding PS case. • Infinite-Server node with service rates φαi (x) = xαi μαi (x αi ). xαi An invariant measure for this BCMP network is p(x) = αi∈M wαi fαi (x) as in Theorem 82, where fα0 (x) = 1 and the other functions are fαi (x) = fαi (x) =

1 μi (1) · · · μi (xi )

1 xi ! xαi ! μαi (1) · · · μαi (xi )

fαi (x) =

1 1 xαi ! μαi (1) · · · μαi (xαi )

FCFS PS or LCFSPR IS.

This completes our discussion of networks.

4.15 Poisson Transition Times In the next five sections, we will discuss the behavior of ergodic CTMCs at their transition times. This section presents criteria under which the times of a certain type of transition of a CTMC form a Poisson process. For instance, in a stationary birth-death process, the times at which deaths occur form a Poisson process, and in a Jackson network process, the times of departures from certain nodes form independent Poisson processes. Probabilities of a CTMC at these special transition times that form a Poisson process will be the focus of the following four sections.

292

4 Continuous-Time Markov Chains

For this discussion,8 X = {X(t) : t ∈ R} will denote an ergodic CTMC with transition rates qij and stationary distribution p. We consider X on the entire time axis since some of the results here are more natural for stationary processes. Its transition times are depicted by the point process  N (B) = 1(Tn ∈ B), B ∈ B, n∈Z

where the transition times Tn are labeled such that . . . < T−2 < T−1 < T0 ≤ 0 < T1 < T2 . . . . ! By  Example 50, E[N (B)] = B E[qX(t) ] dt. This is finite for bounded B if i qi < ∞. The point process N of all the transition times is usually not a Poisson process, although it is for a Markov chain subordinated to a Poisson process. However, an important feature of a CTMC is that certain types of its transition times may be Poisson processes. The results that follow give necessary and sufficient conditions for such subsets of transition times of N to be Poisson processes. We begin by defining what is a “special” transition of the CTMC X. It is natural to describe these transitions in terms of its sample paths. Let D denote the set of all functions x : R → S that are right-continuous with lefthand limits, and are piece-wise constant with a finite number of jumps in any finite time interval.9 Recall that a typical sample path of X is a function in D. In other words, the process X is a D-valued random variable (or a random element in D). Accordingly, we assume the σ-field associated with D is the smallest σ-field D under which, for each t, the projection map x → x(t) is measurable: this ensures that each X(t) is a well-defined S-valued random variable. We will also refer to St X = {X(t + u) : u ∈ R},

t ∈ R,

which is the process X with its time parameter shifted10 by the amount t. The St X is what an observer sees of the path at time t. It is natural to describe a certain “transition” of X at a time t when St X is in a set T in D. Definition 84. Suppose T ∈ D is such that 8

We will now use the shorthand notation X for the CTMC instead of X(t) as in the preceding sections.

9

In Chapter 5, D = D[0, 1] denotes the space of functions on [0, 1] that are right-continuous with left-hand limits, without the restrictions that they be piece-wise constant or take only a finite number of jumps in a finite interval. 10 The St is the time-shift operator on D. The shift notation S and the state-space notation S are different.

4.15 Poisson Transition Times

293

x(0−) = x(0),

x∈T.

(4.47)

We say that a T -transition of X occurs at time t if St X ∈ T . The times at which these T -transitions occur are depicted by the point process NT on R defined by   NT (B) = 1(St X ∈ T ) = 1(STn X ∈ T , Tn ∈ B), B ∈ B. t∈B

n∈Z

These quantities are finite on bounded time sets B, since X can only take a finite number of jumps in such sets. Note that if a T -transition occurs at time t (St X ∈ T ), then condition (4.47) ensures that X(t−) = X(t), which implies that t is a jump time of X. Example 85. Suppose X(t) denotes the value of a stock at time t. Then the times at which the value increases are T -transitions, where T = {x ∈ D : x(0) > x(0−)}. The times between these transitions are the times between increases in the stock value. This type of transition only involves values of X before and after the transition; the past and future are not involved. An example of a transition time that involves more sample-path information is a time at which the stock value jumps into an interval (a, b) and thereafter the stock value reaches b before it reaches the value a. In this case, T = {x ∈ D : x(0−) ∈ / (a, b), x(0) ∈ (a, b), hb (x) < ha (x)}, where ha (x) = inf{t > 0 : x(t) ≤ a} is the time at which the path x hits (−∞, a] starting from time 0, and hb (x) is defined similarly. The time between such transitions is the time for the stock value X to reach [b, ∞) given that it does not reach (−∞, a]. We begin with some elementary observations about the point process NT of T -transitions. Its mean measure, which follows by the generalized L´evy formula in Theorem 52 and Remark 55, is  E[NT (B)] = E[αT (X(t))] dt, B ∈ B, B

where αT (i) =



qij P {X ∈ T |X−1 = i, X0 = j},

i ∈ S.

(4.48)

j=i

The quantity αT (i) is the infinitesimal rate at which a T -transition is initiated from state i. Clearly, NT (B) ≤ N (B), and so E[NT (B)] ≤ E[N (B)].

294

4 Continuous-Time Markov Chains

The next result establishes that NT is stationary when X is. However, NT may be stationary when X is not, which is the case in some examples below. Proposition 86. If X is stationary, then the point process NT of T -transitions of X is stationary with rate  pi αT (i), λT = E[NT (0, 1]] = which is finite when

 i

i

pi qi < ∞.

Proof. From Section 2.15, NT is stationary if, for any B1 , . . . , Bm ∈ B and n 1 , . . . , nm , P {NT (B1 + t) = n1 , . . . , NT (Bm + t) = nm } is the same for each t. (4.49) It suffices for the Bk to be bounded sets, since the distribution of NT is determined by its finite-dimensional distributions on bounded sets. For a bounded B, using the change-of-variable v = u − t,   1(Su X ∈ T ) = 1(Sv (St X) ∈ T ). NT (B + t) = u∈B+t

v∈B

d

Then the assumption St X = X implies (4.49) for m = 1. A similar argument justifies (4.49) for any m, and hence NT is stationary. Our criteria for Poisson T -transitions involve the following concepts. Definition 87. A T -transition has a uniform initiation rate λT if 

αT (i) = λT ,

for each i ∈ S,

where αT (i) = j=i qij P {X ∈ T |X−1 = i, X0 = j} is the infinitesimal initiation rate of a T -transition from state i. Definition 88. The future of NT is independent of the past of X if, for each t ∈ R, the future quantities {NT (B) : B ⊂ (t, ∞)} are independent of {X(s) : s ≤ t}. Here is our first criterion for NT to be Poisson. Theorem 89. The following statements are equivalent. (a) NT is a Poisson process with rate λT and the future of NT is independent of the past of X. (b) T has a uniform initiation rate λT . Proof. First, suppose (b) is true. We will prove (a) for the case in which the sojourn rates qi are bounded. A standard proof in [101] for unbounded rates involves a martingale characterization of Poisson processes, which we do not cover.

4.15 Poisson Transition Times

295

By the uniformization principle in Proposition 25, the CTMC X is equal ˆ = {X(t) ˆ in distribution to a CTMC X : t ∈ R} — a subordinated Markov chain — with transition probabilities pˆij = qij /λ,

pˆii = 1 − qi /λ,

ˆ of its transition times is a Poisson where λ = supi qi . The point process N ˆ n , n ∈ Z. Let N ˆT process with rate λ independent of the embedded states X ˆ denote the point process of T -transitions for X. ˆ are equal in distribution, any deterministic function of one Since X and X of them is equal in distribution to the function of the other one. In particular, ˆT are equal in distribution. Then to prove (a), it suffices to prove NT and N ˆ ˆ (a) for NT and X. ˆ n and To this end, note that by the independence of the Markov chain X ˆ ˆ ˆ the Poisson process N , it follows that NT is a thinning of N , where the ˆ being retained is probability of a point of N   ¯ ˆ ˆ n−1 = i} ˆ ∈ T |X−1 = i, X0 = j} P {STn X P {X pˆij P {X ∈T}= j=i

i

=



ˆ n−1 = i}αT (i)/λ = λT /λ. P {X

i

Here we use pˆij = qij /λ and the uniform initiation rate assumption. This assumption also implies that a T -transition can be triggered from any state. Thus by the Poisson thinning principle (Example 40 in Chapter 3), it follows ˆT is a Poisson process with rate λT . that N ˆ is independent of the In addition, note that since the Poisson process N ˆ ˆ Markov chain Xn , and N has independent increments, it follows that the ˆ is independent of the past of X. ˆ The preceding arguments prove future of N ˆ that X satisfies (a), which implies that X satisfies (a). To prove the converse, suppose (a) is true. Then, for any i and t ≥ 0, λT t = E[NT (0, t]] = E[NT (0, t]|X(0) = i]  t = E[αT (X(s))|X(0) = i] ds. 0

The integrand is continuous in s since X is a CTMC. Taking the derivative of this equation with respect to t yields λT = E[αT (X(t))|X(0) = i],

t ≥ 0.

Then, using the first jump time T1 = inf{t > 0 : X(t) = X(0)}, we can write λT = αT (i)P {T1 > t|X(0) = i} + E[αT (X(t))1(T1 ≤ t)|X(0) = i]. Letting t ↓ 0 proves (b).

296

4 Continuous-Time Markov Chains

Example 90. Births in a Birth-Death Process. Suppose X denotes the population size in an ergodic birth-death process with birth and death rates λi and μi . Its transition rates are qij = λi 1(j = i + 1) + μi 1(j = i − 1). Consider the point process NT of the times at which births occur, where T = {x ∈ D : x(0) = x(0−) + 1}. The initiation rate of births is clearly αT (i) = qi,i+1 = λi ,

for each i ∈ S.

Then Theorem 89 yields the result that λi = λ, for each i, is a necessary and sufficient condition for the birth process NT to be Poisson with rate λ and future births are independent of the past of the process X. Note that this conclusion is true for M/M/s queueing processes. This proves that the arrival process is Poisson with rate λ, which is usually a preliminary assumption in applications. The Poisson criterion for T -transition times in the preceding result applies to non-stationary as well as stationary CTMCs. Our next criterion for Poisson T -transitions is only for stationary CTMCs. This involves considering a CTMC in reverse time and applying Theorem 89. Recall that when X is stationary with distribution p, its time-reversal, ¯ = {X(t) ¯ according to Proposition 63, is a stationary CTMC X : t ∈ R} with −1 transition rates q¯ij = pi pj qji . Note that the time reversal of T is the set T¯ of all paths x ¯(t) in D such that x ¯(t) = x(−t) at each continuity point of x, ¯ is for x ∈ T . Then the initiation rate of T¯ -transitions of X  ¯ ∈ T¯ |X ¯ −1 = i, X ¯ 0 = j} α ¯ T¯ (i) = q¯ij P {X (4.50) j=i

=

p−1 i



pj qji P {X ∈ T |X−1 = j, X0 = i}.

j=i

We say that the time-reversal of T has a uniform initiation rate λT if α ¯ T¯ (i) = λT ,

for each i ∈ S.

Theorem 91. If X is stationary, the following statements are equivalent. (a) NT is a Poisson process with rate λT and the future of X is independent of the past of NT . (b) The time-reversal of T has a uniform initiation rate λT . Proof. Because of the Markov property, a T -transition of X has the same ¯ Furthermore, NT is equal in distribution probability as a T¯ -transition of X. ¯ ¯ ¯ Now, by Theorem 89, (b) is to the point process NT¯ of T -transitions of X. ¯ equivalent to the statement that NT¯ is a Poisson process with rate λT and ¯T¯ is independent of the past of X, ¯ which is equivalent to (a). the future of N

4.15 Poisson Transition Times

297

Example 92. Deaths in a Birth-Death Process. Suppose that X denotes the birth-death process in Example 90, but now assume it is stationary. Consider the point process NT of the times at which deaths occur, which means that T = {x ∈ D : x(0) = x(0−) − 1}. Note that the rate at which deaths are initiated from state i is  αT (i) = qij 1(j = i − 1) = μi 1(i ≥ 1). j=i

This is not the same for each i, even if the μi are all equal. Therefore Theorem 89 does not apply to establish that NT is a Poisson process. To see if Theorem 91 is applicable, we have to determine the initiation rate of the reverse-time of death transitions. We know from Example 60 that the stationary distribution of X(t) is pi = cλ0 · · · λi−1 /(μ1 · · · μi ). Then the initiation rate (4.50) of the reverse-time of death transitions (which are birth transitions for the time reversal of X) is  pj qji 1(j = i + 1) = p−1 for each i ∈ S. p−1 i i pi+1 q(i+1),i = λi , j=i

Thus, by Theorem 91, λi = λ, for each i, is a necessary and sufficient condition for NT to be a Poisson process with rate λ and the past departures are independent of future births or deaths. Of course, if the birth rate is the constant λ, then the death rate should equal this birth rate since X is stationary. However, just because the point process of birth times is Poisson, it is not immediately clear that the process of death-times would be Poisson; but now we know it is. A special case of this example says that the times of departures from a stationary M/M/s queue is a Poisson process with the same rate as the arrivals. For multi-dimensional CTMCs such as Jackson network processes, one may be interested in whether several point processes of transition times are Poisson. We now show that the preceding criteria for Poisson T -transition times extend to multiple transition processes. The key idea is to formulate multiple processes as a partition of a Poisson process as in Section 3.10. Theorem 93. For disjoint transition sets T1 , . . . , T , the following statements are equivalent. (a) The point processes NT1 , . . . , NT are independent Poisson processes with respective rates λT1 , . . . , λT , and their futures are independent of the past of X. (b) Each Tk has a uniform initiation rate λTk , for 1 ≤ k ≤ . Proof. Suppose (b) is satisfied. Then the initiation rate of T = ∪k=1 Tk is

298

4 Continuous-Time Markov Chains

αT (i) =

 

αTk (i) = λT ,

i ∈ S,

k=1

 where λT = k=1 λTk . Then by Theorem 89, NT is a Poisson process with rate λT and the future of NT is independent of the past of X. Now, each of the T -transitions of X is independently triggered by a Tk transition (1 ≤ k ≤ ) (where Tk ⊆ T ) with probability P {STn X ∈ Tk |STn X ∈ T } = P {X ∈ Tk }/P {X ∈ T } = λTk /λT . That is, the processes NT1 , . . . , NT form a partition of NT with the preceding probabilities. Therefore, by Corollary 41 in Chapter 3, these processes are independent Poisson processes with respective rates λT1 , . . . , λT , and, being parts of NT , their futures are independent of the past of X. If (a) holds, then (b) follows by Theorem 89. Here is a companion result for multiple Poisson processes for a stationary CTMC; it follows by an obvious extension of the proof of Theorem 91. Theorem 94. If the CTMC X is stationary, then for disjoint transition sets T1 , . . . , T , the following statements are equivalent. (a) NT1 , . . . , NT are independent Poisson processes with respective rates λT1 , . . . , λT and their pasts are independent of the future of X. ¯ ¯ , 1 ≤ k ≤ . (b) The time-reversal T¯k of Tk has a uniform initiation rate λ Tk Example 95. Jackson Networks. Suppose that X is an ergodic, infinite-capacity open Jackson network process with routing rates λij and service rates φi (xi ), 1 ≤ i ≤ m. Let I ⊂ {1, . . . , m} denote the set of nodes at which arrivals can enter the network from outside. Consider the point process  N0i (B) = 1(X(t) = X(t−) + ei ), B ∈ B, i ∈ I, t∈B

of arrival times of items from outside the network into node i. The processes N0i , i ∈ I, are independent Poisson processes with rates λ0i , i ∈ I; and their futures are independent of the past of X. This follows by Theorem 93 since, for each i ∈ I,  q(x, y)1(y = x + ei ) = λ0i , x ∈ S. y=x

Now suppose X is stationary. Consider the point process Nj0 of departure times (to outside) from node j in the set of nodes J that have departures. Then Nj0 , j ∈ J, are independent Poisson processes with rates wj λj0 , j ∈ J, and their pasts are independent of the future of X. This follows by Theorem 94 since, for each j ∈ J,

4.16 Palm Probabilities

p(x)−1



299

p(y)q(y, x)1(y = x + ej ) = p(x)−1 p(x + ej )q(x + ej , x)

y=x

= wi λj0 ,

x ∈ S.

In some cases, the point process of times at which items move from one node to another is a Poisson processes, provided that a unit may make this move only once during its stay in the network; this is true of acyclic networks as in Exercise 30.

4.16 Palm Probabilities This section continues the study of T -transitions by characterizing the past and future of a CTMC at such a transition. The issue is how to evaluate any probability for a CTMC conditioned that a T -transition occurs. Since the occurrence of a T -transition at any time has probability 0, we cannot use conventional conditional probabilities. Instead, we formulate these conditional probabilities as Palm probabilities. Further properties of Palm probabilities are described in the next three sections. As an example of what lies ahead, consider an M/M/1 queueing process in equilibrium. Items arrive according to a Poisson process, and so the probability of an arrival at any time is 0. However, when an item does arrive to the system, it is of interest to know the distribution of the number of items it encounters in the system and the distribution of the arrival’s sojourn time in the system. Viewing the arrival times as T -transitions, we will prove, using Palm probabilities, that the distribution of the number of items an arrival encounters is the stationary distribution of the process, and that the arrival’s sojourn time has an exponential distribution. For the following discussion, X = {X(t) : t ∈ R} will denote an ergodic CTMC with transition rates qij and stationary distribution p. Unless specified otherwise, we will assume that X is stationary. Then as noted in Proposition 86, the rate at which a T -transition is initiated from state i is  qij P {X ∈ T |X−1 = i, X0 = j}, αT (i) = j=i

and the point process NT of T -transitions of X is stationary with rate  pi αT (i). λT = E[NT (0, 1]] = i

The aim is to formulate conditional probabilities given that a T -transition occurs. Recall that a T -transition of X occurs at time t if St X ∈ T . Given that this event occurs, we want to find the conditional distribution of X, or

300

4 Continuous-Time Markov Chains

NT , or other functions of X. These probabilities are not standard conditional probabilities, since the probability that a transition occurs at any time is 0 (recall Remark 20). However, we can formulate these conditional probabilities as Palm probabilities. A Palm probability is defined in more general contexts as a Radon-Nikod´ ym derivative, but for a CTMC, the definition reduces simply to a ratio of rates. Definition 96. The Palm probability of a stationary CTMC X conditioned that a T -transition occurs at any time t is the probability measure PT defined, for any event A generated11 by the process X, as PT (A) =

1   pi qij P {A, X ∈ T |X−1 = i, X0 = j}. λT i j=i

This probability for a transition at time t is independent of t, and so it is often associated with a transition at time 0. The meaning of a Palm probability as a limit of standard conditional probabilities, like that of a Radon-Nykodym derivative, is given below in Proposition 97. Before getting into examples, we will comment on the definition and present a few properties of Palm probabilities. The Palm probability PT is defined as a function of the underlying probability P , and the two probabilities are different. In particular, while the P -probability of a T -transition is 0, the PT -probability is 1 because PT {NT ({0}) = 1} = PT {X ∈ T } = 1. This property is consistent with saying that PT is a conditional probability “given that a T -transition occurs” at time 0. As another formulation, note that any event A generated by X is a function of the sample paths of X, and so PT (A) = PT {X ∈ TA }, where TA = {x ∈ T : A occurs for the path x}. Consequently, PT (A) =

λTA E[NTA (a, b]] , = λT E[NT (a, b]]

a < b.

(4.51)

This ratio is the expected number of T -transitions at which a TA -event occurs in a fixed time interval divided by the expected number of all T -transitions in the interval (or the portion of T -transitions at which a TA -event occurs). The second ratio is due to E[NT (a, b]] = (b − a)λT since NT is stationary. Another useful expression is, for any T  ⊂ T , 11

The A is in the σ-field of events generated by the random variables {X(t) : t ∈ R}.

4.16 Palm Probabilities

PT {X ∈ T  } =

301

  1 E 1(Sτn X ∈ T  , τn ∈ (a, b]) , (b − a)λT

(4.52)

n∈Z

where · · · < τ−1 < τ0 ≤ 0 < τ1 < · · · are the occurrence times of the T transitions. A Palm probability measure, like any probability measure, has an associated expectation, conditional probabilities, etc. For instance, for any g : D → R, the function g(X) of the CTMC X under the probability PT has the expectation ET [g(X)] =

1   pi qij E[g(X)|X−1 = i, X0 = j], λT i j=i

provided the double sum is finite for |g| in place of g. Now, let us see how a typical Palm probability we are studying is related to a standard conditional probability. The following result shows that the Palm probability is a limit of conditional probabilities. The conditioning says that there is at least one T -transition in (t, 0] and a T -transition is guaranteed to occur at time 0 as t ↑ 0. This limit representation is analogous to that for standard conditional probabilities conditioned on a continuous random variable. Proposition 97. For any event A generated by X, PT (A) = lim P (A|X ∈ T , X(0) = X(t)). t↑0

Proof. Letting TA ⊂ T be as in (4.51), P (A|X ∈ T , X(0) = X(t)) =

P {X ∈ TA , X(0) = X(t)} . P {X ∈ T , X(0) = X(t)}

Conditioning on X(t), X(0) and using the stationarity of X along with limits of transition probabilities in Theorem 18, we have, for t < 0, (−t)−1 P {X ∈ T , X(0) = X(t)}   = pi (−t)−1 pij (−t)P {X ∈ T |X(t) = i, X(0) = j} i

→ λT

j=i

as t ↑ 0.

This limit statement also holds with T replaced by TA . Combining these observations yields lim P (A|X ∈ T , X(0) = X(t)) = t↑0

λTA = PT (A). λT

302

4 Continuous-Time Markov Chains

Palm probabilities for stationary processes and time-dependent Palm probabilities for any non-stationary process are discussed in [60, 100]. Our study is only an introduction that covers Palm probabilities for a CTMC viewed at its transition times. We will not cover extensions (requiring more technical material) that consider other random variables associated with the CTMC or times that are not transition times. For instance, in analyzing an M/M/s system, one may be interested in the times at which a service time exceeds a certain high level, and the state of the system at those times. Or if a random reward is received for processing each item, one may be interested in times at which this reward is 0 due to a defective service. The theory of Palm probabilities also applies to any stochastic process (that need not be stationary or a CTMC), where one considers conditioning on a point occurring at a time t and the Palm probability is a function of t. The definition of time-dependent Palm probabilities for (non-stationary) CTMCs is as follows. Here, we denote the expected infinitesimal rate at which an T -transition is initiated at time t by λT (t) = E[αT (X(t))]   = P {X(t) = i} qij P {X ∈ T |X−1 = i, X0 = j}. j=i

i

Definition 98. The time-dependent Palm probability of a CTMC X conditioned that a T -transition occurs at time t ∈ R is the probability measure PTt defined, for any event A generated by X, by PTt (A) =

λTAt (t) λT (t)

,

where TAt = {x ∈ T : the event A occurs for the path S t x}. It is natural to consider events based on St X (what an observer sees of X from location t). So in particular, PTt {St X ∈ T  } =

λT  (t) , λT (t)

T⊂T.

Is this definition consistent with Definition 96 for a stationary X? It is because if X is stationary, then P {X(t) = i} = pi , and so PTt = PT , t ∈ R. Another consistency condition is that for large t the time-dependent Palm probability, for any non-stationary ergodic CTMC X, is approximately equal to the stationary Palm probability. Remark 99. Time-dependent Palm probabilities defined above converge to stationary Palm probabilities in that w

PTt {St X ∈ ·} → PT {X ∈ ·}

as t → ∞.

This follows from the definition of PTt since P {X(t) = i} → pi , i ∈ S.

4.17 PASTA at Poisson Transitions

303

The preceding remark also holds for more general time-dependent Palm probabilities of a stochastic process X that is asymptotically stationary (which is a property of an ergodic CTMC; see Exercise 22). Many properties of Palm probabilities for stationary properties that do not require the process X to be in equilibrium (like those in the next section) extend to time-dependent Palm probabilities. In these cases, probabilities such as PTt {S t ∈ T  } are used in place of PT {X ∈ T  }.

4.17 PASTA at Poisson Transitions Using the material in the last two sections, we will now explore Palm probabilities of the stationary CTMC X at its Poisson transition times. At such a transition, the initiation rate of transitions is independent of the state, and so it appears that the Palm probability of the state of the chain should be the usual probability. For instance, in a stationary M/M/s system at Poisson arrival times, the distribution of the number of items an arrival encounters should be the same as the stationary distribution. This property is an example of a general theorem we now present. The main result is as follows. It gives necessary and sufficient conditions under which the Palm probability of the state of the stationary CTMC X before (or after) a T -transition is equal to the ordinary probability of being in that state. These equalities are in the same spirit as the classical PASTA12 property that “Poisson arrivals see time averages”. Proposition 100. (PASTA Before a T -Transition). A necessary and sufficient condition for PT {X(0−) = i} = pi ,

i ∈ S,

is that T has a uniform initiation rate. (PASTA After a T -Transition). A necessary and sufficient condition for PT {X(0) = i} = pi ,

i ∈ S,

is that the reverse-time T has a uniform initiation rate. This result coupled with Theorems 89 and 91 imply that if X satisfies either one of these PASTA properties, then NT is a Poisson process. Proof. Recall that T has a uniform initiation rate if αT (i) is the same for each i. By the definition of PT , 12

Since PASTA is a mind-arousing acronym associated with a popular comfort food, we will also use it here to describe what one might call Palm “actions” see time averages.

304

4 Continuous-Time Markov Chains

pi αT (i) . PT {X(0−) = i} =  k pk αT (k)  This ratio equals pi if and only if αT (i) = k pk α(k), for each i, which is equivalent to T having a uniform initiation rate. This proves the first assertion. The second assertion follows by a similar argument since by an interchange of sums  j pj qji P {X ∈ T |X−1 = j, X0 = i})  PT {X(0) = i} =   p k qk P {X ∈ T |X−1 = , X0 = k} pi α ¯ T (i) , =  p ¯ T (k) k kα where α ¯ T (i) = p−1 i

 j

pj qji P {X ∈ T |X−1 = j, X0 = i}.

Here is a classic illustration of PASTA that leads to the characterization of sojourn times in queues. Example 101. Waits in an M/M/s system. Suppose X(t) denotes the number of items in an M/M/s system at time t, where s < ∞. Assume this CTMC is stationary with distribution p; we know its stationary probabilities for i ≥ s are pi = ps (λ/sμ)i−s , where λ and μ are the arrival and service rates. We saw in Examples 90 and 92 that the initiation rate of arrivals has a uniform rate λ, and the time-reversal of departures also has a uniform initiation rate of λ. Hence the arrival and departure processes are both Poisson processes with rate λ. Furthermore, because of these two uniform initiation rates, X satisfies the before and after PASTA properties in Proposition 100, namely PT {X(0−) = i} = pi = PT {X(0) = i}. We will now explore other features of the queueing process at arrivals. First, consider the waiting time W in the queue (prior to its service) of an item that arrives at time 0. This is also the time W = min{t ≥ 0 : X(t) < s} until a server is available  at or after time 0. Viewing arrival times as T transitions with λT = i pi qi,i+1 = λ and using Definition 96, PT {W > 0} = PT {X(0) ≥ s} ∞ 1 pi qi,i+1 = ps /(1 − λ/sμ). = λ i=s As an aside, note that the preceding probability is PT {X(0) ≥ s} = P {X(0) ≥ s, A},

4.17 PASTA at Poisson Transitions

305

where A is the event that following time 0 an arrival occurs before a service completion. This is another illustration of the difference between PT and P . Now, the rest of the distribution of W is ∞

PT {W > t} =

1 pi qi,i+1 Pi+1 {W > t} λ i=s

= ps

∞ 

(λ/sμ)i−s P {M (t) ≤ i − s}.

i=s

Here M is a Poisson process with rate sμ representing the departure process from the s busy servers. Then M (t) is the number of items that can enter service by time t, and so M (t) ≤ i − s is the event that no more than the i − s items the arrival encountered waiting in the queue at time 0 have entered service by time t. Substituting the Poisson probabilities in the last display, we obtain PT {W > t} = PT {W > 0}P {W ∗ > t}, where W ∗ has an exponential distribution with rate sμ − λ. Another way of expressing this result is PT {W ≤ t|W > 0} = P {W ∗ ≤ t}, so the exponential time W ∗ is the duration of an arrival’s wait in the queue given that there is a wait. Exercise 19 explores a slightly different “virtual” waiting time process without Palm probabilities. Next, note that the sojourn time in the system of an arrival at time 0 is ˆ = W + ξ, where ξ is a service time that is independent of W . Using this W independence, we have PT {W + ξ ≤ t|W > 0} = P {W ∗ + ξ ≤ t}, where W ∗ and ξ are independent waiting and service times as above. Therefore, ˆ ≤ t} = PT {W = 0}P {ξ ≤ t} PT {W +PT {W > 0}P {W ∗ + ξ ≤ t}. Also,

ˆ ] = ET [ξ + W ] = 1/μ + sμps /(sμ − λ)2 . ET [W

For the M/M/1 system, the preceding shows that PT {W > 0} = λ/μ, ˆ reduces to an exponential and Exercise 43 shows that the distribution of W distribution with rate μ − λ.

306

4 Continuous-Time Markov Chains

Let us see an example of when PASTA does not hold. Example 102. Times of Jumps From One Set to Another. Consider the times at which the CTMC X jumps from A to B, which are two disjoint, nonempty subsets of S. These are T -transition times where T = {x ∈ D : x(0−) ∈ A, x(0) ∈ B}. Also, when A = B c , these transition times are hitting times of B. By the comments at the beginning of this chapter, the point process NT of these transition  times is stationary; the initiation rate of jumps from A to B is αT (i) = j∈B qij 1(i ∈ A), and and the rate of NT is   pi qij . λT = E[NT (0, 1]] = i∈A

j∈B

Now, these transition times from A to B do not enjoy the PASTA properties in Proposition 100. Indeed, a T -transition does not have a uniform initiation rate since αT (i) = 0 on Ac and it is positive elsewhere. Similarly, the timereversal of T has the initiation rate  pj qji 1(i ∈ B), i ∈ S, α ¯ T¯ (i) = p−1 i j∈A c

and this is 0 on B and it is positive elsewhere.

4.18 Relating Palm and Ordinary Probabilities This section presents formulas that relate expectations and probabilities under PT to their counterparts under P , and vice versa. These basic formulas are often used when studying stationary processes that are functions of a stationary CTMC. Examples are given in the next section. We begin by reviewing a few more properties of stationary processes that need not be Markov processes. Suppose that X = {X(t) : t ∈ R} is a stationary process on a space S with sample paths in D. The following results also apply to a stationary process X  with the time set R+ , because there exists d a stationary process X on R such that X = X  on R+ . Definition 103. A sample-path event A for the stationary process X is shift invariant if {X ∈ A} = {St X ∈ A}, t ∈ R. The process X is ergodic if P {X ∈ A} = 0 or 1 for each shift invariant event. A stationary “ergodic CTMC” is ergodic in this sense (so our earlier use of ergodic is consistent with this one). This and other stationarity properties, including the ergodic theorem below, are proved in [37, 61].

4.18 Relating Palm and Ordinary Probabilities

307

The next result describes a large class of functions of a stationary process X that are also stationary processes. As an elementary example, if Y (t) = f (X(t)), where f : S → S  , then Y is stationary. In contrast, the Markov property does not have a similar heredity property — if X is a Markov processes, then Y is typically not a Markov process. Proposition 104. Associated with the stationary process X with sample paths in D, suppose Y (t) = f (St X), t ∈ R, where f is a function on D to some space S  , which need not be countable. Then Y is stationary. If, in addition, X is ergodic, then so is Y . Proof. The process Y is stationary, since by its definition and the stationarity of X, St Y = {Y (u + t) : u ∈ R} = {f (Su (St X)) : u ∈ R} d

= {f (Su X) : u ∈ R} = Y. Next, assume X is ergodic. To prove Y is ergodic, it suffices to show that, for any shift-invariant event A for Y , there is a corresponding shift-invariant event for X. In this case, the corresponding event is B = {x ∈ D : {f (Su x) : u ∈ R} ∈ A}, because clearly {St Y ∈ A} = {St X ∈ B} for each t. Here is an important ergodic theorem for stationary processes. Theorem 105. Suppose the stationary process X is real-valued with sample paths in D that are Lebesgue-integrable on bounded sets. If X is ergodic and E|X(0)| is finite, then lim t

−1

t→∞



t

X(s) ds = E[X(0)]

a.s.

0

The preceding properties of stationary processes also apply to discretetime processes — just replace the parameter t ∈ R by an integer n ∈ Z. For instance, Theorem 105 would read: If {Xn : n ∈ Z} is a real-valued stationary ergodic process with E|X(0)| < ∞, then lim n−1

n→∞

n 

Xm = E[X0 ]

a.s.

(4.53)

m=1

This is an extension of the classical SLLN for i.i.d. sequences in Chapter 2. We will now return to studying the stationary CTMC X and its Palm probability measure PT for the point process NT of T -transition times

308

4 Continuous-Time Markov Chains

. . . < τ−1 < τ0 ≤ 0 < τ1 < . . . Applications often involve relating means and probabilities under PT to those under P , or vice versa. The next three results are key tools for these relations. Theorem 106. (Campbell Formula)13 For f : R × D → R, 

  t E f (t, S X)NT (dt) = λT ET [f (t, X)] dt, R

R

provided the expressions are finite with |f | in place of f . Proof. We will apply the extended L´evy formula in Theorem 52 to evaluate the expectation of   f (t, St X)NT (dt) = Vn , R

Tn

where Vn = f (Tn , S

n∈Z Tn

X)1(S

X ∈ T ). Using the Markov property,

E[Vn |Tn = t, Xn−1 = i, Xn = j] 

= E f (t, STn X)1(STn X ∈ T )|Tn = t, Xn−1 = i, Xn = j

 = E f (t, X)1(X ∈ T )|X−1 = i, X0 = j . Then by Theorem 52 and the definition of PT ,



     Vn = pi qij E f (t, X)1(X ∈ T )|X−1 = i, X0 = j dt E R

n∈Z

= λT

j=i

i



R

ET [f (t, X)]dt.

This proves the assertion. A variety of stationary processes associated with X are as follows. Proposition 107. For f : R × D → R, the process  Y (t) = f (t − u, Su X)NT (du), R

t ∈ R,

is stationary and ergodic, and 13

An analogous Campbell formula for a more general process Y and a Palm probability PN for a point process N on R (that need not be stationary) with mean measure μ(B) = E[N (B)], is  

 f (t, Y (t))N (dt) = EN [f (t, Y (0))]μ(dt). E R

R

4.18 Relating Palm and Ordinary Probabilities

309

 E[Y (0)] = λT ET

 f (u, X) du ,

R

provided this integral is finite with |f | in place of f . Proof. Since there are a countable number of T -transitions of X, using the change-of-variable v = u − t, we can write  f (t − u, Su X)1(Su X ∈ T ) Y (t) = u∈R

=



f (−v, Sv (St X))1(Sv (St X) ∈ T ).

v∈R

This has the form Y (t) = φ(St X), where φ : D → R, and so Y is stationary and ergodic by Proposition 104. Also, using Theorem 106 and v = −u, 

 f (−u, Su X)NT (du) E[Y (0)] = E R  = λT ET [f (−u, X)] du = λT ET [f (v, X)] dv. R

R

The following is an explicit formula for P in terms of PT . Corollary 108. (Inversion Formula) For g : D → R,

 τ1  E[g(X)] = λT ET g(St X) dt ,

(4.54)

0

provided these expressions are finite with |g| in place of g. Hence  ∞ P {X ∈ A} = λT PT {St X ∈ A, τ1 ≥ t} dt. 0

Proof. We can write g(St X) = g(St X)



1(τn ≤ t < τn+1 )

n∈Z

 = R

g(St−u (Su X))1(t − u ∈ [0, h(Su X))NT (du),

where h(S X) = inf{v > 0 : Sv (Su X) ∈ T }. Then Proposition 107 yields

  E[g(X)] = λT ET g(Su X)1(u ∈ [0, h(X)) du . u

R

This equality is the same as (4.54), since h(X) = τ1 , when τ0 = 0. The second assertion is a special case of (4.54).

310

4 Continuous-Time Markov Chains

4.19 Stationarity Under Palm Probabilities As in the preceding sections, suppose that X is a stationary CTMC and . . . < τ−1 < τ0 ≤ 0 < τ1 < . . . are the occurrence times of T -transitions of X. Consider the sequence {τn+1 − τn : n ∈ Z} of times between T -transitions. This sequence is not stationary under P , but it is stationary under the Palm probability PT . This property is an example of the main result here that the sequence of sample paths of X observed at T -transition times is stationary and ergodic under the Palm probability PT . Consequently, several families of stationary processes associated with the CTMC satisfy a SLLN under PT . We apply the main result to characterize sequences of sojourn and travel times for CTMCs. Since the CTMC X is stationary in the time parameter t, it might suggest that the sequence {X(τn ) : n ∈ Z} of X-values at the T -transitions should be stationary in the parameter n. This sequence is not stationary under P , but it is under PT . The justification for this statement comes from the following important stationarity property for the sequence of sample paths of X at transition times defined by Yn = Sτn X = {X(t + τn ) : t ∈ R},

n ∈ Z.

Theorem 109. The sequence {Yn : n ∈ Z} with values in D is stationary and ergodic under the Palm probability PT . Proof. The sequence Y = {Yn : n ∈ Z} is stationary if and only if PT {SY ∈ C} = PT {Y ∈ C},

C ⊂ S∞,

where SY = {Yn+1 : n ∈ Z} is Y shifted by one time unit. To prove this equality, consider τ (x) = inf{t > 0 : St x ∈ T },

x∈T

which is the time between the T -transition of x at time 0 and the first one after 0 (a T -transition of x occurs at time 0 since S0 x = x ∈ T ). Next, define the transformation θ from T to T by θx = Sτ (x) x, for x ∈ T . Finally, define the iterates θn by θn x = θ(θn−1 x), n ≥ 1. Then it follows by induction, using Sτn+1 X = Sτn+1 −τn (Sτn X), that Sτn x = θn x,

x ∈ T , n ≥ 0.

With this notation and (4.52), we have,14 for t ≥ 0, 14

Here we use the shorthand {Sτn Yk } = {Sτn Yk : n ∈ Z}, and represent other sequences similarly.

4.19 Stationarity Under Palm Probabilities

311

PT {Y ∈ C} =

NT (t)  1  E 1({Sτn Y } ∈ C) tλT n=1

=

NT (t)  1  E 1({θn X} ∈ C) . tλT n=1

By a similar argument followed by the use of the preceding, we have PT {SY ∈ C} =

NT (t)  1  E 1({θn+1 X} ∈ C) tλT n=1

= PT {Y ∈ C} + −

1 P {{θNT (t)+1 X} ∈ C)} tλT

1 P {{θX} ∈ C, NT (t) ≥ 1}. tλT

The NT (t) ≥ 1 in the last statement is because it is implicit in the first summation. Letting t → ∞ proves PT {SY ∈ C} = PT {Y ∈ C}. Since the CTMC X is ergodic, one can show, as in the proof of Proposition 104, that Y is ergodic. The preceding result yields several important SLLNs. Since f (Yn ) is stationary and ergodic, by a discrete-time version of Proposition 104, the following result is a consequence of the ergodic theorem in (4.53). Corollary 110. (SLLN Under Palm Probabilities) For the stationary, ergodic process Yn = Sτn X in Theorem 109 and f : D → R, lim n

n→∞

−1

n 

f (Ym ) = ET [f (Y1 )]

a.s. under PT ,

(4.55)

m=1

provided the last expectation is finite. Example 111. Special cases of the preceding convergence a.s. under PT are n−1

n  m=1

g(X(τm )) → ET [g(X(τ1 ))],

n−1

n 

(τm − τm−1 ) → ET [τ1 ],

m=1

for g : S →  and ET [|g(X(τ1 ))|] < ∞. Here X(τn ) is stationary and ergodic by Theorem 109 since g(X(τn )) = f (Yn ), where f (x) = g(x(0)). A similar statement is true for the sequence τn − τn−1 . Theorem 109 provides the following framework for characterizing sequences of sojourn and travel times; the general mean-value formula (4.56) is analogous to the Little law for queues in Theorem 57 in Chapter 2.

312

4 Continuous-Time Markov Chains

Proposition 112. (Sojourn and Travel Times) Associated with T -transition times τn of X, assume that Wn = h(Sτn X) is a waiting time for a certain event to occur, where h : D → R+ . Then under PT the waiting time sequence {Wn : n ∈ Z} is stationary and ergodic, and lim n−1

n→∞

n 

Wm = ET [W0 ]

a.s. under PT .

m=1

In addition, this limit is given by E[Y (0)] = λT ET [W0 ], where Y (t) =



n∈Z 1(τn

(4.56)

≤ t < τn + Wn ), t ∈ R.

Proof. The first assertion follows by Proposition 104, Theorem 109, and Corollary 110. Next, note that  1(0 ≤ t − u < h(Su X))NT (du). Y (t) = R

Then (4.56) follows since by Proposition 107

  E[Y (0)] = λT ET 1(0 ≤ u < W0 ) du = λT ET [W0 ]. R

Here are several applications of the preceding proposition. Example 113. Waiting Times in a Set. In the context of Proposition 112, suppose the T -transition times τn are the times at which items enter a set B. Let Wn = h(Sτn X) be the sojourn time of X in B starting at time τn , where h(x) = inf{t > 0 : x(t) ∈ B c }. Then by Propositions 112 and 86 with Y (t) = 1(X(t) ∈ B), lim n

n→∞

−1

n 

 Wm =  i∈B c

m=1

pi 

i∈B

pi

j∈B

qij

a.s. under PT .

Example 114. Waiting Times in an M/M/s system. Suppose that X is the stationary M/M/s queueing process as in Example 101, and the T -transition times τn are the times at which items arrive to the system with rate λT = λ. Let Wn = h(Sτn X) denote the length of time the arrival at time τn spends waiting for service, where h(x) = inf{t ≥ 0 : x(t) < s}, and note that  1(τn ≤ t < τn + Wn ), t ∈ R. s ∧ X(t) = n∈Z

Therefore by Proposition 112, Wn is stationary and ergodic, where (4.56) is

4.19 Stationarity Under Palm Probabilities

313

E[s ∧ X(0)] = λET [W0 ]. ˆ n = Wn + ξn that the arrival at time τn spends Next, consider the time W ˆ n is stationary in the system, where ξn is its service time. It follows that W ˆn and ergodic since (Wn , ξn ) is. The distributions and means of Wn and W under PT were presented in Example 101. Example 115. Inter-departure Times from an M/M/s system. Suppose X is the stationary M/M/s queueing process as in Example 101. We know from Example 90 that the point process NT (t) of departures is a Poisson process with rate λT = λ (this is, of course, with respect to the underlying probability measure P ). Then the times between departures Wn = τn − τn−1 are i.i.d. exponential random variables with rate λ. These times are considerably different with respect to the Palm probability PT . In fact the inter-departure times Wn satisfy the statements in Proposition 112, where Y (t) = X(t) and so E[X(0)] = λET [W0 ]. Here is a more complex example of a travel time whose mean is a function of random lengths of the past and future of a Markov chain. Example 116. Travel Times Between Two Sets. In Proposition 112, suppose the T -transition times τn are the times at which the process X exits a set A and thereafter enters a set B before returning to A. That is T = {x ∈ D : x(0−) ∈ A, x(0) ∈ Ac , ηB (x) < ηA (x)}, where ηA (x) = inf{t > 0 : x(t) ∈ A}. Consider the travel time Wn = h(Sτn X) of X from A to B starting at time τn , where h(x) = inf{t > 0 : x(t) ∈ B},

x∈T.

Let γi = P {ηB (X) < ηA (X)|X0 = i}, which is the probability that the embedded Markov chain Xn hits B before A starting at i ∈ Ac . These hitting probabilities are described in Section 1.7. Also, let γ¯i the probability of hitting B before A starting at i ∈ Ac for the embedded Markov chain of the reverse¯ as in Proposition 63 with transition rates q¯ij = p−1 pj qji . time process X i Then by Propositions 112 and 86  n  pi γi γ¯i /  lim n−1 Wm =  i∈A∪B a.s. under PT n→∞ p i i∈A j∈Ac qij γi m=1 In this case, E[Y (0)] =



pi γi γ¯i .

i∈A∪B /

To prove this, note that Y (t) is the indicator of X being in a traverse from A to B at time t, and so

314

4 Continuous-Time Markov Chains

Y (t) = 1(X(t) ∈ / A ∪ B, ηB (St X) < ηA (St X), η¯B (St X) < η¯A (St X)), where η¯A (x) = sup{t < 0 : x(t) ∈ A}, the last exit time of x from A. Then the expression above for E[Y (0)] follows by using the Markov property that conditioned on X(0) = i, the past and future of X are independent, and P {¯ ηB (X) < η¯A (X))|X(0) = i} = γ¯i . This concludes our discussion of Palm probabilities.

4.20 G/G/1, M/G/1 and G/M/1 Queues We will now study the equilibrium behavior of several classical queueing processes. We begin with a G/G/1 single-server queueing system in which the arrival times form a renewal process and the service times are i.i.d. The first result establishes the convergence of the sequence of waiting times of the items. Then the equilibrium behavior of the queue lengths as well as the waiting times are characterized when the arrival process is Poisson (the M/G/1 model), and when the service times are exponential (the G/M/1 model). The queue-length processes in these two models are regenerative processes and their state at certain transition times are Markov chains that have tractable stationary distributions. We will consider a general processing system that operates as follows. Items arrive to the system at times 0 < τ1 < τ2 < . . . that form a renewal process. Denote the i.i.d. inter-arrival times by Un = τn − τn−1 , where τ0 = 0. The service times are i.i.d. nonnegative random variables Vn that are independent of the arrival times. The service discipline is first-come-first-served with no preemptions, and the inter-arrival and service times have finite means. The traffic intensity of the process is ρ = E[V1 ]/E[U1 ], the arrival rate 1/E[U1 ] divided by the service rate 1/E[V1 ]. Let Q(t) denote the quantity of items in the system at time t, and let Wn denote the length of time that the item arriving at time τn spends in the queue before being processed. For simplicity, assume the system is empty at time 0, so Q(0) = 0 and W0 = 0. Definition 117. The process {Q(t) : t ≥ 0} is an G/G/1 queuing process. Special cases are: M/G/1 Process — The input process is Poisson. G/M/1 Process — The service times are exponentially distributed. M/M/1 Process — Poisson input process and exponential service times. The first step is to formally define the processes Q(t) and Wn as functions of the system data Un and Vn . Applying the inductive construction for analogous discrete-time waiting times in Example 24 in Chapter 1, we have the Lindley recursion

4.20 G/G/1, M/G/1 and G/M/1 Queues

315

Wn = (Wn−1 + Vn−1 − Un )+ ,

n ≥ 1.

We also learned that, due to the i.i.d assumptions on the system data, n 

Wn = max

0≤m≤n

d

(V−1 − U ) = max Zm , 0≤m≤n

=m+1

(4.57)

n where Zn = m=1 (Vm − Um ) and Z0 = 0. Another quantity of interest is the time at which the nth item departs which is Dn = τn + Wn + Vn , n ≥ 1. Using these functions of the system data, the quantity of items in the system at time t is given by Q(t) =

∞ 

1(τn ≤ t < Dn ),

t ≥ 0.

n=1

We will now study the limiting behavior of these processes. We were able to describe many features of the M/M/1 queueing process Q(t) since it is a CTMC. However, the G/G/1 system, and even the non-Markovian M/G/1 and G/M/1 processes are considerably more complicated, and so many of their features do not have tractable expressions. As a first step, we show that the waiting times Wn for the G/G/1 system converge in distribution to a random variable W , and that this limit is a finite-valued random variable if the traffic intensity is below 1. Limits of waiting times in G/G/1 systems in heavy traffic (when the traffic intensity is not below 1) are described later in Section 5.16. Theorem 118. For the G/G/1 system, the waiting times Wn satisfy d

Wn → W =

sup Zm

0≤m 1.

(4.59)

Furthermore, F (t) = P {W ≤ t} satisfies the Wiener-Hopf integral equation  F (t) = F (t − s)dG(s), t ≥ 0, (−∞,t]

where G(t) = P {Z1 ≤ t}, t ∈ R.

316

4 Continuous-Time Markov Chains d

Proof. By (4.57), we know Wn = max0≤m≤n Zm and, as n → ∞, this maximum increases a.s., and hence in distribution, to sup0≤m 1 or ρ < 1. Then 0 if ρ < 1 P { sup Zm = ∞} = 1 if ρ > 1. 0≤m 0 since each bi > 0, and the solution is πi = 0 when π0 = 0. Now, the solution π will be a distribution if and only if π0 > 0 and ∞ i G(1) = 1, where G(s) = i=0 πi s . To determine when this occurs, note i that multiplying (4.62) by s and summing on i, we have G(s) = π0 H(s) + s−1 [G(s) − π0 ]H(s), where H(s) =

∞ i=0

ai si . Then

318

4 Continuous-Time Markov Chains

G(s) =

π0 (1 − s)H(s) . H(s) − s

(4.64)

Using L’H˜ opital”s rule and H  (1) = ρ as noted in (4.60), it follows that G(1) = 1 if and only if π0 = 1 − ρ. Hence π is a stationary distribution if and only if ρ < 1. We also know from (4.60) that  H(s) = E[sM(t) ]P {V ∈ dt} = ψ(λ(1 − s)), R+

where M (t) is a Poisson process with rate λ and E[sM(t) ] = e−λ(1−s)t . Substituting this expression for H(s) and π0 = 1 − ρ in (4.64) proves (4.61). For this M/G/1 model with ρ < 1, it is clear that Q(t) is a regenerative process at the times at which it enters state 0. Also, the time between regenerations has a finite mean, since the busy period discussed in Exercise 48 has a finite mean. Hence Q(t) has a limiting distribution. Furthermore, one can show that its limiting distribution is the same as the limiting distribution of the embedded chain Xn (e.g., see [76]). In addition, the SLLN and central limit theorem in Chapter 2 for regenerative-increment processes apply to this queueing process. Knowing the stationary distribution for an ergodic M/G/1 queueing process, we can now characterize the limiting distribution of its waiting times. Theorem 120. For the M/G/1 queueing process with ρ < 1, the waiting d

times satisfy Wn → W as n → ∞, and W has the Laplace transform E[e−sW ] =

(1 − ρ)s , s − λ + λψ(s)

where ψ(s) = E[e−sV ]. Proof. The nth item departing from the system leaves behind Xn = Q(Dn ) items in the system, and so Xn equals the number of arrivals in the time interval (τn , τn + Wn + Vn ] during which the nth item is in the system. In other words, Xn = M (τn , τn + Wn + Vn ], where M is the Poisson arrival process. Then using E[sM(t) ] = e−λt(1−s) , we have  E[sXn ] = E E[sXn |Vn , Wn ] = E[e−λ(Wn +Vn )(1−s) ] = E[e−λ(1−s)Wn ]E[e−λ(1−s)V ]. d

By Theorem 119, Xn → X, where E[sX ] = G(s) in (4.61). Then from the preceding display lim E[e−λ(1−s)Wn ] = G(s)/ψ(λ(1 − s)).

n→∞

4.20 G/G/1, M/G/1 and G/M/1 Queues

319 d

From this with G(s) given by (4.61), we have Wn → W where E[e−λ(1−s)W ] =

(1 − ρ)(1 − s) , ψ(λ(1 − s)) − s

which proves the assertion. We will now present similar results for a G/M/1 queueing process Q(t) with exponential service rate μ. Because of its complicated structure, our ˆ n = Q(τn −), which depicts the focus will be on the embedded process X quantity in the system just prior to the arrival time τn of the nth item. The ˆ n satisfy the recursion quantities X ˆ n+1 = X ˆ n + 1 − Yˆn , X

n ≥ 0,

where Yˆn is the number of items that depart in the time interval (τn , τn+1 ]. Because the exponential service times with rate μ are memoryless, the residual service time of the item in service at time τn is exponential with rate μ, and so the potential departures in an interval between arrivals occur ˆ (t) with rate μ. Consequently, according to a Poisson process M ˆ n = i} = P {M ˆ (U ) = k} αk = P {Yˆn = k|X  e−μt (μt)k P {U ∈ dt} if k < i, = k! R+

(4.65)

d ˆ . The only other where U = Un is independent of the Poisson process M ˆ possibility for Yn is that all items are served in an inter-arrival time and so

ˆ n = i} = P {M ˆ (U ) ≥ i + 1} = 1 − βi = P {Yˆn = i + 1|X

k 

α .

=0

These observations are the basis of the following result. ˆ n : n ≥ 0} is an irreducible, Theorem 121. (G/M/1 Model) The process {X aperiodic Markov chain with transition matrix ⎤ ⎡ β0 α0 0 0 0 · · · ⎢β1 α1 α0 0 0 · · ·⎥ ⎥ ⎢ ⎢β2 α2 α1 α0 0 · · ·⎥ ⎥ P=⎢ ⎢β3 α3 α2 α1 α0 · · ·⎥ ⎥ ⎢ ⎣. . . . . . . . . . . . . . . . . .⎦ .................. The Markov chain Xn is ergodic if and only if ρ < 1. In this case, its stationary distribution is πi = (1 − r)ri ,

i ≥ 0,

where r is the unique root in (0, 1) of r = E[e−μ(1−r)U ].

320

4 Continuous-Time Markov Chains

ˆ n is a Markov chain with the specProof. Under the queueing assumptions, X ified transition matrix since ˆ0, . . . , X ˆ n−1 , X ˆ n = i} = P {Yˆn = i + 1 − j|X ˆ n = i}. ˆ n+1 = j|X P {X ˆ n is irreducible and aperiodic. Also, the positive αi ensure that X For this Markov chain, the balance equations π = πP are π0 = πi =

∞  j=0 ∞ 

βj πj , i ≥ 1.

αj πj+i−1

j=0

To solve this difference equation, we consider a solution of the form πi = cri (this approach was used in the Gambler’s Ruin Model in Example 44 in Chapter 1). Substituting this π in the last equation and dividing by ri−1 , we obtain r = φ(r), where φ(r) =

∞ 

αj rj = E[e−μ(1−r)U ].

j=0 ˆ

The last equality is due to (4.65) and E[sM(t) ] = e−μ(1−s)t . We have shown that πi = cri is an invariant measure, where r satisfies r = φ(r), and it is a finite measure if and only if 0 < r < 1. Now, we know from the branching process Lemma 47 in Chapter 1 that the equation r = φ(r) has a unique solution r in (0, 1) if and only if φ (1) > 1. In this case, ˆ n is ergodic if and only if ρ < 1 and φ (1) = 1 − ρ. Hence the Markov chain X in that case its stationary distribution is πi = (1 − r)ri . For the G/M/1 model, the limiting distribution of the embedded chain ˆ n is not necessarily the limiting distribution of its parent process Q(t). In X fact, Q(t) does not even have a limiting distribution when the inter-arrival ˆ n yields a times Un are a constant. However, the limiting distribution for X tractable limiting distribution for the waiting times. Theorem 122. For the G/M/1 queueing process with ρ < 1, the waiting d

times satisfy Wn → W as n → ∞, where P {W = 0} = 1 − r, P {W > t} = re−μ(1−r)t ,

t ≥ 0,

and r is the smallest root in (0, 1) of r = E[e−μ(1−r)U ]. Proof. We noted above that, because of the memoryless property of the exponential service distribution, the potential departure times during an

4.21 Markov-Renewal Processes*

321

ˆ (t) with rate μ. Then since the inter-arrival time form a Poisson process M ˆ arrival at time τn has to wait for the Xn items ahead of it to be served, it follows that ˆ n > 0} P {Wn > 0} = P {X ∞  ˆ (t) < X ˆn} = ˆ n > k}P {M(t) ˆ P {X = k}. P {Wn > t} = P {M k=0

ˆ n > k} → rk+1 as n → ∞. Applying this to the By Theorem 121, P {X d

preceding display yields Wn → W , where P {W > 0} = r and P {W > t} =

∞ 

ˆ (t) = k} = re−μ(1−r)t . rk+1 P {M

k=0 ˆ

The last equality is due to E[rM (t) ] = e−μ(1−r)t .

4.21 Markov-Renewal Processes* We end this chapter by describing a close relative of a CTMC called a Markovrenewal process. This type of process is a jump process like a CTMC, but the sojourn time in a state has a general distribution that may depend on that state and the next state after the sojourn. Markov-renewal processes may provide more precise models than a CTMC when the exponential sojourn time assumption is not appropriate. Much of the theory of CTMCs readily extends to these processes. We will only present a brief sketch of the equilibrium behavior. Definition 123. A jump process {X(t) : t ≥ 0} on S with embedded process (Xn , ξn ) is a Markov-renewal process if it satisfies the following conditions. (i) Xn is a discrete-time Markov chain on S with transition probabilities P = {pij }, where pii = 0, for each i. (ii) For nonnegative t0 , . . . , tm , P {ξ0 ≤ t0 , . . . , ξm ≤ tn |Xn , n ≥ 0} =

m 

P {ξn ≤ tn |Xn , Xn+1 },

n=0

and there are distributions Fij (t), i, j ∈ S, such that Fij (0) = 0 and, for each n ≥ 0, P {ξn ≤ t|Xn = i, Xn+1 = j} = Fij (t),

t ≥ 0, i, j ∈ S.

322

4 Continuous-Time Markov Chains

Some authors refer to X(t) as a semi-Markov process and (Xn , ξn ) as a Markov-renewal process. A typical example is the cyclic renewal process studied in Chapter 3. The notation and many properties of CTMCs, such as classifying states, carry over to Markov-renewal processes. For instance, Proposition 4.4 justifies that the regularity of X(t) implies that the sojourntime distributions Fij are P-regular in the sense that their respective means ∞ μij are finite and n=0 ξn = ∞ a.s. For the following result, assume the embedded Markov chain Xn is ergodic with stationary distribution π. Then X(t) is also recurrent and irreducible. As in the preceding sections, the  first entrance time of the Markov-renewal νi −1 process X(t) to a state i is τi = n=0 ξn , which is the sum of the sojourn times until Xn reaches i. From Exercise 56,   πj pj μj . Ei [τi ] = πi−1 j∈S

∈S

Assume this is finite. Then X(t) is positive recurrent. Finally, assume for simplicity that at least one of the distributions Fij is non-arithmetic, which ensures that the distribution of τi is also non-arithmetic. Under these assumptions, the process X(t) is an ergodic Markov-renewal process with transition probabilities pij and sojourn distributions Fij . The Markov-renewal process X(t) is a delayed regenerative process at the times it enters a fixed state. Therefore, its limiting distribution is essentially the same form as that for CTMCs in Theorem  39. The only difference is that the mean sojourn time in state i is now j∈S pij μij instead of qi−1 for CTMCs. The proof of the following is Exercise 56. Theorem 124. The ergodic Markov-renewal process X(t) has the limiting distribution, for a fixed i ∈ S,

 τi  1 1  E 1(X(t) = j) dt = pj μj , j ∈ S. pj = Ei [τi ] Ei [τi ] 0 ∈S

Furthermore, pj = πj

 ∈S

pj μj /



πi pi μi ,

j ∈ S.

(4.66)

i,∈S

Ergodic theorems for functionals of Markov-renewal processes are similar to those for CTMCs in Theorems 42 and 45. One simply uses (4.66) as the limiting distribution, and qj is replaced by 1/μj .

4.22 Exercises

323

4.22 Exercises Exercise 1. Compound Poisson Processes. Let X(t) denote a compound Poisson process as in Section 3.15 in Chapter 3 with rate λ and jump-size density f (i), i ∈ Z. Justify that X(t) is a CTMC and specify its transition rates. Assuming X(t) is irreducible on Z, classify its states when the mean of the density f is = 0 or is = 0. Exercise 2. Continuation. Let Xk (t), k = 1, . . . , m, be independent compound Poisson processes with rates λk and jump-size densities fk (i), i ∈ Z. Show that X1 (t)+X2 (t), and X1 (t)−X2 (t) are CTMCs and that they are also compound Poisson processes; specify their defining parameters. More generm ally, show that the process X(t) = k=1 ak Xk (t), for ak ∈ Z, is a CTMC and a compound Poisson process and specify its defining parameters. Hint: the jump-size density of X(t) is a mixture of the fk . Exercise 3. Input-Output System: Reflected Compound Poisson Process. Let X(t) denote the value of a system (e.g., monetary account or storage area in a computer) at time t that takes values in S = {i ∈ Z : a ≤ i ≤ b}, where a < b. The value increases “potentially” by a compound Poisson process X1 (t) with rate λ1 and nonnegative jump-sizes with density f1 , but part or all of an input is rejected (or disregarded) to the extent that it would force X(t) to exceed b. Also, the value decreases “potentially” by a compound Poisson process X2 (t) with rate λ2 and nonnegative jump-sizes with density g, but part or all of a decrease is disregarded to the extent that it would force X(t) to fall below a. That is, letting Z(t) = X1 (t) − X2 (t) and ΔX(t) = X(t) − X(t−),   X(t) = X(0)+Z(t)− (X(s−)+ΔX1 (s)−b)+ −(a−X(s−)+ΔX2(s))+ . 0≤s≤t

This X(t) is the compound Poisson process Z(t) reflected at a and b. Note that it is a continuous-time version of the reflected random walk in Example 23 in Chapter 1. Justify that X(t) is a CTMC and specify its transition rates. For a = 0 and b = ∞, show that lim P {X(t) ≤ i} = P { sup Z(s) ≤ i}.

t→∞

0≤s≤∞

Exercise 4. Markov Chains Driven by Clock Times. Let {Xn : n ≥ 0} be a stochastic process on S whose dynamics are as follows. Whenever the process enters a state i, a set of independent clock times τij , j ∈ Si are started, where Si is the subset of states in S that can be reached from state i in one step. The times τij are geometrically distributed with parameters γij m (P {τij > m} = γij ). Then the sojourn time in state i is the minimum τi = minj∈Si τij , and, at the end of the sojourn, the process jumps to the state j ∈ Si for which τij = τi . Find the distribution of τi .

324

4 Continuous-Time Markov Chains

Think of τij as the time to the next “potential” transition from i to j ∈ Si with probability pij , and the clock time j that is the smallest of these times “triggers” a transition from i to j. Show that Xn is a Markov chain with transition probabilities  P {Xn+1 = j|Xm , m < n, Xn = i} = γij / γik . k∈Si

Exercise 5. Multiclass Exponential Clocks. Consider a jump process {X(t) : t ≥ 0} with countable state space S that evolves as follows. For each pair of states i, j, there is a countable set of sources Y(i, j) that may trigger a transition from i to j, provided that such a transition is possible. Specifically, whenever the process X(t) is in state i, the time for source y to “potentially” trigger a transition to j is exponentially distributed with rate qy (i, j), independent of everything else. Then the time for a transition from i to j is the minimum of these independent exponential times, and so the potential transition time from i to j is exponentially distributed with rate  qy (i, j). qij = y∈Y(i,j)

Thus, as in Example 9, we know that X(t) is a CTMC with transition rates qij , provided they are regular with respect to pij = qij /qi , where qi = j qij . In this setting, the sources that trigger the transitions are of interest, especially if there are costs or rewards associated with the sources. By properties of exponential variables, the probability that source y is the one that triggers the transition is qy (i, j)/qij . Let Yn denote the source that triggers the transition at time Tn+1 . Consider the process Y (t) = Yn+1 ,

if t ∈ [Tn , Tn+1 ) for some n.

This Y (t) is the source that triggers the next transition at or after time t. (a) Show that (X(t), Y (t)) is a CTMC on the set Sˆ = {(i, y) : i ∈ S, y ∈ ∪i,j∈S Y(i, j)}. Specify its exponential sojourn rates q(i,y) and its transition probabilities p(i,y),(j,y ) . (b) Show that (Xn , Yn , Tn ) is a discrete time Markov chain and specify the ˆ following transition probabilities: For s, t ≥ 0, (i, y), (j, y  ) ∈ S, P {Xn+1 = j, Yn+1 = y  , Tn+1 > s + t|Xn = i, Yn = y, Tn = s}. Exercise 6. M/M/s System With Feedback. Consider the M/M/s queueing system in Example 60 with the modification that upon completing its service, an item is either fed back for another service with probability r, or it exits the system with probability 1 − r. In this case, the process X(t) representing the number of items in the system would not change state at a feed-back departure — it experiences a fictitious jump. Justify that X(t) is a CTMC

4.22 Exercises

325

with parameters pˆi,i = rμi /ˆ qi , pˆi,i+1 = λ/ˆ qi ,

pˆi,i−1 = (1 − r)μi /ˆ qi ,

i ≥ 1,

and qˆi = λ+μi , where μi = min{i, s}. Show that X(t) can also be represented as a CTMC with transition rates qi,i+1 = λ, qi,i−1 = (1 − r)μi . Specify its parameters pij and qi under this alternative representation. Exercise 7. Independence of Future and Past Given the Present. Suppose that X(t) is a CTMC and for t > 0, let Y (t) be a random variable generated by the past {X(s) : s < t} and let Z(t) be a random variable generated by the future {X(u) : u > t}. Show that E[Y (t)Z(t)|X(t)] = E[Y (t)|X(t)]E[Z(t)|X(t)]. Exercise 8. Two types of jobs, labeled 1 and 2, arrive to a processor according to independent Poisson processes with rates λ1 and λ2 . Let X(t) denote the type of the last job arrival before time t. Show that X(t) is a CTMC and specify its transition rates. Show that pij (t) = a + be−(λ1 +λ2 )t , and specify the coefficients a and b. Show that the X(t) is ergodic and find its stationary distribution. Exercise 9. Yule Process. Consider a pure birth process with exponential sojourn rates qi = iλ, for some λ > 0. Find an expression for pij (t). Hint:A special case is p1j (t) = e−λt (1 − e−λt )j−1 , j ≥ 1. Exercise 10. Search Process. Items arrive to a system of m cells labeled 1, . . . , m at times that form a Poisson process with rate λ. Each arrival independently enters cell i with probability αi and remains there until it is deleted by a search. Independently of the arrivals, searches occur at times that form a Poisson process with rate μ, and each search is performed at cell i with probability δi . If items are in the search cell, one item is deleted; otherwise no items are deleted and the search is terminated. Let X(t) = (X1 (t), . . . , Xm (t)) denote the numbers of items in the cells at time t. Justify that X(t) is a CTMC and classify its states. Find an invariant measure for it. Prove that X(t) is positive recurrent if and only if αi λ ≤ δi μ, 1 ≤ i ≤ m. Find the mean and variance of the number of items in node i in equilibrium (that is E[Xi (0)] and Var[Xi (0)] when X(t) is stationary). Exercise 11. Continuation. In the search model in the preceding exercise, suppose the system is constrained to have X1 (t) ≤ X2 (t) ≤ · · · ≤ Xm (t). Find an invariant measure for the resulting process X(t) with this constraint. Exercise 12. Merging Process. Two types of items arrive to a merging station at times that form two independent Poisson processes with respective

326

4 Continuous-Time Markov Chains

rates λ1 and λ2 . The units queue up and merge into pairs (one of each type) as follows. Whenever a type 1 item arrives to the station, it either merges with a type 2 item that is waiting at the station, or it enters a queue if are no type 2 items present. Similarly, a type 2 arrival either merges with a type 1 item or it enters a queue. Let Xk (t) denote the number of type k items at the station at time t (k = 1 or 2). Note that either X1 (t) or X2 (t) is 0 at any time. Assume that the station can contain at most m items (which are necessarily of one type), and when this capacity is reached, additional items of the type in the queue are turned away. Examples of such a system are automatically guided vehicles meeting products to be transported, or taxis and customers pairing up at a station. When the system is in equilibrium, find the following quantities: • The probability of ik type k items at the station. • The probability of more than i items at the station. • The expected number of type ik items at the station. Also, find the average number of type k items that are turned away and not served. Hint: model the system by the process X(t) = X1 (t) − X2 (t). Exercise 13. For an ergodic CTMC X(t), show that, for any state i ∈ S, lim n−1 Tn = Ei [τi ].

n→∞

Then show that t−1 TN (t) → 1 a.s. as t → ∞. Exercise 14. Balking and reneging in an M/M/s system. Suppose that X(t) is an M/M/s queueing process with arrival rate λ and service rate μ. Consider the variation in which items balk at entering, such that when n items are in the system, the arrival rate is bi λ. For instance, bi could be the probability that an arrival decides to enter the system when i items are in the system (e.g., bi = 1(i ≤ K)). Assume bi = e−αi/μ , where i/μ is an estimate for the average waiting time when i items are present and α > 0. Specify the transition rates for the system, give a criterion for the process to be ergodic, and determine its stationary distribution. Next, consider the M/M/s process X(t) in which an item in the system may renege and depart from the system with rate ri . That is, the time to the next departure is the minimum of the service times of the items being served and an independent exponential random variable with rate ri . Therefore, when i items are in the system, the next potential departure is exponentially distributed with rate μi ∧ s + ri−1 . Determine when X(t) is ergodic, and find its stationary distribution. Finally, determine conditions under which X(t) is ergodic, and find its stationary distribution if there is both balking and reneging as above. Exercise 15. Multiclass Service System with Blocking. Consider a service system that processes m classes of items, but it can serve only one class

4.22 Exercises

327

at any time. While it is serving items of class c, any arrivals of other classes cannot enter the system and are turned away, but new type c arrivals may enter. Assume class c items enter and depart such that the number in the system behaves as an ergodic CTMC on the nonnegative integers with transition rates qc (x, y), and its stationary distribution pc (x) is known. The system is represented as a CTMC X(t) with states x = (x1 , . . . , xm ), where xc is the nonnegative number of class c items in the system — at most, one of the xc ’s is positive. Assume the system is empty at time 0. Now, the state space S consists of a center S0 = {0} and point sets Sc = {(x1 , . . . , xm ) : xc > 0, xl = 0, l = c},

c = 1, . . . , m,

such that the process X(t) can transfer from a state in Sc to a state in Sc , c = c, only by passing through 0. Under the preceding assumptions, the transition rates of X(t) are  qc (x, y) if x, y ∈ S0 ∪ Sc , for some c, and y = ec if x = 0, q(x, y) = 0 otherwise. Show that X(t) is ergodic and its stationary distribution is π(x) = π(0)pc (xc )/pc (0), x ∈ Sc , c = 0,  and π(0) = [1 + c=0 (pc (0)−1 − 1)]−1 . A related discrete-time model is in Example 107 in Chapter 1. Exercise 16. A Batch-Service System. Suppose that X(t) is a CTMC that denotes the number of customers in a service system at time t whose transition rates are qij = λ1(j = i + 1) + μ1(j = max{0, i − K}). Here λ, μ, and K are positive, and λ < Kμ. This represents a system in which customers arrive by a Poisson process with rate λ and are served in batches as follows. Whenever there are i ≥ K customers in the system, batches of K customers depart at the rate μ; and whenever i < K customers are present, all of the customers depart at the rate μ. Show that X(t) is ergodic and its stationary distribution is pi = ri (1 − r), i ≥ 0, where r is the unique root in (0, 1) of the equation μrK+1 − (λ + μ)r + λ = 0. Exercise 17. Continuation. Assume the batch-service process in the preceding exercise is stationary. Let NT denote the point process of T -transition times at which batches of size K depart from the system. Describe the set T , and show that N is a Poisson process with rate μ + λ(1 − r−1 ). Exercise 18. Average Sojourn Time in a Set. Let Wn (A) denote the amount of time the CTMC spends in a set A on its nth sojourn in that set. The

328

4 Continuous-Time Markov Chains

average sojourn or waiting time in A is W (A) = lim n−1 n→∞

n 

Wm (A) a.s.

m=1

 Show that W (A) = λ(A)−1 i∈A pi . Use this to find the average duration of time that an M/M/1 spends with less than K items in the system. Exercise 19. M/M/s system. Let X(t) denote a stationary M/M/s queueing process as in Example 60, with s < ∞, traffic intensity ρ = λ/sμ < 1, and stationary distribution pi . Let Y (t) denote the number of items in the queue waiting to be served (not in service) at time t. Show that Y (t) is a stationary process and specify its distribution. Also, show that E[Y (0)] =

p0 ρ(sρ)s . (1 − ρ)2 s!

Consider the waiting time W (t) = min{u ≥ t : X(u) < s} until a server is available at or after time t (sometimes called the virtual waiting time in the queue of an arrival at time t before it can enter service). Justify that W (t) is a stationary process, and find E[W (0)]. Show that P {W (0) ≤ t|X(0) = s} = 1 − e−rt , and specify the rate r. Show that P {W (0) > 0} = P {X(0) > s} and P {W (0) > t} =

∞ 

pi Pi {M (t) < i − s} = P {X(0) > s}e−(sμ−λ) ,

i=s

where M (t) is a Poisson process with rate sμ denoting departure times when s servers are busy. Finally, prove P {X(0) ≥ s + m} = (λ/sμ)m . Exercise 20. Passage Times. Let X(t) be a CTMC with transition rates qij (here qii = −qi ). For a fixed set B in the state space S, consider the first passage time τB = inf{t > ξ0 : X(t) ∈ B}. Show that the hitting probabilities vi = Pi {τB < ∞} of set B are the smallest yi in [0, 1] that satisfy  qij yj = 0, i ∈ Bc. j

Show that the mean passage times vi = Ei [τB ] are the smallest nonnegative yi that satisfy 1 + j∈B c qij yj = 0, i ∈ B c . Exercise 21. Passage Values for Uniformized Chains. In the setting of the preceding exercise, for f : S → R, consider the mean values  τB  f (X(t))dt , i ∈ S. vi = Ei 0

4.22 Exercises

329

Assuming λ = supi qi < ∞, show that 'ν −1 ( B  f (Xˆn ) . vi = λ−1 Ei n=0

ˆ n ∈ B}. Specify the where Xˆn is a Markov chain and νB = inf{n ≥ 1 : X ˆ transition probabilities pˆij for Xn in terms of the rates qij for X(t). Show that the vector v has the form ˆ

v = eQ r =

∞  j

ˆ nij rj : i ∈ S Q



n=0

ˆ and the vector r. and specify the matrix Q Exercise 22. Asymptotic Stationarity. Suppose that X(t) is an ergodic CTMC with stationary distribution p and, for t ≥ 0, let Z(t) be a real¯ valued random variable generated by the future {X(u) : u > t}. Let X(t) ¯ ¯ and Z(t) denote these processes when X(t) is a stationary version of X(t). d ¯ Show that (conditioning on X(t)), Z(t) → Z(0), as t → ∞. d ¯ (This asymptotic stationarity statement says in particular that St X → X. A discrete-time version of this is in Proposition 63 in Chapter 1.) Prove the asymptotic independence property that, for t < u, if t → ∞ and u − t → ∞, then P {X(t) = i, X(u) = j} → pi pj . !b For f : S → R and a < b, consider the process Y (t) = a f (X(u + t))du. d

Show that there exists a random variable Y such that Y (t) → Y and give a formula for E[Y ]. ˆ Exercise 23. Let X(t) and X(t) on S denote jump processes as described prior to Proposition 23 with respective parameters (ˆ αi , pˆij , qˆi ) and (αi , pij , qi ), ˆi = αi , qi ≥ qˆi , and where pˆii and pii are in [0, 1). Assume that α pij = qˆi pˆij /qi ,

j = i,

pii = 1 − qˆi (1 − pˆii )/qi .

ˆ Show that X(t) and X(t) are CTMCs that have the same distribution. Exercise 24. Sampling a CTMC. Let X(t) be an ergodic CTMC on S with stationary distribution pi . Suppose this CTMC is sampled (or observed) at times T1 < T2 < . . . that are occurrence times of a renewal process with inter-renewal distribution F . Show that the sampled values X(Tn ) form a Markov chain and specify its transition probabilities. Show that X(Tn ) is ergodic with the same stationary distribution pi that X(t) has. How would you estimate the pi from observations X(T0 ), X(T1 ), . . . , X(Tn )? That is, find an estimator pˆi (n) of pi and show that it is consistent in that pˆi (n) → pi ,

330

4 Continuous-Time Markov Chains

for each i ∈ S, as n → ∞. Is the estimator the same if the sampling times formed a Poisson process or if the inter-sampling times were a constant? Exercise 25. M/M/1 Production-Inventory System. Consider the system shown in Figure 4.2 consisting of an M/M/1 system M that produces units of a product and a warehouse W that houses the units until they are requested. Demands for the product occur at times that form a Poisson process with rate λ. An arriving demand is satisfied from the warehouse if a unit is available, otherwise the demand waits outside of W until a unit arrives from M and then it is satisfied. In either case, the arrival also triggers a unit to be produced at M, where the service rate of single server is μ > λ. Assume the warehouse has a capacity L and, at time 0, W is full and M is empty. Let X(t) denote the number of units in M at time t; this is also the number of demands that are waiting for units. The number of units in W is W (t) = L − X(t). What type of process is X(t)? Specify its stationary distribution. M

> M/M/1 

 - Z W ? Z ZZ ~ Store

-

Fig. 4.2 M/M/1 Production-Inventory System

The system receives a reward at the arrival time of each demand: the reward is R if the demand is satisfied immediately and is r if the demand incurs a wait to be satisfied. Also, there is an inventory cost of h per unit time of holding one unit in inventory. Find the average reward for the system and the average holding cost. Then find the warehouse capacity L that maximizes the average reward minus holding cost. Exercise 26. Suppose (Xn , ξn ) is a Markov chain as in Proposition 23 and as in its proof let ν0 = 0 and νn+1 = min{m > νn : Xm = Xνn }, n ≥ 0. Show that each νn is a stopping time of the chain (Xn , ξn ). Exercise 27. Let X(t) be a birth-death process with state-dependent birth and death rates λi and μi . Let νi = min{n ≥ 1 : Xn = i}. For a cost (or value) function f : S × R+ → R, show that E0

νk



where ηj =

j  k−1  1  f (Xn , ξn ) = (λi + μi )Ei [f (i, ξ1 )]ηi , λ η n=1 j=0 j j i=0

j i=1

λi−1 /μi . First derive an expression for

(4.67)

4.22 Exercises

331

vj = Ej

j+1

ν

 f (Xn , ξn ) ,

0 ≤ j ≤ k,

n=1

based on a first-step analysis and a recursive equation for the vj . Use (4.67) to find an expression for E0 [Tνi ], the mean first passage time to state i. Exercise 28. Counterexamples. Let X(t) be an irreducible CTMC with sojourn rates qi whose embedded chain Xn is a random walk on S = Z+ with transition probabilities p00 = q and pij = p1(j = i + 1) + q1(j = i − 1). Show that any qi are P-regular if p ≤ 1/2. Assuming p < q, show that Xn is ergodic and specify its stationary distribution. In this case, find qi such that X(t) is not ergodic. Assuming p = 1/2, show that Xn is not ergodic, but X(t) is ergodic for qi = 1/ai and 0 < a < 1. Exercise 29. Tandem Network . Suppose X(t) is a Jackson network process for the tandem network shown in Figure 4.3, where arrivals enter node 1 according to a Poisson process with rate λ and each node i consists of a single server with service rate μi . Show that X(t) is ergodic if and only if λ < μi for each i. When it is ergodic, establish its stationary distribution.

λ - 1m

- 2m

- 3m

-

q q q

- mm -

Fig. 4.3 Tandem Network

Now, assume X(t) = (X1 (t), . . . , Xm (t)) is stationary. Use the result in the preceding exercise to find the average duration of time that there are more than L items in the network. Find an expression for P {max1≤i≤m Xi (0) > a}. m Assume μi = μ > λ, for each i, and determine P { i=1 Xi (0) > L}. Exercise 30. Open Acyclic Jackson Network . Suppose X(t) is an ergodic Jackson process representing the open network shown in Figure 4.4. Let pij denote the probability that an item is routed from node i to node j. Show that a solution to the traffic equations is w1 = p01 , w2 = p02 + w1 p12 ,

w3 = p03 + w1 p13 ,

w4 = w2 p24 ,

w5 = w3 .

Assume X(t) is stationary and let Nij (t) denote the number of times an item moves from node i to node j in a time interval [0, t]. Show that Nij is a

332

4 Continuous-Time Markov Chains

 - 1m  Z Z

- 2m >  ? ZZ ~ 3m -

- 4m

-

- 5m

-

Fig. 4.4 Open Acyclic Jackson Network

Poisson process with rate wi λij . In other words, the flow between each pair of nodes in the network is a Poisson process. This property is true for any network in which each item can visit a node at most once. While some of the flows are independent, some of them are dependent. Exercise 31. Input-Output System with Batch Arrivals. Let X(t) denote the quantity of items in an input-output system that operates as follows. Batches of items arrive at times that form a Poisson process with rate λ, and the batch sizes are i.i.d. with geometric distribution (1 − α)αn−1 . The items are served by a single server and the service times are independent exponentially distributed with rate μ > λ/(1 − α). Justify that X(t) is a CTMC and specify its transition rates. Show that its stationary distribution is p i = p0

λ μ



λ + αμ μ

i−1 i ≥ 1.

Exercise 32. Continuation: Batch Services. Consider the system described in the preceding example with the difference that at the end of a service time with i items in the system, min{i, K} items depart, where K is the batch service capacity. So those items present at the start of a service plus all the arrivals during a service time up to the amount K depart at the end of the service. Here Kμ > λ/(1−α). Justify that the quantity of items in the system X(t) is a CTMC with transition rates qij = λ(1 − α)αk−1 1(j = i + k) +μ[1(j = i − B, i > K) + 1(j = 0, i ≤ K)]. Show that its stationary distribution is pi = p0 λr

i−1

/(μ

K−1 

rk ),

i ≥ 1,

k=0

where r is the unique solution (which you need not prove) in (0, 1) of μrK + μ(1 − α)

K−1  k=0

rk = λ + μ.

4.22 Exercises

333

Exercise 33. Central Limit Theorem. Let  X(t) denote an ergodic CTMC with limiting distribution pi = πi /qi / j∈S πj /qj , for i ∈ S, where π is the stationary distribution for Xn , which is ergodic. Consider the functional !t Z(t) = 0 f (X(s)) ds, where f (i) denotes a value per unittime when X(t) is in state i. Theorem 42 showed that t−1 Z(t) → a = i∈S pi f (i), a.s., provided the sum is absolutely convergent. For simplicity, fix i ∈ S and assume X(0) = i. Specify conditions, based on Theorem 65 in Chapter 2, under which (Z(t) − at)/t1/2 → N (0, σ 2 ), d

as t → ∞.

Give an expression for σ 2 using ideas in Example 68 in Chapter 3 and Z(T1 ) − aT1 =

νi 

[f (Xk )Yk − aYk ],

k=1

where νi = min{n ≥ 1 : Xn = i} and Yn is the sojourn time of X(t) in Xn . ˜ be a CTMC Exercise 34. Reversible Multiple Instantaneous Jumps. Let X(t) on S with transition rates q˜ij . Define a CTMC X(t) with transition rates n m m ˜ m = {˜ ˜ij , where Q qij } is the mth product of the matrix qij = m=1 q ˜ = {˜ Q qij } with each qii = 0 and n is fixed. This process is a variation of ˜ ˜ X(t) in which each transition consists of up to n transitions of X(t) occurm ring simultaneously. The compound rate qij represents a “macro” transition ˜ Show that if X(t) ˜ rate for m “instantaneous jumps” of X. is reversible with respect to γ, then X(t) is reversible with respect to γ. Exercise 35. Networks with Variable Waiting Spaces. Consider an m-node open network process Xt = (Xt1 , . . . , Xtm ) that represents the numbers of items at the nodes at time t. Suppose the waiting spaces at the nodes vary such that Yt = (Yt1 , . . . , Ytm ) is the maximum numbers of items allowed at the nodes at time t. Suppose {(Xt , Yt ) : t ≥ 0} is an irreducible CTMC on S = {(x, y) ∈ SX × SY : x ≤ y}, where SX = {x : |x| < ∞} = SY . Assume that its transition rates are ⎧ λjk φj (xj ) if x = Tjk x, y  = y ⎪ ⎪ ⎪ ⎨ and xk < yk for some j, k ∈ M q((x, y), (x , y  )) =  ⎪ q (y, y ) if x = x and y  ≥ x . ⎪ ⎪ Y ⎩ 0 otherwise. The X is an open Jackson process whose node populations are restricted by the process Y with transition rates qY . Assume that the routing rates λjk are reversible with respect to wj , and that qY is reversible with respect to πY .Show that (X, Y ) is reversible with respect to π(x, y) = xjthe process m πY (y) j=1 wj n=1 φ( n)−1 .

334

4 Continuous-Time Markov Chains

Exercise 36. Suppose the transition rates q˜(x, y) represent a CTMC on S that is reversible with respect to γ˜ . Assume the process is subject to the constraint that a transition is possible if and only if h(x, y) ≤ b, for some h : S 2 → R+ and b > 0, and denote the resulting process by X(t). Show that this is a reversible CTMC and specify an invariant measure for it. Exercise 37. Prove that a Jackson process is irreducible if and only if its routing process is irreducible. Recall that λij is irreducible if and only if, for any fixed i = j in M , there are i1 , . . . , i in M such that λii1 λi1 i2 · · · λi j > 0. Exercise 38. Parameters for Closed Jackson Process. The convolution f g of two real-valued functions g and h on Z+ is defined by g h(n) =

n 

g(i)h(n − i),

n ≥ 0.

i=0

For a sequence of such functions g1 , g2 , . . ., show by induction that g1 · · · gm (n) =

m  

gi (xi ),

n ≥ 0, m ≥ 1.

x:|x|=n i=1

Show that the normalizing constant c in Theorem 74 for a closed Jackson network process has the representation c−1 = f1 · · · f m (ν). Associated with a sector of nodes J, define xJ = j∈J xj , and let fJ denote the convolution of the functions {fj , j ∈ J}. Assume the network process X(t) is stationary and let XJ (t) = j∈J Xj (t). Show that, for any disjoint sectors J1 , . . . , J whose union is M , the joint equilibrium distribution of n1 , . . . , n items in these sectors is P {XJ1 (0) = n1 , . . . , XJ (0) = n } = c

 

fJi (ni ),

n1 + · · · + n = ν.

i=1

From these distributions, one can obtain means, variances, covariances, and other items of interest such as expected costs for the process. In particular, show that the mean number of items in a sector J in equilibrium is E[XJ (0)] = c

∞ 

nfJ (n)fJ c (ν − n).

n=1

Exercise 39. Jackson Networks with Feedbacks at Nodes. Consider a Jackson process under the usual assumption that, whenever it is in state x, the time to the next departure from node i is exponentially distributed with rate φi (xi ), but the routing is a little different. Assume that an item departing from node i enters node j with probability p¯ij , independently of everything else, where the probability p¯ii of a feedback may be positive. Justify that the

4.22 Exercises

335

resulting process X(t) is a CTMC and specify its transition rates q(x, Tij x) and exponential sojourn rate q(x) in state x. Exercise 40. Busing System. Items arrive to a waiting station at times that form a Poisson process with rate λ. “Buses” arrive to the station at times that form a Poisson process with rate μ to take items immediately from the system. If a bus arrives and finds the system empty, it departs immediately. Busing is common in computer systems and material handling systems. Assume that the number of items each bus can take is a random variable with the geometric distribution pn−1 (1 − p), n ≥ 1. Also, when there are no items in the queue and an item arrives, then with probability p there is a bus available to take the arrival without delay. Show that if a bus arrives and finds i items waiting, then the actual number Y that departs in a batch has the truncated geometric distribution P {Y = n} = pn−1 (1 − p)1(n < i) + pn−1 1(n = i). Let X(t) denote the number of items in the system at time t. Show that it is a CTMC with transition rates qi,i+1 = λ(1 − p)1(i = 0) + λ1(i ≥ 1), qi,i−n = μpn−1 (1 − p)1(1 ≤ n ≤ i − 1) + μpi−1 1(n = i). Show that X(t) is ergodic if and only if λ < μ + pλ, and in this case, its stationary distribution is pi = p0 (1 − p)λi /(μ + pλ)i ,

i ≥ 1.

Exercise 41. Star-Shaped Network. Let X(t) denote a closed Jackson process with ν items for the star-shaped or central-processor network shown in Figure 4.5. Node 1 is the center node and nodes 2, . . . , m are points of the star such that the routing rates λ1j and λi1 are positive. All the other routing rates are 0. Suppose node 1 operates like an M/M/∞ node with service rate μ1 , and the other nodes operate like M/M/1 nodes with service rate μi for node i. Find a solution wi of the traffic equations, and then find the stationary distribution for X(t). Is this process reversible? Find the equilibrium probability that there are no items at node 1. 2m

6 ?  - 1m  - 3m 5m 6 ? 4m Fig. 4.5 Star-Shaped Network

336

4 Continuous-Time Markov Chains

Exercise 42. Prove that a CTMC X(t) is reversible with respect to p if and only if i, j ∈ S. (4.68) pi pij (t) = pj pji (t), ∞ Hint: Consider the sum pij (t) = n=0 pij (t, n), where pij (t, n) is the probability of X(t) starting in state i and being in j at time t at the n-th state visited by the process. To prove (4.68), it suffices to show (by induction) that i, j ∈ S, n ≥ 0.

pi pij (t, n) = pj pji (t, n),

To prove the converse use pij (t) = qij t + o(t), as t ↓ 0 from Theorem 18. Exercise 43. Sojourn Times in an M/M/1 System. In the context of Exˆ for an arriving item in equilibrium ample 101, show that the sojourn time W and its waiting time W prior to service satisfy ˆ ≤ t} = PT {W ≤ t|W > 0} = 1 − e−(μ−λ)t . PT {W Exercise 44. Finite-Capacity Kelly Network. Consider the network process in Theorem 81 with the modification that arrivals from outside are dependent on the network such that q(x, x + er1 ) = λ(r)φr0 (|x|). This would allow for a finite capacity network by assuming φr0 (n) = 0 for n = νr . For this more general arrival rate, show that the invariant measure as in Theorem 81 would be p(x) =



λ(r)xr frs (x)

rs∈M

|x| 

φr0 (k − 1),

x ∈ S.

k=1

Exercise 45. Multiclass Jackson Network Process. Suppose X(t) is the multiclass network process X(t) in Theorem 82 with service intensities φαi (x) = μαi (xαi ), i = 0, that do not depend on the quantity of items at the nodes. Find an invariant measure for it. Next, consider the modification that α items arrive to the network from outside with rate φα0 (x) = g0 (|x|)hα0 (|xα |), m where |xα | = i=1 xαi is the number of α-items inthe network. Show that xαi an invariant measure for X(t) has the form p(x) = αi∈M wαi fαi (x), with |x|−1

fα0 (x) =



k=0

|xα |−1

g0 (k)



hα0 (k  ).

k =0

Exercise 46. Throughput Rates. Suppose X(t) is the multiclass network process X(t) in Theorem 82 and assume it is ergodic. Find a simple expression for the throughput (or average number of items that jump) from αi to βj

4.22 Exercises

337

ραi,βj = lim t−1 t→∞



1(X(s) = X(s−) − eαi + eβj ).

s≤t

Next, assume the network is closed with ν items in it. Specify the stationary distribution for X(t), and derive the throughput formula ραi,βj = cν c−1 ν−1 wαi λαi,βj , where cν is the normalization constant for the network with ν items in it. Exercise 47. Continuation. Suppose X(t) is the open multiclass network process X(t) in the preceding exercise. Explain how the solutions wαi to the traffic equations and throughput ραi,βj is changed or simplifies under the following two scenarios. (a) Each item carries a class label that does not change: λαi,βj = 0 if α = β. ˜ αβ λ ¯ ij , (b) The class changes are independent of the routing in that λαi,βj = λ ¯ ˜ where λαβ and λij are irreducible transition rates for class changes and node changes, respectively. Exercise 48. Busy Period in an M/G/1 System. Suppose Q(t) is an M/G/1 queueing process with Poisson arrival times 0 < τ1 < τ2 < · · · , Q(0) = 0 and ρ < 1. Consider T = inf{t > τ1 : Q(t) = 0}, which is the time at which the system first becomes empty. Now T = τ1 + Y , where Y is the duration of the busy period for the server. Find E[T ] and show that E[Y ] = ρ/λ(1 − ρ). Exercise 49. Scheduling Patients. Prior to having an operation at a hospital, a patient is given a set of tests depending on the type of procedure (e.g., EKG, sonogram, blood work). This has to be done at least one week before the operation. Patients used to come in at their convenience, usually near noon or late afternoon. To avoid congestion, the hospital required that patients make an appointment for the test. The number of tests was such that patients were scheduled to arrive every u minutes. (The following model is similar to an actual study by Georgia Tech students for a hospital in Atlanta.) As an idealized model, assume that patients do indeed arrive each u minutes, and the durations of the tests are independent exponentially distributed with rate μ < 1/u. The tests are done one at a time and a patient arriving when another one is being tested waits in a queue. Let Q(t) denote the number of patients in the system (waiting or being tested) at time t, and let Wn denote the length of time the nth patient waits in the queue before being ˜ and waiting tested. Find the distributions of the equilibrium queue length Q ˜ time W . Determine what the time between tests u should be under the following criteria. ˜ ≤ w} = .90 for fixed w. (a) Find the shortest u such that P {W ˜ ˜ > m} ≤ .10 for fixed (b) Find the shortest u such that E[W ] ≤ w and P {Q w and m.

338

4 Continuous-Time Markov Chains

Exercise 50. Fork-Join Processing System. Consider an m-node fork-join network that processes jobs as follows. Jobs arrive every u time items (u is a constant) and each job splits into m tasks, which are simultaneously assigned to the m nodes for processing. The nodes operate independently, and each node processes jobs like a single-server G/M/1 system with independent exponential service times with rate μ. When all the m tasks for a job are finished, the job is complete and exits the system. The network is shown in Figure 1.4 in Chapter 1, where the operating rules were different. Assume the system is empty at time 0. Let X(t) = (X1 (t), . . . , Xm (t)) denote the numbers of tasks at the m nodes at time t., and find its limiting distribution. Let Wni denote the time to complete the task at node i for the nth job. Then the sojourn time in the system for the nth job (i.e., the time to process d

the job) is Wn = max{Wn1 , . . . , Wnm }. Show that Wn → W as n → ∞, and determine the distribution of W (which is a product of exponential distributions). Find the distribution of W when G is an exponential distribution. Exercise 51. Extreme-value Process. Claims arrive at an insurance company at times Tn that form a Poisson process N with rate λ. The size Yn of the nth claim that arrives at time Tn has an exponential distribution with rate μ and the claim sizes are independent of their arrival times. The maximum claim up to time t is X(t) = maxk≤N (t) Yk . Justify that X(t) is a CTMC and specify its defining parameters. Show that X(t) → ∞ a.s. as t → ∞, and that d μX(t) − log(λt) → Z, where P {Z ≤ x} = exp{−e−x}, which is the Gumbel distribution. Evaluate the distribution by conditioning on N (t) and using the exponential property that if nan → a, then (1 − an )n → e−a as n → ∞. As an intermediate step, d

justify that μX(Tn ) − log(λn) → Z by evaluating the distribution of μX(Tn ). Exercise 52. Batch-Service System. Consider a batch-service system as in Section 2.12 that processes items as follows. Items arrive to the station according to a Poisson process with rate λ and they enter a queue where they wait to be served. Items are processed in batches, and the number of items in a batch can be any number less than or equal to K (the service capacity). The service times of the batches are independent, exponentially distributed with rate μ independently of everything else. Only one batch can be served at a time and, during a service, additional arrivals join the queue. Batches are served when and only when the queue length is equal or greater than m (a control limit). In particular, if at the end of a service there are i ≥ m items in the queue, then a batch of i ∧ K items is served; and when the queue length is m − 1 and an arrival occurs, then a batch of size m is served. Let Xn denote the queue length at the end of the nth service. Show that the probability of n arrivals during a service is qpn , where p = λ/(λ + μ) and q = 1 − p. Justify that Xn is a Markov chain with

4.22 Exercises

339

transition probabilities pij =

if i < K qpj qpj+k−i (1 − p) if K ≤ i ≤ j − K

and pij = 0 otherwise. Why don’t these probabilities depend on m? Assuming λ < Kμ, prove that Xn is ergodic with stationary distribution πi = (10r)ri , i ≥ 0, where r is the unique solution of qrK+1 − r + p = 0. Exercise 53. Continuation. In the preceding batch-service model, let Tn denote the time of the nth service completion, where T0 = 0. Show that E[T1 |X0 = i] = (m − i)λ−1 1(i ≤ m) + μ−1 . Assume that C + ci is the cost for serving a batch of size i, and hi is the cost per unit time for holding i items in the queue. Show that the expected service plus holding cost in a time interval (Tn , Tn+1 ] given Xn = i is f (i) = C + ci + hiμ−1 + hλμ−2 ,

i > m,

and for i ≤ m, f (i) =

1 1 hm(m − 1)λ−1 − hi(i − 1)λ−1 + C + cm + hmμ−1 + hλμ−2 . 2 2

Justify that the average cost for the system is ∞ 

f (i)πi /

i=0

∞ 

E[T1 |X0 = i]πi .

i=0

This cost is a tractable function φ(m) of the control level m. This cost is minimized at the smallest integer m ≤ K such that Dm ≥ 0, see [33], where Dm = m

1 2

 (m + 1) + λ/μ − c − c2 rm + c(c − λ/μ) − Cλ/h.

Exercise 54. Markov/Poisson Particle System. Consider a particle system in a countable space S similar to the one in Section 3.11 with the following modifications. Each particle moves independently in continuous time according to an ergodic CTMC with transition probabilities pij (t) and stationary distribution pi , i ∈ S. That is, pij (t) is the probability that a particle starting in state i is in state j at time t. Assume the system is empty at time 0 and that particles enter the system according to a space-time Poisson process M on R+ × S, where M ((0, t] × B) is the number of arrivals in (0, t] that enter B ⊆ S, and E[M ((0, t] × {i})] = λtpi . Let Qi (t) denote the quantity of particles in state i at time t. Show that d

(Qi (t) : i ∈ S) → (Qi : i ∈ S),

as t → ∞,

340

4 Continuous-Time Markov Chains

where Qi are independent Poisson random variables with E[Qi ] = λpi . Exercise 55. Continuation. In the setting of the preceding exercise, suppose at time 0 the number of particles in the system is a point process with intensity μ that is independent of the space-time arrival process M of other particles and all the particles move independently as above. The quantity of particles in state i at time t is Xi (t) = Q0i (t) + Qi (t), where Q0i (t) denotes the quantity of particles  in i at time t that were in the system at time 0. Show that E[Q0i (t)] = j∈S pji (t)μ(i), and find αi = limt→∞ E[Xi (t)]. Prove Q0i (t) → Q0i , d

as t → ∞, for i ∈ S,

where Q0i are independent Poisson random variables with E[Qi ] = pi . Show that limt→∞ P {Xi (t) = n} = e−αi (αi )n /n!. Exercise 56. Let X(t) denote the ergodic Markov-renewal process as in Theorem 124. Arguing as in Proposition 41, show that

 τi   1(X(t) = j) dt = πi−1 πj pj μj , Ei 0

∈S

Ei [τi ] =

 Ei

0

πi−1

 j∈S

τi

πj



pj μj

∈S

   f (X(t)) dt = πi−1 πj f (j) pj μj .

Use these formulas to prove Theorem 124.

j∈S

∈S

Chapter 5

Brownian Motion

Brownian motion processes originated with the study by the botanist Brown in 1827 of the movements of particles suspended in water. As a particle is occasionally hit by the surrounding water molecules, it moves continuously in three dimensions. Assuming the infinitesimal displacements of the particle are independent and identically distributed, the central limit theorem would imply that the size of a typical displacement (being the sum of many small ones) is normally distributed. Then the continuous trajectory of the particle in R3 would have increments that are stationary, independent and normally distributed. These are the defining properties of Brownian motion. This diffusion phenomenon, commonly encountered in other contexts as well, gave rise to the theory of Brownian motion and more general diffusion processes. Brownian motion is one of the most prominent stochastic processes. Its importance is due in part to the central limit phenomenon that sums of random variables such as random walks, considered as processes in time, converge to a Brownian motion or to functions of it. Moreover, Brownian motion plays a key role in stochastic calculus involving integration with respect to Brownian motion and semimartingales. This calculus is used to study dynamical systems modeled by stochastic differential equations. For instance, in the area of stochastic finance, stochastic differential equations are the basis for pricing of options by Black-Scholes and related models. Brownian motion is an important example of a diffusion process and it is a Gaussian process as well. Several variations of Brownian motion arise in specific applications, such as Brownian bridge in statistical hypothesis testing. In operations research, the major applications of Brownian motion have been in approximations for queueing systems, and there have also been applications in various areas such as financial models and supply chains. This chapter begins by introducing a Brownian motion as a Markov process that satisfies the strong Markov property, and then characterizes a Brownian motion as a Gaussian process. The second part of the chapter is a study of hitting times of Brownian motion and its cumulative maximum process. This

R. Serfozo, Basics of Applied Stochastic Processes, Probability and its Applications. c Springer-Verlag Berlin Heidelberg 2009 

341

342

5 Brownian Motion

includes a reflection principle for Brownian sample paths, and an introduction to martingales and the optional stopping theorem for them. The next major results are limit theorems: a strong law of large numbers for Brownian motion and its maximum process, a law of the iterated logarithm for Brownian motion, and Donsker’s functional limit theorem showing that Brownian motion is an approximation to random walks. Applications of Donsker’s theorem yield similar Brownian approximations for Markov chains, renewal and regenerative-increment processes, and G/G/1 queueing systems. Other topics include peculiarities of Brownian sample paths, geometric Brownian motion, Brownian bridge processes, multidimensional Brownian motion, Brownian/Poisson particle process, and Brownian motion in a random environment.

5.1 Definition and Strong Markov Property Recall that a random walk in discrete time on the integers is a Markov chain with stationary independent increments. An analogous process in continuous time on R is a Brownian motion. This section introduces Brownian motion as a real-valued Markov process on the nonnegative time axis. Its distinguishing features are that it has stationary, independent, normally-distributed increments and continuous sample paths. It also satisfies the strong Markov property. We begin by describing a “standard” Brownian motion, which is also called a Wiener process. Definition 1. A real-valued stochastic process B = {B(t) : t ∈ R+ } is a Brownian motion if it satisfies the following properties. (i) B(0) = 0 a.s. (ii) B has independent increments and, for s < t, the increment B(t) − B(s) has a normal distribution with mean 0 and variance t − s. (iii) The paths of B are continuous a.s. Property (ii) says that a Brownian motion B has stationary, independent increments. From this one can show that B is a Markov process. Consequently, a Brownian motion is a diffusion process — a Markov process with continuous sample paths. The next section establishes the existence of Brownian motion as a special type of Gaussian process. An introduction to Brownian motion in Rd is in Section 5.14. Because the increments of a Brownian motion B are stationary, independent and normally distributed, its finite-dimensional distributions are tractable. The normal density of B(t) with mean 0 and variance t is 1 −x2 /2t fB(t) (x) = √ e . 2πt

5.1 Definition and Strong Markov Property

343

Denoting this density by ϕ(x; t), it follows by induction and properties (i) and (ii) that, for 0 = t0 < t1 < · · · < tn and x0 = 0, the joint density of B(t1 ), . . . , B(tn ) is fB(t1 ),...,B(tn ) (x1 , . . . , xn ) =

n 

ϕ(xm − xm−1 ;

$ tm − tm−1 ).

(5.1)

m=1

Another nice feature is that the covariance between B(s) and B(t) is E[B(s)B(t)] = s ∧ t.

(5.2)

This follows since, for s < t, Cov(B(s), B(t)) = E[B(s)B(t)] = E[B(s)[(B(t) − B(s)) + B(s)]] = E[B(s)2 ] = s. Several elementary functions of a Brownian motion are also Brownian motions; see Exercise 1. Here is an obvious example. Example 2. Symmetry Property. The process −B(t), which is B reflected d about 0, is a Brownian motion (i.e., −B = B). As a generalization of a standard Brownian motion B, consider the process X(t) = x + μt + σB(t),

t ≥ 0.

Any process equal in distribution to X is a Brownian motion with drift: x is its initial value, μ is its drift coefficient, and σ > 0 is its variation. Many properties of a Brownian motion with drift readily follow from properties of a standard Brownian motion. For instance, X has stationary, independent increments and X(t + s) − X(s) is normally distributed with mean μt and variance σ 2 t. Drift and variability parameters may be useful in Brownian models for representing certain trends and volatilities. Now, let us see how Brownian motions are related to diffusion processes. Generally speaking, a diffusion process is a Markov process with continuous paths. Most diffusion processes in applications, however, have the following form. Suppose that {X(t) : t ≥ 0} is a real-valued Markov process with continuous paths a.s. that satisfies the following properties: For each x ∈ R, t ≥ 0, and ε > 0, lim h−1 P {|X(t + h) − X(t)| > |X(t) = x} = 0, h↓0

lim h−1 E[X(t + h) − X(t)|X(t) = x] = μ(x, t), h↓0

lim h−1 E[(X(t + h) − X(t))2 |X(t) = x] = σ(x, t), h↓0

344

5 Brownian Motion

where μ and σ are functions on R × R+ . The X is a diffusion process on R with drift parameter μ(x, t) and diffusion parameter σ(x, t). As a prime example, a Brownian motion with drift X(t) = μt + σB(t), is a diffusion process whose drift and diffusion parameters μ and σ are independent of x and t. Many functions of Brownian motions are also diffusions (e.g., the Ornstein-Uhlenbeck and Bessel Processes in Examples 8 and 64). We end this introductory section with the strong Markov property for Brownian motion. Suppose that B is a Brownian motion on a probability space (Ω, F , P ). Let FtB ⊆ F be the σ-field generated by {B(s) : s ∈ [0, t]}, and assume F0B includes all sets of P -probability 0 to make it complete. A stopping time of the filtration FtB is a random time τ , possibly infinite, such that {τ ≤ t} ∈ FtB , t ∈ R+ . The σ-field of events up to time τ is FτB = {A ∈ F : A ∩ {τ ≤ t} ∈ Ft , t ∈ R+ }. If τ1 and τ2 are two FtB -stopping times and τ1 ≤ τ2 , then FτB1 ⊆ FτB2 . Theorem 3. If τ is an a.s. finite stopping time for a Brownian motion B, then the process B  (t) = B(τ + t) − B(τ ), t ∈ R+ , is a Brownian motion independent of FτB . Proof. We will prove this only for a stopping time τ that is a.s. bounded (τ ≤ u a.s. for some u > 0). Clearly B  has continuous sample paths a.s. It remains to show that the increments of B  are independent and independent of FτB , and B  (s+t)−B  (s) is normally distributed with mean 0 and variance t. These properties will follow upon showing that, for any 0 ≤ t0 < · · · < tn , and u1 , . . . , un in R+ , 1

E[eSn |FτB ] = e 2

n i=1

u2i (ti −ti−1 )

a.s.,

n where Sn = i=1 ui [B  (ti ) − B  (ti−1 )]. The proof of (5.3) will be by induction. First note that  

 E[eSn+1 |FτB ] = E eSn E[eSn+1 −Sn |FτB+tn ]FτB .

(5.3)

(5.4)

Now, since τ + tn is a bounded stopping time, using Example 26 below, E[eSn+1 −Sn |FτB+tn ] = E[eun+1 [B(τ +tn+1 )−B(τ +tn)] |FτB+tn ] 1

2

= e 2 un+1 (tn+1 −tn ) . This expression with n = 0 and S0 = 0 proves (5.3) for n = 1. Next assuming (5.3) is true for some n, then using the last display and (5.3) in (5.4) yields (5.3) for n + 1.

5.2 Brownian Motion as a Gaussian Process

345

5.2 Brownian Motion as a Gaussian Process This section shows that Brownian motion is a special type of Gaussian Process. Included is a proof of the existence of Gaussian processes, which leads to the existence of Brownian motion. We begin with a discussion of multivariate normal distributions. Suppose that X1 , . . . , Xn are normally distributed (not necessarily independent) random variables with means m1 , . . . , mn . Then clearly, for u1 , . . . , un ∈ R, E

n

 i=1

  u i Xi = u i mi ,

n  

 Var u i Xi = ui uj cij ,

i

i=1

i

(5.5)

j

where cij = Cov(Xi , Xj ). The vector (X1 , . . . , Xn ) is said to have a multivarin ate normal (or Gaussian) distribution if i=1 ui Xi has a normal distribution for any u1 , . . . , un in R. In light of (5.5), the (X1 , . . . , Xn ) has a multivariate normal distribution if and only if its moment generating function has the form

n  % & 1  E e i=1 ui Xi = exp u i mi + ui uj cij , ui ≥ 0. (5.6) 2 i j i The vector (or distribution) associated with the moment generating function (5.6) is called nondegenerate if the n × n matrix C = {cij } has rank n. In this case, the joint density of (X1 , . . . , Xn ) is & % 1  1 f (x1 , . . . , xn ) = $ cˆij (xi − mi )(xj − mj ) , (5.7) exp − 2 i j (2π)n |C| where {ˆ cij } is the inverse of C and |C| is its determinant. It turns out that any multivariate normal vector can be represented by a nondegenerate one as follows. When C does not have rank n, it follows by a property of symmetric matrices that there exists a k × n matrix A with transpose At , where k ≤ n, such that C = At A. Let X denote the multivariate normal vector as a 1 × n matrix with mean vector m. Suppose that Y is a 1 × k nondegenerate multivariate vector of i.i.d. random variables Y1 , . . . , Yk that are normally distributed with mean 0 and variance 1. Then the multivariate normal vector, has the representation d

X = m + Y A.

(5.8)

This equality in distribution follows because the moment generating function of m + Y A is equal to (5.6). Indeed,

346

5 Brownian Motion

⎡

⎤ *  n n k & %    v , E ⎣exp u i mi + ui ( aji )Yj ⎦ = exp u i mi + 2 i=1 i=1 j=1 i where interchanging the summations and using C = At A, ⎡ + n , ⎤ + n ,2 k k     ui aji Yj ⎦ = ui aji v = Var ⎣ j=1

=

i=1

+ n k   j=1

ui atij

i=1

,+

j=1 n 

i=1

,

u aj

=1

=

 i

ui uj cij .

j

A major characteristic of a Brownian motion is that its finite-dimensional distributions are multivariate normal. Other stochastic processes with this property are as follows. Definition 4. A stochastic process X = {X(t) : t ∈ R+ } is a Gaussian process if (X(t1 ), . . . , X(tn )) has a multivariate normal distribution for any t1 , . . . , tn in R+ . Discrete-time Gaussian processes are defined similarly.  A process X is Gaussian, of course, if and only if ni=1 ui X(ti ) has a normal distribution for any t1 , . . . , tn in R+ and u1 , . . . , un in R. A Gaussian process X is called nondegenerate if its covariance matrix cij = Cov(X(ti ),X(tj )) has rank n, for any t1 , . . . , tn in R+ . In that case, X(t1 ), . . . , X(tn ) has a multivariate normal density as in (5.7). The next result establishes the existence of Gaussian processes. It also shows that the distribution of a Gaussian process is determined by its mean and covariance functions. So two Gaussian processes are equal in distribution if and only if their mean and covariance functions are equal. Let c(s, t) be a real-valued function on R2+ that satisfies the following properties: c(s, t) = c(t, s),

s, t, ∈ R+ . (Symmetric)

For any finite set I ⊂ R+ and ut ∈ R,  us ut c(s, t) ≥ 0. (Nonnegative-definite) t∈I s∈I

Theorem 5. For any real-valued function m(t) and the function c(s, t) described above, there exists a Gaussian process {X(t) : t ∈ R+ } defined on a probability space (Ω, F , P ) = (RR+ , B R+ , P ), with E[X(t)] = m(t) and Cov(X(s), X(t)) = c(s, t),

s, t, ∈ R+ .

Furthermore, the distribution of this process is determined by the functions m(t) and c(s, t).

5.2 Brownian Motion as a Gaussian Process

347

Proof. We begin by defining finite-dimensional probability measures μI for a process on (RR+ , B R+ , P ). For any finite subset I in R+ , let μI be the probability measure specified by μI (×t∈I At ), At ∈ B, t ∈ I, that has the joint normal moment generating function G(uI ) = exp

%

ut m(t) +

t∈I

& 1  us ut c(s, t) , 2

uI = (ut : t ∈ R+ ).

t∈I s∈I

Note that for I ⊆ J, if ut = 0, t ∈ J\I.

G(uJ ) = G(uI ),

Consequently, the joint normal distributions μI satisfy the consistency condition that, for any I ⊆ J for finite J and At ∈ B, t ∈ J, if At = R for t ∈ J\I.

μJ (×t∈J At ) = μI (×t∈I At ),

(5.9)

Then it follows by Kolmogorov’s extension theorem (Theorem 5 in the Appendix), that there exists a stochastic process {X(t) : t ∈ R+ } defined on the probability space (Ω, F , P ) = (RR+ , B R+ , P ), whose finite-dimensional probability measures are given by μI . Since the μI are determined by m(t) and c(s, t), so is the distribution of X. Moreover, from the moment generating function for the μI , it follows that E[X(t)] = m(t),

Cov(X(s), X(t)) = c(s, t).

Brownian motion is a quintessential example of a Gaussian process. Proposition 6. A Brownian motion with drift X(t) = μt + σB(t), t ≥ 0, is a Gaussian process with continuous sample paths a.s. starting at X(0) = 0 and its mean and covariance functions are E[X(t)] = μt,

Cov(X(s), X(t)) = σ 2 (s ∧ t),

s, t ∈ R+ .

Proof. For any 0 = t0 < t1 < · · · < tn , letting Yi = X(ti ) − X(ti−1 ), we have + n , n n n i      ui X(ti ) = ui Yi = ui Yk , u1 , . . . , un ∈ R. i=1

i=1

k=1

k=1

i=k

Now the increments Yi are independent, normally distributed random variables with mean 0 and variance σ 2 (ti − ti−1 ). Then the last double-sum term has a normal distribution, and so (X(t1 ), . . . , X(tn )) has a multivariate normal distribution. Hence X is a Gaussian process, and its mean and variance are clearly as shown. The preceding characterization is useful for verifying that a process is a Brownian motion, especially when the multivariate normality condition

348

5 Brownian Motion

is easy to verify (as in Exercise 2). There are other interesting Gaussian processes that do not have stationary independent increments; see Example 8 below and Exercise 10. One approach for establishing the existence of a Brownian motion is to construct it as a Gaussian process as follows. Theorem 7. There exists a stochastic process {B(t) : t ≥ 0} defined on a probability space (Ω, F , P ) = (RR+ , B R+ , P ) such that B is a Brownian motion. Sketch of Proof. Let {B(t) : t ≥ 0} be a Gaussian process as constructed in the proof of Theorem 5 with the special Brownian functions m(t) = 0 and c(s, t) = s ∧ t. A major result (whose proof is omitted) says that this process has stationary independent increments, and B(t) − B(s), for s < t, is normally distributed with mean 0 and variance t− s. A second step is needed, however, to justify that such a process has continuous sample paths. d Since B(t) − B(s) = (t − s)1/2 B(1), for s < t, the process satisfies E[|B(t) − B(s)|a ] = (t − s)a/2 E[|B(1)|a ] < ∞,

a > 0.

Using this property, another major result shows that B can be chosen so that its sample paths are continuous, and hence it is a Brownian motion. The results that complete the preceding two steps are proved in [64]. A Brownian motion is an example of a Markov process with continuous paths that is a Gaussian process. Are there Markov processes with continuous paths (i.e., diffusion processes), other than Brownian motions, that are Gaussian? Yes there are — here is an important example. Example 8. An Ornstein-Uhlenbeck Process is a stationary Gaussian process {X(t) : t ≥ 0} with continuous sample paths whose mean function is 0 and whose covariance function is Cov(X(s), X(t)) =

σ 2 −α|s−t| e , 2α

s, t ≥ 0,

where α and σ are positive. This process as proved in [61] is the only stationary Gaussian process with a continuous covariance function that is a Markov process. (Exercise 9 shows that a Gaussian process X is stationary if and only if its mean function is a constant and its covariance function Cov(X(s), X(t)) only depends on |t − s|.) It is interesting that the process X is also a function of a Brownian motion B in that X is equal in distribution to the process Y (t) = σe−αt B(eαt /2α),

t ≥ 0.

To see this, note that Y has continuous sample paths and clearly it is Gaussian since B is. In addition, E[Y (t)] = 0 for each t and, for s < t,

5.3 Maximum Process and Hitting Times

349

 Cov(Y (s), Y (t)) = σ 2 e−α(s+t) E B(eαs /2α)B(eαt /2α) =

σ 2 −α(t−s) e . 2α

Consequently, Y is a stationary Gaussian process and it has the same mean d and covariance functions as X. Hence Y = X. The Ornstein-Uhlenbeck process X defined above satisfies the stochastic differential equation dX(t) = −αX(t)dt + σdB(t).

(5.10)

The example above assumes that X(0) has a normal distribution with mean 0 and variance σ 2 . The solution of this equation is  t e−α(t−s) dB(s). X(t) = X(0)e−αt + σ 0

The stochastic differential equation and the integral with respect to Brownian motion, which is beyond the scope of this work, is discussed in [61, 64]. Scientists realized that the Brownian motion representation for a particle moving in a medium was an idealized model in that it does not account for friction in the medium. To incorporate friction in the model, Langevin 1908 proposed that the Ornstein-Uhlenbeck process X could represent the velocity of a particle undergoing a Brownian motion subject to friction. He assumed that the rate of change in the velocity satisfies (5.10), where −αX(t)dt models the change due to friction; the friction works in the opposite direction to the velocity and α is the coefficient of friction divided by the mass of the particle. The stochastic process for this model was formalized later by Ornstein and Uhlenbeck 1930 and Doob 1942.

5.3 Maximum Process and Hitting Times For a Brownian motion B, its cumulative maximum process is M (t) = max B(s), s≤t

t ≥ 0.

This process is related to the hitting times τa = inf{t > 0 : B(t) = a},

a ∈ R.

Namely, for each a ≥ 0 and t, {τa ≤ t} = {M (t) ≥ a}.

(5.11)

350

5 Brownian Motion

In other words, the distribution of the maximum process is determined by that of the hitting times and vice versa. This section presents expressions for these distributions and an important property of the hitting times. We begin with a preliminary fact. Remark 9. The hitting time τa is an a.s. finite stopping time of B. The τa is a stopping time since B has continuous paths a.s. Its finiteness follows by Theorem 32 below, which is proved by the martingale optional stopping theorem in the next section. The finiteness also follows by the consequence (5.30) of the law of the iterated logarithm in Theorem 38 below. The first result is a reflection principle that an increment B(t) − B(τ ) after a stopping time τ has the same distribution as the reflected increment d −(B(t) − B(τ )). This is basically the symmetry property B = −B in Example 2 manifested at the stopping time τ . A version of this principle for stochastic processes is in Exercises 20 and 21. Proposition 10. (Reflection Principle) If τ is an a.s. finite stopping time of B, then, for any a and t, P {B(t) − B(τ ) ≤ a, τ ≤ t} = P {B(t) − B(τ ) ≥ −a, τ ≤ t}. Proof. Letting B  (t) = B(τ + t) − B(τ ), t ≥ 0, we can write B(t) − B(τ ) = B  (t − τ ),

for τ ≤ t.

(5.12)

By the strong Markov property in Theorem 3, B  is a Brownian motion independent of Fτ . Using this and (5.12) along with the symmetry property d

B  = −B  and {τ ≤ t} ∈ Fτ ,

 P {B(t) − B(τ ) ≤ a, τ ≤ t} = E P {B  (t − τ ) ≤ a|τ ≤ t, Fτ } P {τ ≤ t}

 = E P {−B  (t − τ ) ≤ a|τ ≤ t} P {τ ≤ t} = P {B  (t − τ ) ≥ −a, τ ≤ t}. Then using (5.12) in the last probability completes the proof. We will now apply the reflection principle to obtain an expression for the joint distribution of B(t) and M (t). Theorem 11. For x < y and y ≥ 0, P {B(t) ≤ x, M (t) ≥ y} = P {B(t) ≥ 2y − x}, P {M (t) ≥ y} = 2P {B(t) ≥ y}. d

Furthermore, M (t) = |B(t)| for each t, and the density of M (t) is

(5.13)

5.3 Maximum Process and Hitting Times

351

2 −x2 /2t fM(t) (x) = √ , e 2πt Hence E[M (t)] =

$

x ≥ 0.

Var[M (t)] = (1 − 2/π)t.

2t/π,

(5.14)

Proof. Assertion (5.13) follows since by (5.11) and Proposition 10 with τ = τy , B(τ ) = y, and a = x − y, we have, for x ≤ y and y ≥ 0, P {B(t) ≤ x, M (t) ≥ y} = P {B(t) ≤ x, τy ≤ t} = P {B(t) ≥ 2y − x, τy ≤ t} = P {B(t) ≥ 2y − x}. The last equality is because 2y − x ≥ y and {B(t) ≥ 2y − x} ⊆ {B(t) ≥ y} ⊆ {τy ≤ t}. Next, using what we just proved with x = y, we have P {M (t) ≥ y} = P {B(t) ≤ y, M (t) ≥ y} + P {B(t) ≥ y, M (t) ≥ y} = 2P {B(t) ≥ y}. Taking the derivative of this with respect to y yields the density of M (t). In d addition, 2P {B(t) ≥ y} = P {|B(t)| ≥ y} implies M (t) = |B(t)|. Exercise 11 proves (5.14). d

Even though M (t) = |B(t)| for each t, the processes M and |B| are not equal in distribution; M is nondecreasing while |B| is not. Exercise 19 points d

out the interesting equality in distribution M = M − B for the processes. d Because of the reflection property −B = B of a Brownian motion B, its minima is also a reflection of its maxima. Remark 12. The minimum process for B is M (t) = min B(s), s≤t

t ≥ 0. d

It is related to the maximum process M by M = −M . Hence P {M (t) ≤ a} = 2P {B(t) ≥ −a},

a < 0.

d

That M = −M follows by M (t) = − maxs≤t −B(t) and the reflection propd

erty −B = B. Also, Theorem 11 yields the distribution of M (t). We will now obtain the distribution of the hitting time τa from Theorem 11. This result is also a special case of Theorem 32 for hitting times for a Brownian motion with drift, which also contains the Laplace transform of τa and shows that E[τa ] = ∞.

352

5 Brownian Motion

Corollary 13. For any a ≥ 0, √ P {τa ≤ t} = 2[1 − Φ(a/ t)],

t > 0,

where Φ is the standard normal distribution. Hence, the density of τa is fτa (t) = √

a 2πt3

2

e−a

/2t

, d

Proof. From (5.11), Theorem 11, and B(t) =

t > 0. √

tB(1), we have

P {τa ≤ t} = P {M (t) ≥ a}

√ = 2P {B(t) ≥ a} = 2[1 − Φ(a/ t)].

Taking the derivative of this yields the density of τa . The family of hitting times {τa : a ≥ 0} for B is an important process in its own right. It is the non-decreasing left-continuous inverse process of the maximum process M since τa = inf{t : B(t) = a} = inf{t : M (t) = a}. By Corollary 13, we know the density of τa and that E[τa ] = ∞. Here is more information about these hitting times. Proposition 14. The process {τa : a ≥ 0} has stationary independent increments and, for a < b, the increment τb − τa is independent of Fτa and it is equal in distribution to τ(b−a) . Proof. Since τb − τa = inf{t : B(τa + t) − B(τa ) = b − a}, it follows by the strong Markov property at τa that τb − τa is independent of Fτa and it is equal in distribution to τb−a . Also, it follows by an induction argument that {τa : a ≥ 0} has stationary independent increments.

5.4 Special Random Times In this section, we derive arc sine and arc cosine probabilities for certain random times of Brownian motion by applying properties of the maximum process. We first consider two random times associated with a Brownian motion B on the interval [0, 1] and its maximum process M (t) = maxs≤t B(s). These times have the same arc sine distribution, which is discussed in Exercise 14. Theorem 15. (L´evy Arc Sine Law) For a Brownian motion B on [0, 1], the time τ = inf{t ∈ [0, 1] : B(t) = M (1)} has the arc sine distribution

5.4 Special Random Times

353

P {τ ≤ t} =

√ 2 arcsin t, π

t ∈ [0, 1].

(5.15)

In addition, the time τ  = sup{t ∈ [0, 1] : B(t) = 0} has the same distribution. Proof. First note that, for t ≤ 1, τ ≤t

⇐⇒

max B(s) − B(t) ≥ max B(s) − B(t). s≤t

t≤s≤1

Denote the last inequality as Y1 ≥ Y2 and note that these random variables are independent since B has independent increments. Now, by the translation and symmetry properties of B and Theorem 11, Y1 = M (t) = |B(t)| = t1/2 |Z1 |, d

d

d

Y2 = M (1 − t) = |B(1 − t)| = (1 − t)1/2 |Z2 |, d

d

d

where Z1 and Z2 are normal random variables with mean 0 and variance 1. From these observations, we have P {τ ≤ t} = P {Y1 ≥ Y2 } = P {tZ12 ≥ (1 − t)Z22 } = P {Z22 /(Z12 + Z22 ) ≤ t},

(5.16)

where we may take Z1 and Z2 to be independent. Then (5.15) follows since the last probability, due to the symmetry property of the normal distribution, is the arsine distribution by Exercise 14. Next, note that by Remark 12 on the minimum process, we have P {τ  < t} = P { max B(s) > 0} + P { min B(s) < 0} t≤s≤1

t≤s≤1

= 2P {Y > −B(t)}. where Y = maxt≤s≤1 B(s) − B(t) is independent of B(t). By the symmetry of B and Theorem 11, we have −B(t) = B(t) = t1/2 Z1 , d

d

Y = M (1 − t) = |B(1 − t)| = (1 − t)1/2 |Z2 |, d

d

d

where Z1 and Z2 are normal random variables with mean 0 and variance 1. Assuming Z1 and Z2 are independent, the preceding observations and (5.16) yield P {τ  < t} = 2P {(1 − t)1/2 |Z2 | < t1/2 Z1 } = P {(1 − t)Z22 < tZ12 } = P {τ ≤ t}. This proves that τ  also has the arc sine distribution.

354

5 Brownian Motion

Next, we consider the event that a Brownian motion returns to the origin 0 in a future time interval. Theorem 16. The event A that a Brownian motion B hits 0 in a time interval [t, u] has the probability P (A) =

$ 2 arccos t/u, π

where 0 < t < u.

(5.17)

Proof. For t < u and u = 1, it follows by Theorem 15 that P (A) = P {τ > t} = 1 −

√ √ 2 2 arcsin t = arccos t. π π

The proof for u = 1 is Exercise 15.

5.5 Martingales A martingale is a real-valued stochastic process defined by the property that the conditional mean of an “increment” of the process conditioned on past information is 0. A random walk and Brownian motion whose mean step sizes are 0 have this property. However, the increments of a martingale are generally dependent, unlike the independent increments of a random walk or Brownian motion. Martingales are used for proving convergence theorems, analyzing hitting times of processes, finding optimal stopping rules, and providing bounds for processes. Moreover, they are key tools in the theory of stochastic differential equations. In this section, we introduce martingales and discuss several examples associated with Brownian motion and compound Poisson processes. We also present the important submartingale convergence theorem. The next two sections cover the optional stopping theorem for martingales and its applications to Brownian motion. Throughout this section, X = {X(t) : t ≥ 0} will denote a realvalued continuous-time stochastic process that has right-continuous paths and E[ |X(t)| ] < ∞, t ≥ 0. Associated with the underlying probability space (Ω, F, P ) for the process X, there is a filtration {Ft : t ≥ 0}, which is a family of σ-fields contained in F that is increasing (Fs ⊆ Ft , s ≤ t) and right-continuous (FtB = ∩u>t FuB ), and F0 contains all events with P probability 0. Furthermore, the process X is adapted to the filtration Ft in that {X(t) ≤ x} ∈ Ft , for each t and x. Definition 17. The process X is a martingale with respect to Ft if E[X(t)|Fs ] = X(s) a.s. The process X is a submartingale if

0 ≤ s < t.

(5.18)

5.5 Martingales

355

E[X(t)|Fs ] ≥ X(s) a.s.

0 ≤ s < t.

If the inequality is reversed, then X is a supermartingale. Taking the expectation of (5.18) yields the characteristic of a martingale that E[X(t)] = E[X(s)], s ≤ t. The martingale condition (5.18) is equivalent to E[X(t) − X(s)|Fs ] = 0, which says that the conditional mean of an increment conditioned on the past is 0. A classic illustration of a martingale is the value X(t) of an investment (or the fortune of a gambler) at time t in a marketplace described by the events in Ft . The martingale property (5.18) says that the investment is subject to a “fair market” in that its expected value at any time t conditioned on the environment Fs up to some time s < t is the same as the value X(s). On the other hand, the submartingale property implies that the market is biased toward “upward” movements of the value X resulting in E[X(t)] ≥ X(s) a.s., for s ≤ t. Similarly, the supermartingle property implies “downward” movements resulting in E[X(t)] ≤ X(s) a.s. In typical applications, Ft = FtY , which is the σ-field generated by the events of a right-continuous process {Y (s) : s ≤ t} on a general state space. In this setting, we say that X is a martingale with respect to Y . In some instances, it is natural that X is a martingale with respect to the filtration Ft = FtX of its own history. Martingales in discrete time are defined similarly. In particular, real-valued random variables Xn with E[ |Xn | ] < ∞ form a martingale with respect to increasing σ-fields Fn if E[Xn+1 |Fn ] = Xn ,

n ≥ 0.

The Xn is a submartingale or supermartingale if the equality is replaced by ≥ or ≤, respectively. Standard examples are sums and products of independent random variables; see Exercise 30. Note that a Brownian motion B is a martingale with respect to itself since, for s ≤ t, 

  E[B(t)|FsB ] = E B(t) − B(s)B(s) + B(s) = B(s). Similarly, if X(t) = x + μt + σB(t) is a Brownian motion with drift, then E[X(t)|FsB ] = μ(t − s) + X(s),

s ≤ t.

356

5 Brownian Motion

Therefore, X is a martingale, submartingale, or supermartingale with respect to B according as μ is = 0, > 0, or < 0. We will also encounter several functions of Brownian motion that are martingales of the following type. Example 18. Martingales For Processes with Stationary Independent Increments. Suppose that Y is a real-valued process that has stationary independent increments and, for simplicity, assume that Y (0) = 0. Suppose that the moment generating function ψ(α) = E[eαY (1) ] exists for α in a neighborhood of 0, and that E[eαY (t) ] as a function of t is continuous at 0, for fixed α. Then by Exercise 7, E[eαY (t) ] = ψ(α)t and E[Y (t)] = at,

Var[Y (t)] = bt,

where a = E[Y (1)] and b = Var[Y (1)]. For instance, Y may be Brownian motion with drift, a Poisson process or a compound Poisson process. An easy check shows that two martingales with respect to Y are Y (t) − at,

(Y (t) − at)2 − bt,

and

t ≥ 0.

The means of these martingales are 0. Next, consider the process Z(t) = eαY (t) /ψ(α)t ,

t ≥ 0.

Clearly Z(t) is a deterministic, nonnegative function of {Y (s) : s ≤ t}, and E[Z(t)] = 1. Then Z is a martingale (sometimes called an exponential martingale) with respect to Y . Indeed, E[Z(t)|FsY ] = Z(s)

E[eα(Y (t)−Y (s)) |Z(s)] = Z(s). ψ(α)t−s

Example 19. Martingales for Brownian Motion. For a Brownian motion with drift Y (t) = x + μt + σB(t), the preceding example justifies that the following functions of Y are martingales with respect to B: (Y (t) − x − μt)2 − σ 2 t,

−c2 σ2 t/2

ec[Y (t)−x−μt]−e

In particular, B(t)2 − t and ecB(t)−c

2

t/2

,

t ≥ 0, c = 0.

are martingales with respect to B.

Having a constant mean suggests that a martingale should also have a nice limiting behavior. According to the next major theorem, many submartingales as well as martingales converge a.s. to a limit. This result for discrete-time processes also holds in continuous-time. Theorem 20. (Submartingale Convergence) If Xn is a martingale, or a submartingale that satisfies supn E[Xn+ ] < ∞, then there exists a random variable X with E[ |X| ] < ∞ such that Xn → X a.s. as n → ∞.

5.5 Martingales

357

This convergence can be viewed as an extension of the fact that a nondecreasing sequence of real numbers that is bounded converges to a finite limit. For a submartingale, the nondecreasing tendency is E[Xn+1 |Fn ] ≤ Xn , and a bound on E[Xn+ ] is enough to ensure convergence a.s. — the submartingale itself need not be nondecreasing. The theorem establishes the existence of the limit X, but it does not specify its distribution. Properties of X can sometimes be derived in specific cases depending on characteristics of Xn . In addition to the convergence Xn → X a.s., it follows by Theorem 15 in the Appendix that E[ |Xn − X| ] → 0 as n → ∞ when the Xn are uniformly integrable. Although Theorem 20 is a major result, we will only give the following example since it is not needed for our results. For a proof and other applications, see for instance [37, 61, 62, 64]. Example 21. Doob Martingale. Let Z be a random variable with E[ |Z| ] < ∞, and let Fn be an increasing filtration on the underlying probability space for Z. The conditional expectation Xn = E[Z |Fn ],

n ≥ 1,

is a martingale with respect to Fn . Then by Theorem 20 X = lim E[Z|Fn ] exists a.s. n→∞

That Xn is a martingale follows since

 E[ |Xn | ] ≤ E E[ |Z| |Fn ] = E[ |Z| ] < ∞, 

  E[Xn+1 |Fn ] = E E[Z |Fn+1 ] Fn ] = E[Z |Fn ] = Xn . Consider the case Xn = E[Z |Y1 , . . . , Yn ] in which Xn is a martingale with respect to Yn . For instance, Xn could be an estimate for the mean of Z based on observations Y1 , . . . , Yn associated with Z. By an additional argument it follows that the limit of Xn is X = E[Z |Y1 , Y2 , . . .]. Therefore, E[Z |Y1 , . . . , Yn ] → E[Z |Y1 , Y2 , . . .] a.s. as n → ∞. In particular, if Z is the indicator of an event A in the σ-field generated by Y1 , Y2 , . . ., then P (A |Y1 , . . . , Yn ) → P (A)

a.s. as n → ∞.

358

5 Brownian Motion

5.6 Optional Stopping of Martingales This section presents the optional stopping theorem for martingales. It was instrumental in the proof of the strong Markov property for Brownian motion in Theorem 3; the next section uses it to analyze hitting times of Brownian motion. For the following discussion, suppose that X is a martingale with respect to a filtration Ft , and that τ is a stopping time of the filtration: {τ ≤ t} ∈ Ft , for each t. We will now address the following questions: Is the martingale property, E[X(t)] = E[X(0)] also true when t is a stopping time? More generally, is E[X(σ)] = E[X(τ )] true for any stopping times σ and τ ? The optional stopping theorem below says that E[X(τ )] = E[X(0)] is indeed true for a bounded stopping time τ . A corollary is that the equality is also true for a finite stopping time when X satisfies certain bounds. This would imply, for instance, that the expected value of an investment in a fair market at the stopping time is the same as the initial value. In other words, in this fair market, there would be no benefit for the investor to choose to stop and freeze his investment at a time depending only on the past history of the market (independent of the future). There are several optional stopping theorems with slightly different assumptions. For our purposes, we will use the following version from [61]. Its discrete-time version is Theorem 28 below. Theorem 22. Associated with the martingale X, assume that σ and τ are stopping times of Ft such that τ is bounded a.s. Then X(σ ∧ τ ) = E[X(τ )|Fσ ]

a.s.

(5.19)

Hence E[X(τ )] = E[X(0)]. If σ is also bounded, then E[X(σ)] = E[X(τ )]. Proof. Our proof for this continuous time setting uses an approximation based on the analogous discrete-time result in Theorem 28 below. For a fixed ¯ k = X(k2−n ), k ∈ Z+ . Clearly X ¯ k is a discrete-time martingale n ∈ Z+ , let X with respect to F¯k = Fk2−n . Define σn = 2n σ + 1/2n ,

τn = 2n τ + 1/2n.

 Now σm = 2n σm for fixed m, and τn = 2n τn are integer-valued stopping times of F¯k . Then by Theorem 28 below,

¯ σ ∧τ  = E[X ¯ τ  |F¯σ ] a.s. X m n n m This expression in terms of the preceding definitions is X(σm ∧ τn ) = E[X(τn )|Fσm ] a.s.

5.6 Optional Stopping of Martingales

359

Letting m → ∞ in this expression results in σm → σ and X(σ ∧ τn ) = E[X(τn )|Fσ ] a.s. Then letting n → ∞ in this expression yields (5.19). The justifications of the last two limit statements, which are in [61], will not be covered here. The assertion E[X(0)] = E[X(τ )] follows by taking expectations in (5.19) with σ = 0. Finally, when σ as well as τ is bounded, then (5.19) and this expression with the roles of σ and τ reversed yield E[X(τ )|Fσ ] = X(σ ∧ τ ) = E[X(σ)|Fτ ]

a.s.

Then expectations of these terms give E[X(σ)] = E[X(τ )]. Theorem 22 can also be extended to stopping times that are a.s. finite, but not necessarily bounded. To see this, suppose that τ is an a.s. finite stopping time of Ft . The key idea is that, for fixed s and t, the τ ∧ s and τ ∧ t are a.s. bounded stopping times of Ft . Then by Theorem 22, X(τ ∧ s) = E[X(τ ∧ t)|Fτ ∧s ],

s < t.

This property justifies the following fact, which is used in the proof below. Remark 23. Stopped Martingales. The stopped process X(τ ∧t) is a martingale with respect to Ft . Corollary 24. Associated with the martingale X, suppose that τ is an a.s. finite stopping time of Ft , and that either one of the following conditions is satisfied.

 (i) E supt≤τ |X(t)| < ∞. (ii) E[ |X(τ )| ] < ∞, and limu→∞ E[ |X(u)|1(τ > u)] = 0. Then E[X(τ )] = E[X(0)]. Proof. Since X(τ ∧ t) is a martingale with respect to Ft , Theorem 22 implies E[X(τ ∧ u)] = E[X(0)], for u > 0. Now, we can write |E[X(τ )] − E[X(0)]| = |E[X(τ )] − E[X(τ ∧ u)]| ≤ E[ |X(τ ) − X(u)|1(τ > u)] |. If (i) holds, then |X(τ ) − X(u)| ≤ 2Z, where Z = supt≤τ |X(t)|. Since τ is finite a.s., 1(τ > u) → 0 a.s. as u → ∞, and so by the dominated convergence theorem, |E[X(τ )] − E[X(0)]| ≤ 2E[Z1(τ > u)] → 0. On the other hand, if (ii) holds, then

   |E[X(τ )] − E[X(0)]| ≤ E |X(τ )| + |X(u)| 1(τ > u) → 0. Thus, E[X(τ )] = E[X(0)] if either (i) or (ii) is satisfied.

360

5 Brownian Motion

The next proposition and example illustrate computations involving optional stopping. Proposition 25. (Wald Identities) Let X be a process with stationary independent increments as in Example 18, with E[ |X(1)| ] < ∞ and ψ(α) = E[eαX(1) ]. Suppose τ is an a.s. finite stopping time of X. Then E[X(τ )] = E[X(1)]E[τ ].

 If in addition τ is bounded or E supt≤τ |X(t)| < ∞, then E[eαX(τ ) ψ(α)−τ ] = 1,

for any α with ψ(α) ≥ 1.

(5.20)

Proof. Example 18 establishes that X(t) − E[X(1)]t is a martingale with respect to X. Now, τ ∧ t is a bounded stopping time of X, and so by the optimal stopping theorem, E[X(τ ∧ t) − E[X(1)](τ ∧ t)] = 0. Letting t → ∞ in this expression, the dominated and monotone convergence theorems yield E[X(τ )] = E[X(1)]E[τ ]. Similarly, Z(t) = eαX(t) ψ(α)−t is a martingale with respect to X, and under the assumptions the optional stopping theorem or Corollary 24 imply that E[Z(τ )] = E[Z(0)] = 1, which gives (5.20). Example 26. Brownian Optional Stopping. For a Brownian motion B, we know by Example 19 that the processes B(t) and B(t)2 − t are martingales with respect to B with zero means. Then as in the preceding proposition, we have the following result. If τ is an a.s. finite stopping time of B, then E[B(τ )] = 0. In addition, E[τ ] = E[B(τ )2 ] if τ is bounded a.s. 2 Example 19 also noted that X(t) = ecB(t)−c t/2 is a martingale with respect to B with mean 1. If τ is an a.s. bounded stopping time of B, then E[X(τ )] = 1 by the optional stopping theorem. Consequently, the conditional moment generating function for an increment of B following τ is E[ec[B(τ +u)−B(τ )]|Fτ ] = ec

2

u/2

= E[ecB(u) ].

This was the key step in proving the strong Markov property of B for bounded stopping times. The rest of this section is devoted to proving the discrete-time optional stopping theorem used in the proof of Theorem 22. We begin with a preliminary result. Proposition 27. Let X and Y be random variables on a probability space, and let F and G be two σ-fields on the space. Suppose there is an event A ∈ F ∩ G such that X = Y a.s. on A and F = G on A (A ∩ F = A ∩ G). Then E[X|F] = E[Y |G] a.s. on A.

5.7 Hitting Times for Brownian Motion with Drift

361

Proof. Let Z = E[X|F] − E[Y |G] and C = A ∩ {Z > 0}. Under the hypotheses, C ∈ F ∩ G and 

E[Z1C ] = E E[X|F]1C − E[Y |G]1C = E[X1C − Y 1C ] = 0. Here 1C is the random variable 1(ω ∈ C). Because a nonnegative random variable V is 0 a.s. if and only if E[V ] = 0, it follows that Z1C = 0 a.s., which implies Z ≤ 0 a.s. on A. A similar argument with C = A ∩ {Z < 0}, shows Z ≥ 0 a.s. on A. This proves the assertion. Theorem 28. Suppose that {Xn : n ∈ Z+ } is a martingale with respect to Fn , and that σ and τ are stopping times of Fn such that τ is bounded a.s. Then Xσ∧τ = E[Xτ |Fσ ] a.s.. Hence E[Xτ ] = E[X0 ]. If σ is also bounded, then E[Xσ ] = E[Xτ ]. Proof. For m ≤ n, one can show that Fτ = Fm on {τ = m}. Then by Proposition 27 and the martingale property, E[Xn |Fτ ] = E[Xn |Fm ] = Xm = Xτ ,

a.s. on {τ = m}.

Since this is true for each m ≤ n, we have E[Xn |Fτ ] = Xτ ,

a.s. if τ ≤ n

a.s.

(5.21)

Now, consider the case σ ≤ τ ≤ n a.s. Then Fσ ⊆ Fτ . Using this and (5.21) for τ and for σ, we get  

 E[Xτ |Fσ ] = E E[Xn |Fτ ]Fσ = E[Xn |Fσ ] = Xσ a.s. In addition, E[Xτ |Fσ ] = Xτ a.s. if τ ≤ σ ∧ n. For the general case, similar reasoning using the preceding two results and Proposition 27 yield E[Xτ |Fσ ] = E[Xτ |Fσ∧τ ] = Xσ∧τ

a.s. on {σ ≤ τ }

E[Xτ |Fσ ] = E[Xσ∧τ |Fσ ] = Xσ∧τ

a.s. on {σ > τ }.

This proves Xσ∧τ = E[Xτ |Fσ ] a.s. The other assertions follow as in the proof of Theorem 22.

5.7 Hitting Times for Brownian Motion with Drift We will now address the following questions for a Brownian motion with drift. What is the probability that the process hits b before it hits a, for a < b?

362

5 Brownian Motion

What is the distribution and mean of the time for the process to hit b? We answer these questions by applications of the material in the preceding section on martingales and optional stopping. Consider a Brownian motion with drift X(t) = x + μt + σB(t), where B is a standard Brownian motion. For a < x < b, let τa and τb denote the times at which X hits the respective states a and b. In addition, let τ = τb ∧ τa , which is the time at which X escapes from the open strip (a, b). Our focus will be on properties of these hitting times. Remark 29. Finiteness of Hitting Times. If μ ≥ 0, then τb is finite a.s. since using Remark 9, τb = inf{t ≥ 0 : X(t) = b} ≤ inf{t ≥ 0 : B(t) = (b − x)/σ} < ∞ a.s. Similarly, τa is finite a.s. if μ ≤ 0. Also, τ is finite since either τa or τb is necessarily finite. We begin with a result for a Brownian motion with no drift. Theorem 30. The probability that the process X(t) = x + σB(t) hits b before a is P {τb < τa } = (x − a)/(b − a). (5.22) Also, E[τ ] = (x − a)(b − x)/σ 2 . Proof. By Example 19, X is a martingale with respect to B with mean x. Also, E[supt≤τ |X(t)|] is finite since X(t) ∈ (a, b) for t ≤ τ . Then by the optional stopping theorem (Theorem 22) for τ , E[X(τ )] = E[X(0)] = x.

(5.23)

Now, since τ = τa ∧ τb , we can write X(τ ) = a1(τa ≤ τb ) + b1(τb < τa ).

(5.24)

Then E[X(τ )] = a[1 − P {τb < τa }] + bP {τb < τa }. Substituting this in (5.23) and solving for P {τb < τa }, we obtain (5.22). Next, we know by Example 19 that Z(t) = (X(t)−x)2 −σ 2 t is a martingale with respect to B. Then the optional stopping theorem for the bounded stopping time τ ∧ t yields E[Z(τ ∧ t)] = E[Z(0)] = 0. That is, σ 2 E[τ ∧ t] = E[(X(τ ∧ t) − x)2 ]. Now since τ ∧ t ↑ τ and X(t) is bounded for t ≤ τ , it follows by the monotone and bounded convergence theorems that σ 2 E[τ ] = E[(X(τ ) − x)2 ].

5.7 Hitting Times for Brownian Motion with Drift

363

Then using (5.24) in the last expectation followed by (5.22), we have σ 2 E[τ ] = (a − x)2 P {τa ≤ τb } + (b − x)2 P {τb < τa } = (x − a)(b − x). The preceding result for a Brownian motion with no drift has the following analogue for a Brownian motion X(t) = x + μt + σB(t) with drift μ = 0. Theorem 31. The probability that the process X hits b before a is P {τb < τa } =

eαx − eαa , eαb − eαa

where α = −2μ/σ 2 . In addition,

 E[τ ] = μ−1 (a − x) + (b − a)P {τb < τa } .

(5.25)

(5.26)

Proof. As in Example 19, Z(t) = exp{cX(t) − cx − (cμ + c2 σ 2 /2)t} is a martingale with respect to B. Letting c = α, this martingale reduces to Z(t) = eαX(t)−αx . Now, E[supt≤τ |Z(t)|] is finite, since X(t) ∈ [a, b] for t ≤ τ . Then by Corollary 24 on optional stopping, 1 = E[Z(0)] = E[Z(τ )] = e−αx E[eαX(τ ) ]. Now, using X(τ ) = a1(τa ≤ τb ) + b1(τb < τa ) in this expression yields eαx = eαa (1 − P {τb < τa }) + eαb P {τb < τa }. This proves (5.25). To determine E[τ ], we apply the optional stopping theorem to the martingale B(t) = σ −1 [X(τ ) − x − μτ ] and the bounded stopping time τ ∧ t to get 0 = E[B(τ ∧ t)]. That is, μE[τ ∧ t] + x = E[X(τ ∧ t)]. Letting t → ∞ in this expression, we have τ ∧ t ↑ τ , and so the monotone and bounded convergence theorems yield μE[τ ] + x = E[X(τ )] = bP {τb < τa } + a(1 − P {τb < τa }). This proves (5.26). The last result of this section characterizes the distribution of the hitting time τb for a Brownian motion X(t) = μt + σB(t) with drift μ. Theorem 32. Let τb denote the time at which the Brownian motion X hits b > 0. If μ ≥ 0, then E[τb ] = −b/μ (which is ∞ if μ = 0) and the Laplace transform and density of τb are

364

5 Brownian Motion

$ E[e−λτb ] = exp{−bσ −2 [ μ2 + 2σ 2 λ − μ]}, b exp{−(b − μx)2 /(2σ 2 x}. fτb (x) = √ σ 2πx3

(5.27) (5.28)

2

If μ < 0, then τb may be infinite and P {τb < ∞} = e2bμ/σ . Proof. For the case μ < 0, it follows by (5.25) (with x = 0 and α > 0) that 2

P {τb < ∞} = lim P {τb < τa } = e2bμ/σ . a→−∞

Next, consider the case μ ≥ 0. For positive constants α and λ, consider the process Z(t) = eαX(t)−λt . This is a martingale (see Example 19) and it 2 reduces to Z(t) = ecB(t)−c t/2 , where $ c = ασ, α = σ −2 [ μ2 + 2σ 2 λ − μ]. This choice of α ensures that α2 σ 2 /2 + αμ − λ = 0. Now, applying the optional stopping theorem to the martingale Z(t) and the bounded stopping time τb ∧ t, we obtain 1 = E[Z(0)] = E[Z(τb ∧ t)] = E[eαX(τb ∧t)−λ(τb ∧t) ]. Since X is continuous a.s., we have lim [αX(τb ∧ t) − λ(τb ∧ t)] = αb − λτb

t→∞

a.s.

Then by the preceding displays and the bounded convergence theorem, 1 = E[ lim Z(τb ∧ t)] = eαb E[e−λτb ]. t→∞

This proves (5.27). Inverting this Laplace transform yields the density formula (5.28). Finally, the derivative of this transform at λ = 0 yields E[τb ] = −b/μ.

5.8 Limiting Averages and Law of the Iterated Logarithm This section contains strong laws of large numbers for Brownian motion and its maximum process, and a law of the iterated logarithm for Brownian motion. As usual B will denote a standard Brownian motion. The strong law of large numbers for it is as follows. Theorem 33. A Brownian motion B has the limiting average

5.8 Limiting Averages and Law of the Iterated Logarithm

lim t−1 B(t) = 0

t→∞

365

a.s.

Proof. Since B has stationary independent increments, it has regenerative increments with respect to the deterministic times Tn = n. Then the assertion will follow by the SLNN in Theorem 54 in Chapter 2 for processes with regenerative increments upon showing that n−1 B(n) → 0 a.s., and

 E max |B(t)| < ∞. (5.29) n−1≤t≤n

Now, the SLLN for i.i.d. random variables ensures that n−1 B(n) = n−1

n 

[B(m) − B(m − 1)] → E[B(1)] = 0,

a.s.

m=1

Also, (5.29) follows since E[M (1)] < ∞ and max

n−1≤t≤n

d

|B(t)| = max |B(t)| ≤ M (1) − M (1), 0≤t≤1

d

where M (t) = mins≤t B(s) = −M (t) by Remark 12. If a real-valued process X, such as a Brownian motion or a functional of a Markov process, has a limiting average t−1 X(t) → c a.s., you might wonder if its maximum M (t) = sups≤t X(s) also satisfies t−1 M (t) → c a.s. Wonder no longer. The answer is given by the next property, n which is analogous to the elementary fact that if n−1 cn → c, then n−1 k=1 ck → c. Proposition 34. Let x(t) and a(t) be real-valued functions on R+ such that 0 ≤ a(t) → ∞,

a(t)−1 x(t) → c,

as t → ∞.

Then the maximum m(t) = sups≤t x(s) satisfies limt→∞ a(t)−1 m(t) = c. Proof. For any ε > 0, let t be such that a(t)−1 x(t) < c + ε, for t ≥ t . Then for t ≥ t , % & a(t)−1 x(t) ≤ a(t)−1 m(t) = max a(t)−1 m(t ), a(t)−1 sup x(s) t ≤s≤t

. ≤ max a(t)−1 m(t ), c + ε . Letting t → ∞ in this display, it follows that

c ≤ lim inf a(t)−1 m(t) ≤ lim sup a(t)−1 m(t) ≤ c + ε. t→∞

t→∞

Since this is true for any ε, we have a(t)−1 m(t) → c. We will now apply this result to the maximum process M (t) = maxs≤t X(s) for a Brownian motion with drift X(t) = x + μt + σB(t).

366

5 Brownian Motion

Proposition 35. The Brownian motion with drift X and its maximum process have the limiting averages t−1 X(t) → μ,

t−1 M (t) → μ

a.s. as t → ∞.

Proof. This follows since the SLLN t−1 B(t) → 0 implies that t−1 X(t) = t−1 x + μ + σt−1 B(t) → μ a.s., and then t−1 M (t) → μ a.s. follows by Proposition 34. The preceding result implies that M (t) → ∞ or −∞ a.s. according as the drift μ is positive or negative. This tells us something about the maximum M (∞) = sup X(t) t∈R+

on the entire time axis. First, we have the obvious result M (∞) = lim M (t) = ∞ a.s. when μ > 0. t→∞

Second, M (∞) = ∞ a.s. when μ = 0 by the law of the iterated logarithm in (5.30) below. For the remaining case of μ < 0, we have the following result. Theorem 36. If μ < 0 and X(0) = 0, then M (∞) has an exponential distribution with rate −2μ/σ 2 . Proof. The assertion follows, since letting t → ∞ in {M (t) > b} = {tb < t} and using Theorem 32, 2

P {M (∞) > b} = P {τb < ∞} = e2μb/σ . We will now consider fluctuations of Brownian motions that are described by a law of the iterated logarithm. Knowing that the limiting average of Brownian motion B is 0 as t → ∞, a follow-on issue is to characterize its fluctuations about 0. These fluctuations, of course, can be described for a √ “fixed” t by the normal distribution of B(t); e.g., P {|B(t)| ≤ 2 t} ≈ .95. However, to get a handle on rare fluctuations as t → ∞, it is of interest to find constants h(t) such that lim sup t→∞

B(t) =1 h(t)

a.s.

In other words, h(t) is the maximum height of the fluctuations of B(t) above 0, and B(t) gets near h(t) infinitely often (i.o.) as t → ∞ in that P {B(t) ∈ [h(t) − ε, h(t)] i.o.} = 1,

ε > 0.

5.8 Limiting Averages and Law of the Iterated Logarithm

367

Since the reflection −B is a Brownian motion, the preceding would also yield lim inf t→∞

B(t) = −1 a.s. h(t)

These fluctuations are related to those as t ↓ 0 as follows. Remark 37. lim sup t→∞

B(t) = 1 a.s. h(t)

⇐⇒

lim sup t↓0

B(t) =1 th(1/t)

a.s.

This is because the time-inversion process X(t) = tB(1/t) is a Brownian motion by Exercise 2. Indeed, the equivalence is true since, using s = 1/t, lim sup t→∞

B(t) X(s) = lim sup . h(t) sh(1/s) s↓0

Remark 37 says that h(t) is the height function for fluctuations of B as t → ∞ if and only if th(1/t) is the height function for fluctuations as t ↓ 0. The height functions for both of these cases are as follows. The proof, due to Khintchine 1924, is in [37, 61, 64]. Theorem 38. (Law of the Iterated Logarithm) B(t) lim sup $ = 1, 2t log log(1/t) t↓0 B(t) lim inf $ = −1, t↓0 2t log log(1/t)

B(t) =1 lim sup √ 2t log log t t→∞ B(t) = −1 lim inf √ t→∞ 2t log log t

a.s. a.s.

Note that the lim supt↓0 result implies that B(t) > 0 i.o. near 0, and so inf{t > 0 : B(t) > 0} = 0

a.s.

Similarly, the lim inf t↓0 result implies that B(t) < 0 i.o. near 0 a.s. Consequently, B(t) = 0 i.o. near 0 a.s. because B has continuous paths a.s. The other results for t → ∞ imply that, for any fixed a > 0, we have B(t) > a and B(t) < −a i.o. a.s. as t → ∞, and so B passes through [−a, a] i.o. a.s. Furthermore, the extremes of B are sup B(t) = ∞ t∈R+

a.s.,

inf B(t) = −∞ a.s.

t∈R+

(5.30)

368

5 Brownian Motion

5.9 Donsker’s Functional Central Limit Theorem By the classical central limit theorem (Theorem 63 in Chapter 2), we know that a random walk under an appropriate normalization converges in distribution to a normal random variable. This section extends this result to stochastic processes. In particular, viewing a random walk as a process in continuous time, if the time and space parameters are rescaling appropriately, then the random walk process converges in distribution to a Brownian motion. This result, called Donsker’s functional central limit theorem, also establishes that many functionals of random walks can be approximated by corresponding functionals of Brownian n motion. Throughout this section Sn = i=1 ξk will denote a random walk in which the step sizes ξn are i.i.d. with mean 0 and variance 1. For each n, consider the stochastic process Xn (t) = n−1/2 S nt ,

t ∈ [0, 1].

That is, Xn (t) = n−1/2 Sk

if k/n ≤ t < (k + 1)/n for some k < n.

This process is a continuous-time representation of the random walk Sk in which the location Sk is rescaled (or shrunk) to the value n−1/2 Sk , and the time scale is rescaled such that the walk takes [nt] steps in time t. Then as n becomes large the steps become very small and frequent and, as we will show, Xn converges in distribution to a standard Brownian motion B as n → ∞. We begin with the preliminary observation that the finite-dimensional distributions of Xn converge in distribution to those of B. That is, for any fixed t1 < · · · < tk , d

(Xn (t1 ), . . . , Xn (tk )) → (B(t1 ), . . . , B(tk )),

as n → ∞.

(5.31)

d

In particular, for each fixed t, we have Xn (t) → B(t), as n → ∞. d

The latter follows since n−1/2 Sn → B(1) by the classical central limit theorem, and so Xn (t) = (nt/n)1/2 nt−1/2 S nt → t1/2 B(1) = B(t). d

d

Similarly, (5.31) follows by a multivariate central limit theorem. Expression (5.31) only provides a partial description of the convergence in distribution of Xn to B; we will now give a complete description of the convergence that includes sample path information. Throughout this section, D = D[0, 1] will denote the set of all functions x : [0, 1] → R that are right-continuous with left-hand limits. Assume that the σ-field associated with D is the smallest σ-field under which the projection

5.9 Donsker’s Functional Central Limit Theorem

369

map x → x(t) is measurable, for each t. Almost every sample path of Xn is a function in D, and so the process Xn is a D-valued random variable (or a random element in D). We will consider D as a metric space in which the distance between two functions x and y is x − y, based on the uniform or supremum norm x = sup |x(t)|. t≤1

Other metrics for D are discussed in [11, 115]. Convergence in distribution of random elements in D, as in other metric spaces, is as follows. Random elements Xn in a metric S convergence in distribution to X in S as n → ∞, d denoted by Xn → X in S, if lim E[f (Xn )] = E[f (X)],

n→∞

d

for any bounded continuous function f : S → R. The convergence Xn → X is equivalent to the weak convergence of their distributions w

P {Xn ∈ ·} → P {X ∈ ·}.

(5.32)

Several criteria for this convergence are in the Appendix. d An important consequence of Xn → X in S is that it readily leads to the convergence in distribution of a variety of functionals of the Xn as follows. d

Theorem 39. (Continuous Mapping) Suppose that Xn → X in S as n → ∞, and f : S → S  is a measurable mapping, where S  is another metric space. If C ⊆ S is in the σ-field of S such that f is continuous on C and X ∈ C d a.s., then f (Xn ) → f (X) in S  as n → ∞. d

Proof. Recall that Xn → X is equivalent to (5.32), which we will denote by w d w μn → μ. Then f (Xn ) → f (X) is equivalent to μn f −1 → μf −1 since P {f (Xn ) ∈ A} = P {Xn ∈ f −1 (A)} = μn f −1 (A). w

Also note that by Theorem 10 in the Appendix, μn → μ is equivalent to lim inf μn (G) ≥ μ(G), n→∞

for any open G ⊆ S.

Now using this characterization, for any open set G ⊆ S  , lim inf μn f −1 (G) ≥ lim inf μn (f −1 (G)◦ ) ≥ μ(f −1 (G)◦ ). n→∞

n→∞

Here A◦ is the interior of the set A. Clearly f −1 (G)◦ ⊃ C ∩ f −1 (G), and μ(C) = 1 by the assumption X ∈ C a.s. Then μ(f −1 (G)◦ ) = μf −1 (G). Using

370

5 Brownian Motion

this in the preceding display yields lim inf n→∞ μn f −1 (G) ≥ μf −1 (G), which d

w

proves μn f −1 → μf −1 , and hence f (Xn ) → f (X). We are now ready to present the functional central limit theorem proved by Donsker in 1951 for the continuous-time random walk process Xn (t) = n−1/2 S nt ,

t ∈ [0, 1].

Theorem 40. (Donsker’s FCLT) For the random walk process Xn defined d

above, Xn → B in D as n → ∞, where B is a standard Brownian motion. The proof of this theorem will follow after a few observations and preliminary results. Donsker’s theorem is called a “functional central limit theorem” because, under the continuous-mapping theorem, many functionals of the random walk also converge in distribution to the corresponding functionals of the limiting Brownian motion. Two classic examples are as follows; we cover other examples later. d

Example 41. If Xn → B in D, then, for t1 < · · · < tk ≤ 1, (n−1/2 S nt1  , . . . , n−1/2 S ntk  ) → (B(t1 ), . . . , B(tk )). d

(5.33) d

This convergence is equivalent to (5.31). Now (5.33) says f (Xn ) → f (B), where f : D → Rk is defined, for fixed t1 < · · · < tk , by f (x) = (x(t1 ), . . . , x(tk )). Clearly f is continuous on the set C of continuous functions in D and B ∈ C a.s. Then (5.33) follows from the continuous-mapping theorem. d

Example 42. The convergence Xn → B implies n−1/2 max Sm → max B(s). d

m≤n

s≤1

The distribution of the limit is given in Theorem 11. The convergence follows by the continuous-mapping theorem since the function f : D → R+ defined by f (x) = maxs≤1 x(s) is continuous in that xn − x → 0 implies maxs≤1 xn (s) → maxs≤1 x(s). Donsker’s FCLT is also called an invariance principle because in the cond vergence Xn → B, the Brownian motion limit B is the same for “any” distribution of the step size of the random walk, provided it has a finite mean and variance. When the mean and variance are not 0 and 1, respectively, the result applies with the following change in notation.

5.9 Donsker’s Functional Central Limit Theorem

371

Remark 43. If ξn are i.i.d. random variables with finite mean μ and variance σ 2 , then (ξk −μ)/σ are i.i.d. with mean 0 and variance 1, and hence Donsker’s theorem holds for

nt

Xn (t) = n−1/2



(ξk − μ)/σ,

t ∈ [0, 1].

k=1

 Consequently, the random walk Sn = nk=1 ξk , for large n, is approximately equal in distribution to a Brownian motion with drift. In particular, using d n1/2 B(t) = B(nt), d

S nt ≈ μnt + σB(nt),

d

Sn ≈ μn + σB(n).

Does the convergence in distribution in Donsker’s theorem hold for processes defined on the entire time axis R+ ? To answer this, consider the space D[0, T ] of all functions x : [0, T ] → R that are right-continuous with lefthand limits, for fixed T > 0. Similarly to D[0, 1], the D[0, T ] is a metric space with the supremum norm. Now let D(R+ ) denote the space of all functions x : R+ → R that are right-continuous with left-hand limits. Consider D(R+ ) as a metric space in which convergence xn → x in D(R+ ) holds if xn → x in D[0, T ] holds for each T that is a continuity point of x. d

Remark 44. Convergence in D(R+ ). Donsker’s convergence Xn → B holds in D[0, T ], for each T , and in D(R+ ) as well. The proof for D[0, T ] is exactly the same as that for D[0, 1]. The convergence also holds in D(R+ ), since B is continuous a.s. Donsker’s approach for proving Theorem 40 is to prove the convergence (5.31) of the finite-dimensional distributions and then establish a certain tightness condition. This proof is described in Billingsley 1967; his book and one by Whitt 2002 cover many fundamentals of functional limit theorems and weak convergence of probability measures on metric spaces. Another approach for proving Theorem 40, which we will now present, is by applying Skorohod’s embedding theorem. The gist of this approach is that one can construct a Brownian motion B and stopping times τn for it such d that {Sn } = {B(τn )}. Then further analysis of Xn and B defined on the P

d

same probability space establishes Xn − B → 0, which yields Xn → B. The key embedding theorem for this analysis is as follows. It says that any random variable ξ with mean 0 and variance 1 can be represented as B(τ ) for an appropriately defined stopping time τ . Furthermore, any i.i.d. sequence ξn of such variables can be represented as an embedded sequence B(τn ) − B(τn−1 ) in a Brownian motion B for appropriate stopping times τn . The proof is in [37, 61]. Theorem 45. (Skorohod Embedding) Associated with the random walk Sn , there exists a standard Brownian motion B with respect to a filtration and

372

5 Brownian Motion

stopping times 0 = τ0 ≤ τ1 ≤ . . . such that τn − τn−1 are i.i.d. with mean 0 d and variance 1, and {Sn } = {B(τn )}. Another preliminary leading to Donsker’s theorem is the following Skorohod approximation result that the uniform difference between the random walk and a Brownian motion on [0, t] is o(t1/2 ) a.s. as t → ∞. This material and the proof of Donsker’s theorem below is from Kallenberg 2004. Theorem 46. (Skorohod Approximation of Random Walks) There exists a standard Brownian motion B on the same probability space as the random walk Sn such that t−1/2 sup |S s − B(s)| → 0, P

as t → ∞.

(5.34)

s≤t

Proof. Let B and τn be as in Theorem 45, and define them on the same probability space as Sn (which is possible) so Sn = B(τn ) a.s. Define D(t) = t−1/2 sup |B(τ s ) − B(s)|. s≤t

Then (5.34) is equivalent to P {D(t) > ε} → 0 for ε > 0. To prove this convergence, let δt = sups≤t |τ s − s|, t ≥ 0. For a fixed γ > 0, consider the inequality P {D(t) > ε} ≤ P {D(t) > ε, t−1 δt ≤ γ} + P {t−1 δt > γ}.

(5.35)

Note that n−1 τn → 1 a.s. by the strong law of large numbers for i.i.d. random variables, and so t−1 |τ t − t| → 0 a.s. Then the limiting average of the supremum of these differences is t−1 δt → 0 a.s. by Proposition 34. Next, consider the modulus of continuity of f : R+ → R, which is w(f, t, γ) =

sup r,s≤t, |r−s|≤γ

|f (r) − f (s)|,

Clearly D(t) ≤ w(B, t + tγ, tγ),

t ≥ 0.

when t−1 δt ≤ γ. d

Using this observation in (5.35) and {t−1/2 B(r) : r ≥ 0} = {B(rt) : r ≥ 0} from the scaling property in Exercise 2, we have P {D(t) > ε} ≤ P {t−1/2 w(B, t + tγ, tγ) > ε} + P {t−1 δt > γ} = P {w(B, 1 + γ, γ) > ε} + P {t−1 δt > γ}. Letting t → ∞ (t−1 δt → 0 a.s.), and then letting γ → 0 (B has continuous paths a.s.), the last two probabilities tend to 0. Thus P {D(t) > ε} → 0, which proves (5.34). We will now obtain Donsker’s theorem by applying Theorem 46.

5.10 Regenerative and Markov FCLTs

373

Proof of Donsker’s Theorem. Let B and Sn = B(τn ) a.s. be as in the proof of Theorem 46, and define Bn (t) = n−1/2 B(nt). Clearly Xn − Bn  = n−1/2 sup |S nt − B(nt)| = n−1/2 sup |S s − B(s)|. t≤1

s≤n

P

Then Xn − Bn  → 0 by Theorem 46. Next, note that, by Exercise 1, the scaled process Bn is a Brownian motion. ˜ n and a Brownian motion B ˜ on the same Now, as in [61], one can construct X d ˜ = (Xn , Bn ). Then we have ˜ n , B) probability space such that (X d

d

P

˜ n − B ˜ = Xn − Bn  → 0. Xn − B = X d

This proves Xn → B.

5.10 Regenerative and Markov FCLTs This section presents an extension of Donsker’s FCLT for processes with regenerative increments. This in turn yields FCLTs for renewal processes and ergodic Markov chains in discrete and continuous time. For this discussion, suppose that {Z(t) : t ≥ 0} is a real-valued process with Z(0) = 0 that is defined on the same probability space as a renewal process N (t) whose renewal times are denoted by 0 = T0 < T1 < . . . The increments of the two-dimensional process (N (t), Z(t)) in the interval [Tn−1 , Tn ) are denoted by ζn = (Tn − Tn−1 , {Z(t) − Z(Tn−1 ) : t ∈ [Tn−1 , Tn )}). Recall from Section 2.10 that Z(t) has regenerative increments over Tn if ζn are i.i.d. Theorem 65 in Chapter 2 is a central limit theorem for processes with regenerative increments. An analogous FCLT is as follows. Assuming they are finite, let μ = E[T1 ],

a = E[Z(T1 )]/μ,

σ 2 = Var[Z(T1 ) − aT1 ],

and assume σ > 0. In addition, let Mn =

sup

|Z(t) − Z(Tn )|,

n ≥ 0,

Tn 0, consider the process

374

5 Brownian Motion

Xr (t) =

Z(rt) − art $ , σ r/μ

t ∈ [0, 1].

This is the regenerative-increment process Z with space-time scale changes analogous to those for random walks. A real-valued parameter r instead of an integer is appropriate since Z is a continuous-time process. Theorem 47. (Regenerative Increments) For the normalized regeneratived

increment process Xr defined above, Xr → B as r → ∞, where B is a standard Brownian motion. The proof uses the next two results. Let D1 denote the subspace of functions x in D that are nondecreasing with x(0) = 0 and x(t) ↑ 1 as t → 1. The composition mapping from the product space D × D1 to D, denoted by (x, y) → x ◦ y, is defined by x ◦ y(t) = x(y(t)), t ∈ [0, 1]. Let C and C1 denote the subspaces of continuous functions in D and D1 , respectively. Proposition 48. The composition mapping from D × D1 to D is continuous on the subspace C × C1 . Proof. Suppose (xn , yn ) → (x, y) in D × D1 such that (x, y) ∈ C × C1 . Using the sup norm and the triangle inequality, xn ◦ yn − x ◦ y ≤ xn ◦ yn − x ◦ yn  + x ◦ yn − x ◦ y. Now, the last term tends to 0 since x ∈ C is uniformly continuous. Also, xn ◦ yn − x ◦ yn  = xn − x → 0. Thus xn ◦ yn → x ◦ y in D, which proves the assertion. The continuity of composition mappings under weaker assumptions is discussed in [11, 115]. The importance of the composition mapping is illustrated by the following result. In the setting of Theorem 47, the regenerative- increment property of Z implies that ξn = Z(Tn ) − Z(Tn−1 ) − a(Tn − Tn−1 ) are i.i.d. with mean 0 and variance σ 2 . Lemma 49. Under the preceding assumptions, define the process Xr (t) = d

Then Xr → B as r → ∞.

N (rt)  1 $ ξk , σ r/μ k=1

t ∈ [0, 1].

5.10 Regenerative and Markov FCLTs

 rt

˜ r (t) = √1 Proof. Letting X σ

375

r/μ

k=1 ξk ,

it follows by Donsker’s theorem that

d ˜r → μ1/2 B as r → ∞. With no loss in generality, assume μ−1 < 1. Consider X the process N (rt)/r if N (r)/r ≤ μ−1 Yr (t) = t/μ if N (r)/r > μ−1 .

Note that ˜ r ◦ Yr (t) = X

1 $ σ r/μ

rYr (t)



ξk .

k=1

This equals Xr (t) when N (r)/r ≤ μ−1 , and so for any ε > 0, ˜ r ◦ Yr  > ε} ≤ P {N (r)/r > μ−1 } → 0. P {Xr − X The convergence follows since N (r)/r → μ−1 a.s. by the SLLN for renewal d ˜ r ◦ Yr → 0. Then processes (Corollary 11 in Chapter 2). This proves X  − X r

d d ˜ r ◦ Yr → B. to prove Xr → B, it suffices by Exercise 53 to show that X Letting I(t) = t, t ∈ [0, 1], note that

Yr − μ−1 I ≤ sup |N (rt)/r − μ−1 t| t≤1

= r−1 sup |N (s) − μ−1 s| → 0

a.s.

s≤r

The convergence follows by Proposition 34 since the SLLN for N implies d ˜ r , Yr ) → r−1 |N (r) − μ−1 r| → 0 a.s. Now, we have (X (μ1/2 B, μ−1 I), where the limit functions are continuous. Then Proposition 48 and Exercise 1 yield d d ˜ r ◦ Yr → X μ1/2 B ◦ μ−1 I = B. d ˜ r ◦ Yr → Thus X B, which completes the proof.

Remark 50. The assertion in Lemma 49 implies that Xr (1)

N (r)  1 d $ = ξk → B(1), σ r/μ k=1

which is Anscombe’s result in Theorem 64 in Chapter 2. We now establish the convergence of Xr (t) = (Z(rt) − art)/(σ

$

r/μ).

Proof of Theorem 47. We can write Xr (t) = Xr (t) + where

√ μ Vr (t), σ

(5.36)

376

5 Brownian Motion

Xr (t) =

Z(TN (rt) ) − aTN (rt) $ , σ r/μ

 Vr (t) = r−1/2 Z(rt) − Z(TN (rt) ) − a(rt − TN (rt) ) . d

Recognizing that Xr is the process in Lemma 49, we have Xr → B. Then the d

d

proof of Xr → B will be complete upon showing that Vr → 0. Letting sup |Z(t) − Z(Tn )| + a(Tn+1 − Tn ), ξ¯n = Tn 0. Then Theorem 47 for the CTMC functional Z is as follows. An analogous result for discrete-time Markov chains is in Exercise 48. Corollary 52. (CTMC) Under the preceding assumptions, for r > 0, define the process ! rt f (Y (s))ds − art $ Xr (t) = 0 , t ∈ [0, 1]. σ r/μ d

Then Xr → B as r → ∞.

5.11 Peculiarities of Brownian Sample Paths While sample paths of a Brownian motion are continuous a.s., they are extremely erratic. This section describes their erratic behavior. Continuous functions are typically monotone on certain intervals, but this is not the case for Brownian motion paths. Proposition 53. Almost every sample path of a Brownian motion B is monotone on no interval. Proof. For any a < b in R+ , consider the event A = {B is nondecreasing on [a, b]}. Clearly A = ∩∞ n=1 An , where An = ∩ni=1 {B(ti ) − B(ti−1 ) ≥ 0} and ti = a + i(b − a)/n. The A is measurable since each An is. Because P {B(ti ) − B(ti−1 ) ≥ 0} = 1/2 and the increments of B are independent, we have P (An ) = 2−n , and so P (A) ≤ limn→∞ P (An ) = 0. This conclusion is also true for the event A = {B is nonincreasing on [a, b]}. Thus B is monotone on no interval a.s.

378

5 Brownian Motion

For the next result, we say that for a Brownian motion B on a closed interval I, its local maximum is supt∈I B(t), and its local minimum is inf t∈I B(t). There are processes that have local maxima on two disjoint intervals that are equal with a positive probability, but this is not the case for Brownian motion. Proposition 54. The local maxima and minima of a Brownian motion B are a.s. distinct. Proof. It suffices to show that, for disjoint closed intervals I and J in R+ , MI = MJ

a.s.,

where each of the quantities MI and MJ is either a local minimum or a local maximum. First, suppose MI and MJ are both local maxima. Let u denote the right endpoint of I and v > u denote the left endpoint of J. MJ − MI = sup[B(t) − B(v)] − sup[B(t) − B(u)] + B(v) − B(u). t∈J

t∈I

The three terms on the right are independent, and the last one is nonzero a.s. (since the increments are normally distributed). Therefore, MI = MJ a.s. This result is also true by similar arguments when each of the quantities MI and MJ are both local minima, or when one is a local minimum and the other is a local maximum. We now answer the question: How much time does a Brownian motion spend in a particular state? Proposition 55. The amount of time that a Brownian motion B spends in a fixed state a over the entire time horizon is the Lebesgue measure La of the time set {t ∈ R+ : B(t) = a}, and La = 0 a.s. Proof. Since La is nonnegative, it suffices to show E[La ] = 0. For n ∈ Z+ , consider the process Xn (t) = B(nt/n), t ≥ 0. Clearly Xn (t) → B(t) a.s. as n → ∞ for each t. Then by Fubini’s theorem,   P {B(t) = a}dt = lim P {Xn (t) = a}dt E[La ] = R+ R+ n→∞  P {Xn (t) = a}dt. ≤ lim inf n→∞

R+

The last integral (of a piecewise constant function) is 0 since Xn (t) has a normal distribution, and so E[La ] = 0. Proofs of the next two results are in [61, 64].

5.12 Brownian Bridge Process

379

Theorem 56. (Dvoretzky, Erd¨ os, and Kakutani 1961) Almost every sample path of a Brownian motion B does not have a point of increase: for positive t and δ, P {B(s) ≤ B(t) ≤ B(u) : (t − δ)+ ≤ s < t < u ≤ t + δ} = 0. Analogously, every sample path of B does not have a point of decrease. Theorem 57. (Paley, Wiener and Zygmund 1933) Almost every sample path of a Brownian motion is nowhere differentiable. More insights into the wild behavior of a Brownian motion path are given by its linear and quadratic variations. The (linear) variation of a real-valued function f on an interval [a, b] is Vab (f )

= sup

n %

& |f (tk ) − f (tk−1 )| : a = t0 < t1 < · · · < tn = b .

k=1

If this variation is finite, then f has the following properties: • It can be expressed as the difference f (t) = f1 (t) − f2 (t) of two increasing functions, where f1 (t) = Vat (f ). • The f has a derivative at almost every point in [a, b]. ! • Riemann-Stieltjes integrals of the form [a,b] g(t)df (t) exist. In light of these observations, Theorem 57 implies that almost every sample path of a Brownian has an “unbounded” variation on any finite interval of positive length. Further insight into the behavior of Brownian paths in terms of their quadratic variation is in Exercise 33. Because the sample paths of!a Brownian motion B have unbounded variation a.s., a stochastic integral [a,b] X(t)dB(t) for almost every sample path cannot be defined as a classical Riemann-Stieltjes integral. Another approach is used for defining stochastic integrals with respect to a Brownian motion or with respect to a martingale. Such integrals are the basis of the theory of stochastic differential equations.

5.12 Brownian Bridge Process We will now study a special Gaussian process called a Brownian bridge. Such a process is equal in distribution to a Brownian motion on [0, 1] that is restricted to hit 0 at time 1. An important application is its use in the nonparametric Kolmogorov-Smirnov statistical test that a random sample comes from a specified distribution. In particular, for large samples, the normalized difference between the empirical distribution and the true distribution is approximately the maximum of a Brownian bridge.

380

5 Brownian Motion

Throughout this section {X(t) : t ∈ [0, 1]} will denote a stochastic process on R, and B(t) will denote a standard Brownian motion. The process X is a Brownian bridge if it is a Gaussian process with mean 0 and covariance function E[X(s)X(t)] = s(1 − t), 0 ≤ s ≤ t ≤ 1. Such a process is equal in distribution to the following Brownian motion “tied down” at 1. Proposition 58. The process X(t) = B(t) − tB(1), t ∈ [0, 1], is a Brownian bridge. Proof. This follows since X is clearly a Gaussian process with zero mean and E[X(s)X(t)] = E[B(s)B(t) − tB(s)B(1)] − sE[B(1)B(t) − tB(1)2 ] = s(1 − t), s < t. The last equality uses E[B(u)B(v)] = u, for u ≤ v. Because of its relation to Brownian motion, many basic properties of a Brownian bridge X can be related to those of Brownian motion. For instance, X has continuous paths that are not differentiable. Note that the negation −X(t), and time reversal X(1 − t) are also Brownian bridges; related ideas are in Exercises 49 and 50. We will now show how a Brownian bridge is a fundamental process related to empirical distributions. Suppose that ξ1 , ξ2 , . . . are i.i.d. random variables with distribution F . The empirical distribution associated with ξ1 , . . . , ξn is Fn (t) = n−1

n 

1(ξk ≤ x),

x ∈ R, n ≥ 1.

k=1

This function is an estimator of F based on n samples from it. The estimator is unbiased since clearly E[Fn (x)] = F (x). It is also a consistent estimator since by the classical SLLN, Fn (x) → F (x)

a.s. as n → ∞.

(5.37)

This convergence is also uniform in x as follows. Proposition 59. (Glivenko-Cantelli) The empirical distributions satisfy sup |Fn (x) − F (x)| → 0 x

a.s. as n → ∞.

Proof. Consider any −∞ = x1 < x2 < · · · < xm = ∞, and note that since F and Fn are nondecreasing, for x ∈ [xk−1 , xk ], Fn (xk−1 ) − F (xk ) ≤ Fn (x) − F (x) ≤ Fn (xk ) − F (xk−1 ).

5.12 Brownian Bridge Process

381

Then sup |Fn (x) − F (x)| ≤ max |Fn (xk−1 ) − F (xk )| + max |Fn (xk ) − F (xk−1 )|. x

k

k

Letting n → ∞ and letting the differences xk − xk−1 tend to 0, and then applying (5.37) to the preceding display proves the assertion for continuous F . Exercise 40 proves the assertion when F is not continuous. An important application of empirical distributions concerns the following nonparametric text that a sample comes from a specified distribution. Example 60. Kolmorogov-Smirnov Statistic. Suppose that ξ1 , ξ2 , . . . are i.i.d. random variables with a distribution F that is unknown. As mentioned above, the empirical distribution Fn (x) is a handy unbiased, consistent estimator of F . Now, suppose we want to test the simple hypothesis H0 that the sample is from a specified distribution F , versus the alternative hypothesis H1 that the sample is not from this distribution. One approach is to use the classical chi-square test. Another approach is to use a test based on the Kolmogorov-Smirnov statistic defined by  |Fn (x) − F (x)|. Dn = x

This is a measure of the distance between the empirical distribution Fn and F (which for simplicity we assume is continuous). The test would reject H0 if Dn > c, and accept it otherwise. The c would be determined by the probability P {Dn > c|H0 } = α, for a specified level of significance α. The conditioning on H0 means conditioned that F is the true distribution. When n is large, one can compute c by using the approximation P {n1/2 Dn ≤ x|H0 } ≈ P { sup |B(t) − tB(1)| ≤ x} 0≤t≤1

=2

∞ 

(−1)k+1 e−2k

2

x2

.

k=1

This approximation follows from Theorem 61 below, and the summation formula is from [37]. We will now establish the limiting distribution of the Kolmogorov-Smirnov statistic. Theorem 61. The empirical distribution Fn associated with a sample from the distribution F satisfies n1/2 sup |Fn (x) − F (x)| → sup |X(t)|, d

x

where X is a Brownian bridge.

0≤t≤1

(5.38)

382

5 Brownian Motion

Proof. From Exercise 40, we know that Fn = Gn (F (·)) and sup |Fn (x) − F (x)| = sup |Gn (t) − t|, 0≤t≤1

x

 where Gn (t) = n−1 nk=1 1(Uk ≤ t) is the empirical distribution of the Un , which are i.i.d. with a uniform distribution on [0, 1]. The ξn and Un are defined on the same probability space. In light of this observation, assertion (5.38) is equivalent to n−1/2 Yn  → X, d

n where Yn (t) = k=1 (1(Uk ≤ t) − t), 0 ≤ t ≤ 1, and x = supt≤1 |x(t)|, for x ∈ D. To prove this convergence, it suffices by the continuous-mapping d theorem to show that n−1/2 Yn → X in D, since the map x → x from D to D is continuous (in the uniform topology). Let κn be a Poisson random variable with mean n that is independent of d the Uk . We will prove n−1/2 Yn → X based on Exercise 53 by verifying n−1/2 Yκn → X, d

n Letting Nn (t) =

 κn k=1

−1/2

P

Yn − Yκn  → 0.

(5.39) (5.40)

1(Uk ≤ t), where Nn (1) = κn , we can write

n−1/2 Yκn (t) = n−1/2 (Nn (t) − nt) − tn−1/2 (Nn (1) − n). Now Nn is a Poisson process on [0, 1] with rate n by the mixed-sample representation of Poisson processes in Theorem 26 of Chapter 3. Then from the functional central limit theorem for renewal processes in Corollary 51, the process n−1/2 (Nn (t) − nt) converges in distribution in D to a Brownian motion B. Applying this to the preceding display, it follows that the process n−1/2 Yκn (t) converges in distribution in D to the process B(t) − tB(1), which is a Brownian bridge. This proves (5.39). Next, note that n|  |n−κ  d    (1(Uk ≤ t) − t) n−1/2 Yn − Yκn  = n−1/2 sup 

0≤t≤1

=n

−1/2

k=1

|κn − n|Zn ,

(5.41)

where Zn = sup0≤t≤1 |G|κn −n| (t) − t|. Since κn is the sum of n i.i.d. Poisson random variables with mean 1, it follows by the classical central limit theorem d P that n−1/2 |κn − n| → |B(1)|. This convergence also implies |κn − n| → ∞. P

Now sup0≤t≤1 |Gn (t) − t| → 0 by Proposition 59 and so this convergence

5.13 Geometric Brownian Motion

383 P

is also true with n replaced by |κn − n|; that is, Zn → 0. Applying these observations to (5.41) verifies (5.40), which completes the proof.

5.13 Geometric Brownian Motion This section describes a geometric Brownian and related processes that are used for modeling stock prices or values of investments. Let X(t) denote the price of a stock (commodity or other financial instrument) at time t. Suppose the value of the stock has many small up and down movements due to continual trading. One possible model is a Brownian motion with drift X(t) = x + μt + σB(t). This might be appropriate as a crude model for local or short-time behavior. It is not very good, however, for medium or long term behavior, since the stationary increment property is not realistic (e.g., a change in price for the stock when it is $50 should be different from the change when the value is $200). A more appropriate model for the stock price, which is used in practice, is X(t) = xeμt+σB(t) .

(5.42)

Any process equal in distribution to X is a geometric Brownian motion with 2 drift μ and volatility σ. Since E[eαB(t) ] = eα t/2 , the moments of X(t) are given by 2 2 k ≥ 1. E[X(t)k ] = xk ekμt+k tσ /2 , For instance, E[X(t)] = xeμt+tσ

2

/2

= x[1 + (μ + σ 2 /2)t] + o(t)

as t ↓ 0.

The X is a diffusion process that satisfies the differential property dX(t) = (μ + σ 2 /2)X(t)dt + σX(t)dB(t). We will not prove this characterization, but only note that by the moment formula above, it follows that the instantaneous drift and diffusion parameters for X are μ(x, t) = (μ + σ 2 /2)x, σ(x, t) = σ 2 x2 . Although the geometric Brownian motion X does not have stationary independent increments, it does have a nice property of ratios of the increments. In particular, the ratio at the end and beginning of any time interval [s, s + t] is d X(t + s)/X(s) = eμt+σ(B(s+t)−B(s)) = eμt+σB(t) , so its distribution is independent of s. Also, these ratios over disjoint equallength time intervals are i.i.d. This means that as a model for a stock price,

384

5 Brownian Motion

one cannot anticipate any upward or downward movements in the price “ratios”. So in this sense, the market is equitable (or not biased). Does this also mean that the market is fair in the martingale sense that X(t) is a martingale with respect to B? The answer is generally no. However, X is a martingale if and only if μ + σ 2 /2 = 0 (a very special 2 condition). This follows since e−t(μ+σ /2) X(t) is a martingale with respect to B with mean x by Example 19 (and E[X(t)] = x when X(t) is such a martingale). The geometric Brownian model (5.42) has continuous paths that do not account for large discrete-jumps in stock prices. To incorporate such jumps, another useful model is as follows. Example 62. Prices with Jumps. Suppose the price of a stock at time t is given by X(t) = eY (t) , where Y (t) is a real-valued stochastic process with stationary independent increments (e.g., a compound Poisson or L´evy process). These properties of Y also ensure that the price ratios are i.i.d. in disjoint, fixedlength intervals. Assume as in Exercise 7 that the moment generating function ψ(α) = E[eαY (1) ] exists for α in a neighborhood of 0, and E[eαY (t) ] is continuous at t = 0 for each α. Then it follows that E[X(t)k ] = ψ(k)t ,

k ≥ 1.

In particular, if Y (t) is a compound Poisson process with rate λ and its jumps have the moment generating function G(α), then ψ(α) = e−λ(1−G(α)) . Consequently, E[X(t)k ] = e−λt(1−G(k)) , k ≥ 1. Other possibilities are that Y is a sum of a Brownian motion plus an independent compound Poisson process, or that X is the sum of a geometric Brownian motion plus an independent compound Poisson process. We will not get into advanced investment models using geometric Brownian motion such as Black-Scholes option pricing. However, the following illustrates an elementary computation for an option. Example 63. Stock Option. Suppose that the price of one unit of a stock at time t is given by a geometric Brownian motion X(t) = eB(t) . A customer has the option of buying one unit of the stock at a fixed time T at a price K, but the customer need not make the purchase. The value of the option to the customer is (X(t) − K)+ since the customer will not buy the stock if X(t) < K. We will disregard any fee that the customer would pay in order to obtain the option. The expectation of the option’s value is  ∞ + P {X(T ) − K > x} dx E[(X(T ) − K) ] = 0 ∞ = P {B(T ) > log(x + K)} dx. 0

5.14 Multidimensional Brownian Motion

385

This integral can be integrated numerically by using an approximation for the normal distribution of B(T ). A variation of this option is in Exercise 39.

5.14 Multidimensional Brownian Motion Brownian motions in the plane and in multidimensional spaces are natural models for phenomena driven by several independent (or dependent) single-dimension Brownian motions. This section gives some insight into these multidimensional processes. A stochastic process B(t) = (B1 (t), . . . , Bd (t)), t ≥ 0, in Rd is a multidimensional Brownian motion if B1 , . . . , Bd are independent Brownian motions on R. Many basic properties of this process follow from results in one dimension. For instance, the multidimensional integral formula  P {x + B(t) ∈ A}dx = |A|, Rd

the Lebesgue measure of A, follows from the similar formula for d = 1 in Exercise 6. The preceding integral is used in Section 5.15 for particle systems. Applications of Brownian motions in Rd typically involve intricate functions of the single-dimension components whose distributions determine system parameters (e.g., Exercise 54). Here is another classical application. Example 64. Bessel Processes. Associated with a Brownian motion B(t) in Rd , consider its radial distance to the origin defined by  1/2 R(t) = B1 (t)2 + · · · + Bd (t)2 ,

t ≥ 0.

Any process equal in distribution to R is a Bessel process of order d. When d = 1, we have the familiar reflected Brownian motion process R(t) = |B(t)|. Exercise 19 mentioned that this is a Markov process and it specifies its distribution (also recall Theorem 11). The R(t) is also a Markov process on R for general d. Its transition prob! ability P {R(t) ∈ A|R(0) = x} = A pt (x, y) dy has the density pt (x, y) = t−1 (xy)1−d/2 y d−1 Id/2−1 (xy/t), where Iβ is the modified Bessel function of order β > −1 defined by Iβ (u) =

∞  k=0

(u/2)2k+β , k!Γ (k + β + 1)

u ∈ R.

This is proved in [61]. We will only derive the density of R(t) when R(0) = 0. To this end, consider

386

5 Brownian Motion

R(t)2 /t = (B1 (t)2 + · · · + Bd (t)2 )/t = B1 (1)2 + · · · + Bd (1)2 . d

The last sum of squares of d independent standard normal random variables is known to have a χ-squared density f with d degrees of freedom. This f is a gamma density with parameters α = d/2 and λ = 1/2 (see the Appendix). Therefore, knowing that R(0) = 0, P {R(t) ≤ r} = P {R(t)2 /t ≤ r2 /t} =



r 2 /t

f (x) dx.

(5.43)

0

The density of R(t) is shown in Exercise 55. Although the hitting times of R(t) are complicated, we can evaluate their means. Consider the time τa = inf{t ≥ 0 : R(t) = a} to hit a value a > 0. This is a stopping time of R(t) and τa ≤ inf{t ≥ 0 : |B1 (t)| = a} < ∞ a.s. since the last stopping time is finite a.s. as noted in Theorem 11. Now, Exercise 56 shows that R(t)2 − t is a martingale with mean 0. Then the optional stopping result in Corollary 24 yields E[R(τa )2 − τa ] = 0. Therefore E[τa ] = a2 . We will now consider a multidimensional process whose components are “dependent” one-dimensional Brownian motions with drift. Let B(t) be a Brownian motion in Rd , and let C = {cij } be a d × d matrix of nonnegative  real numbers that are symmetric (cij = cji ) and nonnegative-definite ( i j ui uj cij ≥ 0, for u ∈ Rd ). As in the representation (5.8) of a multivariate normal vector, let A be a k × d matrix with transpose At and k ≤ d such that At A = C. Consider the process {X(t) : t ≥ 0} in Rd defined by X(t) = x + μt + B(t)A, where x and μ are in Rd . Any process equal in distribution to X is a generalized Brownian motion in Rd with initial value x, drift μ and covariance matrix C = At A. A major use for multidimensional Brownian motions is in approximating multidimensional random walks. The following result is an analogue of Donsker’s Brownian motion approximation for one-dimensional random walks in Theorem 40. Suppose that ξk , k ≥ 1, are i.i.d. random vectors in Rd with mean vector μ = (μ1 , . . . , μd ) and covariances cij = E[(ξk,i − μi )(ξk,j − μj )], 1 ≤ i, j ≤ d. Define the processes {Xn (t) : t ≥ 0} in Rd , for n ≥ 1, by

nt

Xn (t) = n−1/2



(ξk − μ),

t ≥ 0.

k=1 d

Theorem 65. Under the preceding assumptions, Xn → X, as n → ∞, where X is a generalized Brownian motion on Rd starting at 0, with no drift, and with covariance matrix {cij }.

5.15 Brownian/Poisson Particle System

Sketch of Proof. Consider Xn,i (t) = n−1/2 component of Cram´er-Wold

387

 nt

k=1 (ξk,i − μi ), which is the ith d Xn . By Donsker’s theorem, Xn,i → Xi for each i. Now, the d theorem states that (Xn,1 , . . . , Xn,d ) → (X1 , . . . , Xd ) in Rd d  d d d i=1 ai Xn,i → i=1 ai Xi in R for any a ∈ R . However,

if and only if the latter holds by another application of Donsker’s theorem. Therefore, the finite-dimensional distributions of Xn converge to those of X. To complete d the proof that Xn → X, it suffices to verify a certain tightness condition on the distributions of the processes Xn , which we omit.

5.15 Brownian/Poisson Particle System This section describes a system in which particles occasionally enter an Euclidean space and move about independently as Brownian motions and eventually exit. The system data and dynamics are represented by a marked Poisson process like those in Chapter 2. The focus is on characterizing certain Poisson processes describing particle locations over time and departures as intricate functions of the arrival process and particle trajectories. The Brownian motion structure of the trajectories lead to tractable probabilities. Consider a system of discrete particles that move about in the space Rd as follows. The locations and entry times  of the particles are represented by the space-time Poisson process N = n δ(Xn ,Tn ) on Rd × R, where Xn is the location in Rd at which the nth particle enters at time Tn . This Poisson process is homogeneous in that E[N (A × I)] = α|A|λ|I|, where |A| is the Lebesgue measure of the Borel set A. Here λ is the arrival rate of particles per unit time in any unit area, and α is the arrival rate per unit area in any unit time period. Note that, for bounded sets A and I, the N (A × I) is finite, but N (Rd × I) and N (A × R) are infinite a.s. (because their Poisson means are infinite). We assume that each particle moves in Rd independently as a d-dimensional Brownian motion B(t), t ≥ 0, for a length of time V with distribution G and then exits the space. d More precisely, let Vn , n ∈ Z, be independent with Vn = V , and let Bn , d n ∈ Z, be independent with Bn = B. Assume {Bn }, {Vn } are independent and independent of N . Viewing the Bn and Vn as independent marks of (Xn , Tn ), the data for the entire system is defined formally by the marked Poisson process  δ(Xn , Tn , Bn , Vn ), on S = Rd × R × C(R, Rd ) × R+ . M= n

388

5 Brownian Motion

Here C(R, Rd ) denotes the set of continuous functions from R to Rd . The mean measure of M is given by E[M (A × I × F × J)] = α|A|λ|I|P {B ∈ F }P {V ∈ J}. The interpretation is that the nth particle has a space-time entry at (Xn , Tn ) and its location at time t is given by Xn + Bn (t − Tn ), where t − Tn ≤ Vn . At the end of its sojourn time Vn it exits the system at time Tn + Vn from the location Xn + Bn (Vn ). Let us see where the particles are at any time t. It is not feasible to account for all the particles that arrive up to time t, which is N (Rd × (−∞, t]) = ∞. So we will consider particles that enter in a bounded time interval I prior to t, which is t − I (e.g., t − [a, b] = [t − b, t − a]). Now, the number of particles that enter Rd in a time interval I prior to t and are in A at time t is  δ(Tn ,Xn +Bn (t−Tn )) (I × A)1(Vn > t − Tn ). Nt (I × A) = n

The Nt is a point process on R+ × Rd . Proposition 66. The family of point processes {Nt : t ∈ R} is stationary in t, and each Nt is a Poisson process on R+ × Rd with mean measure  E[Nt (I × A)] = αλ|A| (1 − G(u))du. (5.44) I

Proof. By the form of its mean measure, the Poisson process M with its time axis shifted by an amount t is  d St M = δ(Xn ,Tn +t,Bn ,Vn ) = M, t ∈ R. n

Therefore, M is stationary in the time axis. To prove that Nt is stationary in t, it suffices by Proposition 104 in Chapter 3 to show that Nt = f (St M ), for some function f .  Accordingly, for a locally-finite counting measure ν = n δ(xn ,tn ,bn ,vn ) on S, define the counting measure f (ν) on R+ × Rd by  δ(−tn ,xn +bn (−tn )) 1(vn > −tn ). f (ν) = n

Then clearly, Nt = f (St M ), which proves that Nt is stationary. Next, note that Nt is a deterministic map of the Poisson process M restricted to the subspace {(x, s, b, v) ∈ S : s ≤ t, v > t − s}, in which any point (x, s, b, v) in the subspace is mapped to (s, x + b(t − s)). Then by Theorem 32 in Chapter 2, Nt is a Poisson process with mean measure given by

5.16 G/G/1 Queues in Heavy Traffic



 E[Nt (I × A)] = αλ t−I

Rd

389

 P {x + B(t − s) ∈ A} dx P {V > t − s}ds.

Because B is a Brownian motion, the integral in parentheses reduces to |A| by Exercise 6. Therefore, using the change of variable u = t − s in the last expression yields (5.44). Next, let us consider departures from the system. The number of particles that enter Rd during the time set I and depart from A during the time set ¯ (I × A × J), where N ¯ is a point process of the form J is N  ¯= N δ(Tn ,Xn +Bn (Vn ),Tn +Vn ) on {(s, x, u) ∈ R × Rd × R : s ≤ u}. n

¯ is a Poisson process with Proposition 67. The point process of departures N mean measure given by  ¯ (I × A × J)] = αλ|A| E[N |I ∩ (J − v)|dG(v). (5.45) R+

¯ is a deterministic map g of the Poisson process M , Proof. By its definition, N where g(Xn , Tn , Bn , Vn ) = (Tn , Xn + Bn (Vn ), Tn + Vn ). Then by Theorem 32 ¯ is a Poisson process with mean in Chapter 2, N     ¯ (I ×A×J)] = αλ 1(s+v ∈ J) P {x+B(v) ∈ A} dx dG(v) ds. E[N I

R+

Rd

The integral in parentheses reduces to |A| by Exercise 6. Then an interchange of the order of integration in the last expression yields (5.45). There are several natural generalizations of the preceding model with more dependencies among the marks and entry times and points, e.g., see Exercise ¯ may still be Poisson, their mean values 51. Although the processes Nt and N would be more complicated.

5.16 G/G/1 Queues in Heavy Traffic Section 4.20 of Chapter 4 showed that the waiting times Wn for successive items in a G/G/1 queueing system are a function of a random walk. This suggests that the asymptotic behavior of these times can be characterized by the Donsker Brownian motion approximation of a random walk, and that is what we shall do now. We first describe the limit of Wn when the traffic intensity ρ = 1, and then present a more general FCLT for the Wn when the system is in heavy traffic: the traffic intensity is approximately 1. Consider a G/G/1 queueing system, as in Section 4.20 of Chapter 4, in which items arrive at times that form a renewal process with inter-arrival

390

5 Brownian Motion

times Un , and the service times are i.i.d. nonnegative random variables Vn that are independent of the arrival times. The service discipline is first-comefirst-served with no preemptions. The inter-arrival and service times have finite means and variances, and the traffic intensity of the system is ρ = E[V1 ]/E[U1 ]. For simplicity, assume the system is empty at time 0. Our interest is in the length of time Wn that the nth arrival waits in the queue before being processed. Section 4.20 of Chapter 4 showed that these waiting times satisfy the Lindley recursive equation Wn = (Wn−1 + Vn−1 − Un )+ ,

n ≥ 1,

and consequently, n 

Wn = max

0≤m≤n

(Vk−1 − Uk ).

(5.46)

k=m+1

Under the assumptions on the Un and Vn , it follows that d

Wn = max Sm , 0≤m≤n

(5.47)

n where Sn = m=1 ξm and ξm = Vm − Um . In case ρ < 1, Theorem 118 of Chapter 4 noted that d

Wn → max Sm . 0≤m 0. Then d /n → W M

in D(R+ ) as n → ∞,

where M (t) = maxs≤t B(s), the maximum process for a standard Brownian motion B. Hence Wn d d √ → M (1) = |B(1)|. σ n Proof. Note that

5.16 G/G/1 Queues in Heavy Traffic

391

1 1 d /n (t) = √ max Sm = √ sup S[ns] . W σ n m≤ nt σ n s≤t d /n (t) = That is, W f (Xn )(t), t ≥ 0, where

Xn (t) =

S nt √ , σ n

t ≥ 0,

and f : D(R+ ) → D(R+ ) is the supremum map defined by f (x)(t) = sup x(s), 0≤s≤t

x ∈ D(R+ ).

Now the random walk Sn has steps with mean E[ξ1 ] = 0, since ρ = 1; and d

σ 2 = Var(ξ1 ). Then Xn → B by Donsker’s theorem. Next, it is clear that if xn − x → 0 in D[0, T ], then f (xn ) − f (x) ≤ xn − x → 0

in D[0, T ].

P

Then since Xn − B → 0 in D[0, T ] for each T , it follows that P

f (Xn ) − f (B) → 0 in D(R+ ). d /n = f (Xn ) and f (B) = M proves W /n → This along with W M. d d /n (1) → M (1) = |B(1)|, which proves the second assertion; In particular, W d

that M (1) = |B(1)| follows by Theorem 11. The preceding result suggests that for any G/G/1 system in which ρ ≈ 1, d /n ≈ M would be valid. A formal statement to this effect the approximation W is as follows. Consider a family of G/G/1 systems indexed by a parameter r with interr arrival times service times Vnr . Denote the other quantities by ρr , nUn and r r r Wn , Sn = m=1 ξm , etc., and consider the process /r (t) = W

r W rt √ , σ r

t ≥ 0.

Theorem 69. Suppose the family of G/G/1 systems are such that ρr → 1, sup E[(ξ1r − E[ξ1r ])2+ε ] < ∞,

for some ε > 0,

r

r1/2 E[ξ1r ] → 0,

Var(ξ1r ) → σ 2 > 0,

d /r → M in D(R+ ) as r → ∞. Then W

as r → ∞.

392

5 Brownian Motion

/r = f (Xr ), where f is the supremum Proof. As in the proof of Theorem 68, W √ map and Xr (t) = S rt /σ r, t ≥ 0. Then to prove the assertion, it suffices d

to show that Xr → B as r → ∞. Now, we can write Xr (t) = Yr (t) + (rt/r)r1/2 E[ξ1r ], where

rt 1  r (ξ − E[ξ1r ]). Yr (t) = √ σ r m=1 m d

Under the hypotheses, Yr → B by a theorem of Prokhorov 1956, and hence d Xr → B. The preceding results are typical of many heavy-traffic limit theorems that one can obtain for queueing and related processes by the framework presented by Whitt [115]. In particular, when a system parameter, such as the waiting time above, can be expressed as a function of the system data (cumulative input and output processes), and that data under an appropriate normalization converges in distribution, then under further technical conditions, the system parameter also converges in distribution to the function of the limits of the data. Here is one of the general models in [115]. Example 70. Generalized G/G/1 System. Consider a generalization of the G/G/1 systems above in which the inter-arrival times Unr and service times Vnr (the system data) are general random variables that may be dependent. Then the waiting times Wnr can still be expressed as a function of the system data as in (5.46). In other words, Wnr = S˜n − min S˜m 0≤m≤n

where S˜n =

n

k=1 (Vk−1

/r (t) = W

− Uk ). As above, consider the processes r W rt √ , σ r

Xr (t) =

S rt √ , σ r

t ≥ 0.

/r = h(Xr ), where h : D(R) → D(R) is the one-sided Then we can write W reflection map defined by h(x)(t) = x(t) − inf x(s), 0≤s≤t

t ≥ 0.

The reflection map h (like the supremum map above) is continuous in the uniform topology on D[0, T ] since h(x) − h(y) ≤ 2x − y,

x, y ∈ D[0, T ].

5.17 Brownian Motion in a Random Environment

393

Then the continuous-mapping theorem yields the following result. r d /r → W in D(R) as Convergence Criterion. If Xr → X in D(R), then W r → ∞, where W (t) = X(t) − inf X(s), t ≥ 0. 0≤s≤t

To apply this for a particular situation, one would use properties of the r inter-arrival times and service times (as in Theorem 69) to verify Xr → X. There are a variety of conditions under which the limit X is a Brownian motion, a process with stationary independent increments, or an infinitely divisible process; and other topologies on D(R+ ) are often appropriate [115].

5.17 Brownian Motion in a Random Environment Section 3.14 describes a Poisson process with a random intensity measure called a Cox process. The random intensity might represent a random environment or field that influences the locations of points. This section describes an analogous randomization for Brownian motions in which the time scale is determined by a stochastic process. Let {X(t) : t ∈ R+ } and η = {η(t) : t ∈ R+ } be real-valued stochastic processes defined on the same probability space, such that η(t) is a.s. nondecreasing with η(0) = 0 and η(t) → ∞ a.s. as t → ∞. The process X is a Brownian motion directed by η if the increments of X are conditionally independent given η, and, for any s < t, the increment X(t) − X(s) has a conditional normal distribution with variance τt − τs . These conditions, in terms of the moment generating function for the increments of X, say that, for 0 = t0 < t1 < · · · < tn and u1 , . . . , un in R+ , n

% &   E exp ui [X(ti ) − X(ti−1 )]  η i=1

= exp

n %1 

2

& u2i [η(ti ) − η(ti−1 )]

(5.48) a.s.

i=1

A directed Brownian motion is equal in distribution to a standard Brownian motion with random time parameter as follows. Remark 71. A process X is a Brownian motion directed by η if and only if d X = B ◦ η  , where B and η  are defined on a common probability space d such that B is a standard Brownian motion independent of η  , and η  = η. This follows by the definition above and consideration of the moment generating function of the increments of the processes. The process B ◦ η  is a

394

5 Brownian Motion

Brownian motion subordinated to η (like a Markov chain subordinated to a Poisson process, which we saw in Chapter 4). In case η is strictly increasing, Exercise 43 shows that X = B ◦ η a.s., where B is defined on the same probability space as X and η. A Brownian motion X directed by η inherits many properties of standard Brownian motions. The proofs usually follow by conditioning on η and using properties of this process. Here are some examples. Example 72. E[X(t)] = 0, and Var[X(t)] = E[η(t)]. Example 73. Consider τaX = inf{t : X(t) ≥ a}. Then  P {τaX ≤ t} = P {η(t) ≤ u}P {τa ∈ du}, R+

where τa = inf{t : B(t) = a}. Example 74. Suppose that X1 , . . . , Xm are Brownian motions directed by η1 , . . . , ηm , respectively, and (X1 , η1 ), . . . , (Xm , ηm ) are independent. Then X(t) = X1 (t) + · · · + Xm (t) is a Brownian motion directed by η(t) = η1 (t) + · · · ηm (t). Example 75. FCLT. For a Brownian motion X directed by η, define Xr (t) = b−1/2 X(rt) r

t ≥ 0, −1/2

where br → ∞ are constants. By Remark 71 and the scaling br

d

B = B(br ·),

d

we can write Xr = B ◦ Yr , where Yr (t) = η  (rt)/br and the Brownian motion B and η  are independent. Then by the property of the composition mapping in Proposition 48, we obtain the following result, where I is the identity d d function: If Yr → I in D(R+ ), then Xr → B in D(R+ ).

5.18 Exercises For the following exercises, B will denote a standard Brownian motion. Exercise 1. Show that each of the following processes is also a Brownian motion. (a) B(t + s) − B(s) Translated process for a fixed time s. (b) −B(t) Reflected process (c) c−1/2 B(ct) Scaling, for c > 0 (d) B(T ) − B(T − t), t ∈ [0, T ], Time-reversal on [0, T ] for fixed T .

5.18 Exercises

395

Exercise 2. The time-inversion of a Brownian motion B is the process X(0) = 0, X(t) = tB(1/t), t > 0. Prove that X is a Brownian motion. First show that X(t) → 0 a.s. as t ↓ 0. Exercise 3. Suppose that h : R+ → R+ is a continuous, strictly increasing function with h(0) = 0, and h(t) ↑ ∞. Find the mean and covariance functions d

for the process B(h(t)). Show that B(h(t)) = X(t) for each t, where X(t) = (h(t)/t)−1/2 B(t). Are the processes B(h(·)) and X equal in distribution? Exercise 4. For 0 < s < t, show that the conditional density of B(s) given B(t) = b is normal with conditional mean and variance E[B(s)|B(t) = b] = bs/t,

Var[B(s)|B(t) = b] = s(t − s)/t.

For t1 < s < t2 , show that P {B(s) ≤ x|B(t1 ) = a, B(t2 ) = b} = P {B(s−t1 ) ≤ x−a|B(t2 −t1 ) = b−a}. Using these properties prove that the conditional density of B(s) given B(t1 ) = a, B(t2 ) = b is normal with conditional mean and variance a + (b − a)(s − t1 ) (t2 − t1 )

and

(s − t1 )(t2 − s) . (t2 − t1 )

Exercise 5. Consider the process X(t) = ae−αt + σ 2 B(t), t ≥ 0, where a, α ∈ R and σ > 0. Find the mean and covariance functions for this process. Show that X is a Gaussian process by applying Theorem 5. Does X have independent increments, and are these increments stationary? Is X a martingale, submartingale or supermartingale with respect to B? Exercise 6. For a density f on R that is symmetric (f (x) = f (−x), x ∈ R), show that    f (y)dy dx = |A| (The Lebesgue measure of A). R

A−x

Use this to verify that, for a Brownian motion B(t),  P {x + μt + σB(t) ∈ A}dx = |A|, R

independent of t, μ and σ. Exercise 7. Let Y (t) be a real-valued process with stationary independent increments. Assume that the moment generating function ψ(α) = E[eαY (1) ] exists for α in a neighborhood of 0, and g(t) = E[eαY (t) ] is continuous at t = 0 for each α. Show that g(t) is continuous in t for each α, and that

396

5 Brownian Motion

g(t) = ψ(α)t . Use the fact that g(t + u) = g(t)g(u), t, u ≥ 0, and that the only continuous solution to this equation has the form g(t) = etc , for some c that depends on α (because g(t) depends on α). Show that E[Y (t)] = tE[Y (1)],

Var[Y (t)] = tVar[Y (1)].

Exercise 8. Let X(t) = μt + σB(t) be a Brownian motion with drift, where μ > 0. Suppose that a signal is triggered whenever M (t) = sups≤t X(s) reaches the levels 0, 1, 2, . . .. So the nth signal is triggered at time τn = inf{t : M (t) = n}, n ≥ 0. Show that these times of the signals form a renewal process and find the mean, variance and Laplace transform of the times between signals. In addition, obtain this information under the variation in which a signal is triggered whenever M (t) reaches the levels 0 = L0 < L1 < L2 < . . . , where Ln − Ln−1 are independent exponential random variables with rate λ. Exercise 9. When is a Gaussian Process Stationary? Recall that a stochastic process is stationary if its finite-dimensional distributions are invariant under shifts in time. This is sometimes called strong stationarity. A related notion is that a real-valued process {X(t) : t ≥ 0} is weakly stationary if its mean function E[X(t)] is a constant and its covariance function Cov(X(s), X(t)) depends only on |t − s|. Weak stationarity does not imply strong stationarity. However, if a real-valued process is strongly stationary and its mean and covariance functions are finite, then the process is weakly stationary. Show that a Gaussian process is strongly stationary if and only if it is weakly stationary. Exercise 10. Suppose that Yn , n ∈ Z, are independent normally distributed random variables with mean μ and variance σ 2 . Consider the moving average process n ∈ Z, Xn = a0 Yn + a1 Yn−1 + · · · + am Yn−m , where a0 , . . . , am are real numbers. Show that {Xn : n ∈ Z} is a Gaussian process that is stationary and specify its mean and covariance functions. Justify that this process is not Markovian and does not have stationary independent increments. Exercise 11. Derive the mean and variance formulas in (5.14) for M (t). Exercise 12. Equal Cruise-Control Settings. Two autos side by side on a highway moving at 65 mph attempt to move together at the same speed by setting their cruise-control devises at 65 mph. As in many instances, nature does not always correspond to one’s wishes and the actual cruise-control settings are independent normally distributed random variables V1 and V2 with mean μ = 65 and standard deviation σ = 0.4. Find the probability that the autos move at the same speed. Find P {|V1 − V2 | < .3}.

5.18 Exercises

397

Exercise 13. Letting τa = inf{t : B(t) = a}, find the probability that B hits 0 in the time interval (τa , τb ), where 0 < a < b. Exercise 14. Arc sine Distribution. Let U = sin2 θ, where √ θ has a uniform distribution on [0, 2π]. Verify that P {U ≤ u} = arcsin u, u ∈ [0, 1], which is the arc sine distribution. Let X1 , X2 be independent normally distributed random variables with d mean 0 and variance 1. Show that X12 /(X12 + X22 ) = U . Hint: In the inte2 2 2 gral representation for P {X1 /(X1 + X2 ) ≤ u}, use polar coordinates where (x1 , x2 ) is mapped to r = (x21 + x22 )1/2 and θ = arctan x2 /x1 . d

Is it true that U = 1 − U ? Exercise 15. Prove Theorem 15 for u = 1. Using this result, find the distribution of η = sup{t ∈ [0, u] : B(t) = 0} for u = 1. ˜ are independent Brownian motions. Exercise 16. Suppose that B and B ˜ a ) at the time when B hits a, Find the moment generating function of B(τ ˜ a ) : a ∈ R+ } considered as a which is τa = inf{t : B(t) = a}. Show that {B(τ stochastic process has stationary independent increments. Exercise 17. For the hitting time τ = inf{t > 0 : B(t) ∈ (−a, a)}, where a > 0, prove that its Laplace transform is √ E[e−λτ ] = 1/ arccos(a 2λ). Mimic the proof of Theorem 32 using the facts that B(τ ) is independent of τ , and P {B(τ ) = −a} = P {B(τ ) = a} = 1/2. Exercise 18. Continuation. In the context of the preceding exercise, verify that E[τ ] = a2 , and E[τ 2 ] = 5a4 /3. Exercise 19. Let M (t) = sups≤t B(s), and consider the process X(t) = M (t) − B(t), t ≥ 0. Show that d

d

X(t) = M (t) = |B(t)|,

t ≥ 0.

(The processes X and |B(·)| are Markov processes on R+ with the same transition probabilities, and hence they are equal in distribution. However, they are not equal in distribution to M , since the latter is nondecreasing.) Show that  z P {X(t) ≤ z|X(s) = x} = [ϕ(y − x; t − s) + ϕ(−y − x; t − s)] dy, −∞

where ϕ(x; t) = e−x

2

√ /2t / 2πt. In addition, verify that 2

P {M (t) > a|X(t) = 0} = e−a

/2t

.

398

5 Brownian Motion

Exercise 20. Reflection Principle for Processes. Suppose that τ is an a.s. finite stopping time for a Brownian motion B, and define X(t) = B(t ∧ τ ) − (B(t) − B(t ∧ τ )),

t ≥ 0. d

Prove that X is a Brownian motion. Hint: Show that X = B by using the strong Markov property along with the process B  (t) = B(τ + t) − B(τ ), t ≥ 0, and the representations B(t) − B(t ∧ τ ) = B  ((t − τ )+ ),

B(t) = B(t ∧ τ ) + B  ((t − τ )+ ).

Exercise 21. Continuation. For the hitting time τa = inf{t > 0 : B(t) = a}, show that the reflected process X(t) = B(t)1(τa ≤ t) + (2a − B(t))1(τa > t) is a Brownian motion. Use the result in the preceding exercise. Exercise 22. Use the reflection principle to find an expression for P {B(t) > y, min B(s) > 0}. s≤t

Exercise 23. The value of an investment is modeled as a Brownian motion with drift X(t) = x + μt + σB(t), with an upward drift μ > 0. Find the distribution of M (t) = mins≤t X(s). Use this to find the distribution of the lowest value M (∞) = inf t∈R+ X(t) when x = 0. In addition, find P {X(t) − M (t) > a},

a > 0.

Exercise 24. The values of two stocks evolve as independent Brownian motions X1 and X2 with drifts, where Xi (t) = xi + μi t + σi Bi (t), and x1 < x2 . Find the probability that X2 will stay above X1 for at least s time units. Let τ denote the first time that the two values are equal. Find E[τ ] when μ1 < μ2 and when μ1 > μ2 . Exercise 25. Show that P {B(1) ≤ x|B(s) ≥ 0, s ∈ [0, 1]} = 1 − e−x

2

/2

.

˜ Hint: Consider B(t) = B(1) − B(1 − t) and show that the conditional prob˜ ˜ (1) = B(1)}, ˜ ˜ (t) = sups≤t B(s). ˜ ability is equal to P {B(1) ≤ x|M where M N (t) Exercise 26. Consider a compound Poisson process Y (t) = n=1 ξn , where N (t) is a Poisson process with rate λ and the ξn are i.i.d. and independent of N . Suppose ξ1 has a mean μ, variance σ 2 and moment generating function ψ(α) = E[eαξ1 ]. Show that the following are martingales with respect to Y :

5.18 Exercises

399

X1 (t) = Y (t) − λμt, X2 (t) = (Y (t) − λμt)2 − tλ(μ2 + σ 2 ), αY (t)−λt(1−ψ(α)) , t ≥ 0. X3 (t) = e Find the mean E[Xi (t)], for each i. Exercise 27. Suppose X(t) denotes the stock level of a certain product at !t time t and the holding cost up to time t is Y (t) = h 0 X(s) ds, where h is the cost per unit time of holding one unit in inventory. Show that if X is a Brownian motion B, then the mean and covariance functions of Z are E[Y (t)] = 0,

Cov(Y (s), Y (t)) = h2 s2 (t/2 − s/6),

s ≤ t.

Find the mean and covariance functions of Y if X(t) = x + μt + σB(t), a Brownian motion with drift; or if X is a compound Poisson process as in the preceding problem. !t Exercise 28. Prove that Y (t) = 0 B(s) ds, t ≥ 0, is a Gaussian process with mean 0 and E[Y (t)2 ]  = t3 /3. n Hint: Show that Z = i=1 ui X(ti ) has a normal distribution for any t1 , . . . , tn in R+ , and u1 , . . . , un in R. Since a Riemann integral is the limit of sums of rectangles, we know that Z = limn→∞ Zn , where Zn =

n  i=1

ui

n 

(ti /n)B(kti /n).

k=1

Justify that each Zn is normally distributed, and that its limit (using moment generating functions) must also be normally distributed. !t Exercise 29. Continuation. Suppose X(t) = exp{ 0 B(s) ds}, t ≥ 0. Verify 6 that E[X(t)] = et /6 . Exercise 30. Let Y = {Yn , n ≥ 0} be independent random variables (that need not be identically distributed) with finite means. Suppose X0 is a deterministic function of Y0 with finite mean. Define Xn = X0 +

n  i=1

Yi ,

Xn

= X0

n 

Yi ,

n ≥ 1.

i=1

Show that Xn is a discrete-time martingale with respect to Y if E[Yi ] = 0. How about if E[Yi ] ≥ 0? Is Xn a martingale with respect to itself? What can you say about Xn if the Yi are positive with E[Yi ] = 1? or ≥ 1? Exercise 31. Wald Equation for Discounted Sums. Suppose that ξ0 , ξ1 , . . . are costs incurred at discrete times they are i.i.d. with mean μ. Consider and n the discounted cost process Zn = m=0 αm ξm , where α ∈ (0, 1) is a discount

400

5 Brownian Motion

factor. Suppose that τ is a stopping time of the process ξn such that E[τ ] < ∞ and E[ατ ] exists for some 0 < α < 1. Prove that E[Zτ ] =

μ(1 − αE[ατ ]) . (1 − α)

Do this by finding a convenient martingale and applying the optional stopping theorem; there is also a direct proof without the use of martingales. n Next, consider the process Sn = m=0 ξm , n ≥ 0, and show that E[

τ 

αm Sm ] =

m=0

μE[τ ] αμ(1 − αE[ατ ]) − . 1−α (1 − α)2

Exercise 32. Continuation. In the preceding problem, are the results true under the weaker assumption that ξ0 , ξ1 , . . . are such that E[ξ0 ] = μ and E[ξn |ξ0 , . . . , ξn−1 ] = μ, n ≥ 1? Exercise 33. Quadratic Variation. Consider the quadratic increments  V (t) = (B(ti ) − B(ti−1 ))2 i

over a partition 0 = t0 < t1 < · · · < tk = t of [0, t]. Verify E[V (t)] = t and  (ti − ti−1 )2 Var[B(1)2 ]. Var[V (t)] = i

Next, for each n ≥ 1, let Vn (t) denote a similar quadratic increment sum for a partition 0 = tn0 < tn1 < · · · < tnkn = t, where maxk (tnk − tn,k−1 ) → 0. Show that E[Vn (t)2 − t] → 0 (which says Vn (t) converges in mean square distance to t). The function t is the quadratic variation of B in that it is the unique function (called a compensator) such that B(t) − t is a martingale. One can also show that Vn → t a.s. when the partitions are nested. Exercise 34. Random Time Change of a Martingale. Suppose that X is a martingale with respect to Ft and that {τt : t ≥ 0} is a nondecreasing process of stopping times of Ft that are bounded a.s. Verify that X(τt ) is a martingale with respect to Ft , and that its mean is E[X(τt )] = E[X(0)]. Exercise 35. Optional Switching. Suppose that Xn and Yn are two martingales with respect to Fn that represent values of an investment in a fair market that evolve under two different investment strategies. Suppose an investor begins with the X-strategy and then switches to the Y -strategy at an a.s. finite stopping time τ of Fn , and that Xτ = Yτ . Then the investment value would be

5.18 Exercises

401

Zn = Xn 1(n < τ ) + Yn 1(n ≥ τ ), where Xτ is the value carried forward at time τ . Show that there is no benefit for the investor to switch at τ by showing that Zn is a martingale. Use the representation Zn+1 = Xn+1 1(n < τ ) + Yn+1 1(n ≥ τ ) − (Xτ − Yτ )1(τ = n + 1). Exercise 36. Prove that if σ and τ are stopping times of Ft , then so are σ ∧ τ and σ + τ . Exercise 37. Prove that if X(t) and Y (t) are submartingales with respect to Ft , then so is X(t) ∨ Y (t). Exercise 38. Consider a geometric Brownian motion X(t) = xeB(t) . Find the mean and distribution of τa = inf{t : X(t) = a}. Exercise 39. Recall the investment option in Example 63 in which a customer may purchase a unit of a stock at a price K at time T . Consider this option with the additional stipulation that the customer “must” purchase a unit of the stock before time T if its price reaches a prescribed level a, and consequently the other purchase at time T is not allowed. In this setting, the customer must purchase the stock at the price a prior to time t, if maxx≤t X(s) = eM(t) > a, where M (t) = maxs≤t B(s). Otherwise, the option of a purchase at the price K is still available at time T . In this case, the value of the option is Z = (1 − a)1(M (T ) > log a) + (X(T ) − K)+ 1(M (T ) ≤ log a). Prove that  √ E[Z] = 2(1 − a)[1 − Φ(log a/ T )] +

log a

0

 0

y

(ey − K)fT (x, y) dx dy,

where ft (x, y) is the joint density of B(t), M (t) and Φ is the standard normal distribution. Verify that ft (x, y) =

2(2y − x) −(2y−x)2 /2t √ e , 2πt3

x ≤ y, y ≥ 0.

Verify that E[Z] is minimized at the value a at which the integral term equals the preceding term. This would be the worst scenario for the customer. Exercise 40. Prove Proposition 59 when F is not continuous. Use the fact d from Exercise 11 in Chapter 1 that ξn = F −1 (Un ), where Un are i.i.d. with the uniform distribution on [0, 1]. By Theorem 16 in the Appendix, you can on the same probability space. Then the empirical assume the ξn and Un are n distribution Gn (t) = n−1 k=1 1(Uk ≤ t) of the Un satisfies Fn = Gn (F (·)). Conclude by verifying that

402

5 Brownian Motion

sup |Fn (x) − F (x)| = sup |Gn (t) − t| → 0 x

a.s. as n → ∞,

t≤1

where the limit is due to Proposition 59 for a continuous distribution. Exercise 41. Let X be a Brownian motion directed by η. Suppose the process η has stationary independent nonnegative increments and E[e−αη(t) ] = ψ(α)t , where ψ(α) = E[e−αη(1) ]. Determine the moment generating function of X(1) (as a function of ψ) and show that X has stationary independent increments. Exercise 42. Show that if Xn is a nonnegative supermartingale, then the limit X = limn→∞ Xn exists a.s. and E[X] ≤ E[X0 ]. Use the submartingale convergence theorem and Fatou’s lemma. Exercise 43. Let X be a Brownian motion directed by η, where the paths of η are strictly increasing a.s. Show that X(t) = B(η(t)), t ∈ R+ , where B is a Brownian motion (on the same probability space as X and η) that is independent of η. Hint: Define B(t) = X(ˆ η (t)), where ηˆ(t) = inf{s ≥ 0 : η(s) = t}. Argue that ηˆ(η(t)) = t and X(t) = B(η(t)), for each t, and that n n & 

% 1 2  ui [B(ti ) − B(ti−1 )]  η = exp{ u (ti − ti−1 )} E exp 2 i=1 i i=1

a.s.

Thus B is a Brownian motion and it is independent of η since the last expression is not random. Exercise 44. As a variation of the model in Section 5.17, a real-valued process X is a Brownian motion with drift μ and variability σ directed by η if, for 0 = t0 < t1 < · · · < tn and u1 , . . . , un in R+ , n

% &   E exp ui [X(ti ) − X(ti−1 )]  η i=1

= exp

n %

ui μ[η(ti ) − η(ti−1 )]

i=1

n & 1 2 2 ui σ [η(ti ) − η(ti−1 )] a.s. 2 i=1

Show that if t−1 η(t) → c a.s. for some c > 0, then t−1 X(t) → cμ,

a.s.

t−1 max X(s) → cμ a.s. s≤t

Exercise 45. Prove Theorem 39 on continuous mappings for “separable” metric spaces by applying the coupling result for the a.s. representation of convergence in distribution (Theorem 16 in the Appendix).

5.18 Exercises

403

Exercise 46. Use Donsker’s theorem to prove that P {n−1/2 (Sn − min Sk ) > x} → e−x k≤n

2

/2

.

Exercise 47. In the context of Donsker’s theorem, consider the range Yn = max Sk − min Sk k≤n

k≤n

d

of the random walk. Show that n−1/2 Yn → Y , where E[Y ] = 2 Y as a functional of a Brownian motion.

$ 2/π. Express

Exercise 48. FCLT for Markov Chains. Let Yn be an ergodic Markov chain on a countable state space S with stationary distribution π. For a function f : S → R, consider the process

nt 1  Xn (t) = √ [f (Yk ) − a], σ n

t ∈ [0, 1].

k=1

d

Specify assumptions (and a, σ) under which Xn → B as n → ∞, and prove it. Exercise 49. Show that if B is a Brownian motion, then (1 − t)B(t/(1 − t)) and tB(1 − t)/t) are Brownian bridges. In addition, show that if X is a Brownian bridge, then (1+t)X(t/(1+t)) and (1+t)X(1/(1+t)) are Brownian motions. Hint: Take advantage of the Gaussian property. Exercise 50. For a Brownian bridge X, find expressions for the distribution of M (1) = mint≤1 X(t) and M (1) = maxt≤1 X(t). Exercise 51. Consider the Brownian/Poisson model in Section 5.15 with the difference that the Poisson input process N is no longer time-homogeneous and its mean measure is E[N (A × I)] = α|A|Λ(I), where Λ is a measure on the time axis R. As in Section 5.15, let Nt (I × A) denote the number of particles that enter S in the time interval t − I and are in A at time t. Verify that each Nt is a Poisson process with  P {V > t − s}Λ(ds). E[Nt (I × A)] = α|A| t−I

Is the family {Nt : t ∈ R} stationary as it is in Section 5.15?

404

5 Brownian Motion d

Exercise 52. Continuity of Addition in D × D. Assume that (Xn , Yn ) → (X, Y ) in D × D and Disc(X) ∩ Disc(Y ) is empty a.s., where Disc(x) denotes d

the discontinuity set of x. Prove that Xn + Yn → X + Y . d d d ˆn → ˆn → X in D and Xn −X 0. Do Exercise 53. Show that Xn → X in D if X d d this by proving and applying the property that if Xn → X in D and Yn → y d d in D for non-random y, then (Xn , Yn ) → (X, y) in D2 and Xn + Yn → X + y in D, when X has continuous paths a.s.

Exercise 54. Suppose that (B1 (t), B2 (t)) is a Brownian motion in R2 , and define τa = inf{t : B1 (t) = a}. Then X(a) = B2 (τa ) is the value of B2 when B1 hits a. The process {X(a) : a ≥ 0} is, of course, a Brownian motion directed by {τa : a ≥ 0}. Show that X has stationary independent increments and that X(a) has a Cauchy distribution with density f (x) =

1 , aπ(1 + (x/a)2 )

x ∈ R.

Hint: Find the characteristic function of X(a).  1/2 Exercise 55. Consider the Bessel process R(t) = B1 (t)2 · · · Bd (t)2 as in (5.43). Show that its density is fR(t) =

2 (2t)n/2 Γ (n/2)

rd−1 e−r

2

/2t

.

Evaluate this for ! ∞Γ (α) = (α − 1)Γ (α − 1). Show √ d = 3 by using the fact that that Γ (1/2) = π by its definition Γ (α) = 0 xα−1 e−x dx and the property √ !∞ √ 2 of the normal distribution that 2 0 e−t /2 dt = π. Exercise 56. Continuation. For the Bessel process R(t) in the preceding exercise, show that R(t)2 − t is a martingale with mean 0. Exercise 57. Suppose that X(t) is a Brownian bridge. Find an expression in terms of normal distributions for pt = P {|X(1/2) − X(t)| > 0}. Is pt strictly increasing to 1 on [1/2, 1]? Exercise 58. Let X(t) denote a standard Brownian motion in R3 and let A denote the unit ball. Find the distribution of the hitting time τ = inf{t : d

X(t) ∈ Ac }. Is τ = inf{t : |B(t)| > 1}? Exercise 59. For the G/G/1 system described in Section 5.16, consider the waiting times n  (V−1 − U ). Wn = max 0≤m≤n

=m+1

Show that if ρ > 1, then n−1 Wn → E[V1 − U1 ] a.s. as n → ∞.

Chapter 6

Appendix

This appendix covers background material from probability theory and real analysis. Included is a review of elementary notation and concepts of probability as well as theorems from measure theory, which are major tools of applied probability. More details can be found in the following textbooks: Probability Theory — Billingsley 1968, Breiman 1992, Chung 1974, Durrett 2005, Feller 1972, Grimmett and Stirzaker 2001, Kallenberg 2004, Shiryaev 1995. Real Analysis — Ash and Dol´eans-Dade 2000, Bauer 1972, Hewitt and Stromberg 1965.

6.1 Probability Spaces and Random Variables The underlying frame of reference for random variables or a stochastic process is a probability space. A probability space is a triple (Ω, F , P ), where Ω is a set of outcomes, F is a family of subsets of Ω called events, and P is a probability measure defined on these events. The family F is a σ-field (or σ-algebra): If A ∈ F, then so is its complement Ac , and if a sequence An is in F, then so is its union ∪n An . The probability measure P satisfies the properties of being a measure: It is a function P : F → [0, 1] such that, for any finite or countably infinite collection of disjoint sets An in F ,  P (An ). P (∪n An ) = n

Furthermore, P satisfies P (Ω) = 1. Under this definition, P (A) ≤ 1, P (Ac ) = 1 − P (A), where Ac = Ω\A (the complement of A), and

R. Serfozo, Basics of Applied Stochastic Processes, Probability and its Applications. c Springer-Verlag Berlin Heidelberg 2009 

405

406

6 Appendix

P (A) ≤ P (B), P (An ) → P (A),

A ⊂ B, if An ↑ A or An ↓ A.

The definition of a random variable involves the notion of a measurable function. Suppose (S, S) and (S  , S  ) are measurable spaces (sets with associated σ-fields). A function f : S → S  is measurable if f −1 (A) = {x ∈ S : f (x) ∈ A} ∈ E,

for each A ∈ E  .

That is, the set of all x’s that f maps into A is in E. Typically, S will be the outcome space Ω, the real line R, the d-dimensional Euclidean space Rd , or a metric space. We adopt the standard convention that the σ-field S for S is its Borel σ-field — the smallest σ-field containing all open sets in S (or the σ-field consisting of countable unions of open sets and all complements of these). We sometimes write B and B+ for the Borel σfields of R and R+ . A useful property is that if f : S → S  and g : S  → S  are measurable, then the composition function g ◦ f (x) = g(f (x)) is measurable. The rest of this sections concerns classical real-valued random variables. We discuss random variables on metric spaces in Section 6.3. A random variable X on a probability space (Ω, F, P ) is a measurable mapping from Ω to R. The measurability of X ensures that F contains all sets of the form {X ∈ B} = {ω ∈ Ω : X(ω) ∈ B},

B ∈ B+ .

(6.1)

These are the types of events for which P is defined. One usually constructs (or assumes) the σ-field F is large enough such that the random variables of interest are measurable. For instance, if X, Y and Z are of interest, one can let F = σ(X, Y, Z), the “smallest σ-field” containing all sets of the form (6.1) for X, Y and Z, so that they are measurable. A statement about events or random variables is said to hold almost surely (a.s.) if the statement holds with probability one (some say the statement is true almost everywhere (a.e.) on Ω with respect to P ). For instance X + Y ≤ Z a.s. Also, we sometimes omit a.s. from elementary statements like X = Y and X ≤ Y that hold a.s. All of the probability information of X in “isolation” (not associated with other random quantities on the probability space) is contained in its distribution function F (x) = P {X ≤ x}, x ∈ R. Here P {X ≤ x} = P ({ω : X(ω) ≤ x}). A distribution function has at most a finite or countable number of discontinuities (which may be a dense set); this is a well-known property of any increasing function. We sometimes write the distribution as FX (x). Several standard distribution functions are in the next section. The random variable X is discrete if the range of X is a countable set S in R. In this case, the probability function of X is P {X = x}, x ∈ S; and

6.2 Table of Distributions

P {X ∈ A} =

407



P {X = x},

A ⊂ S.

x∈A

The mean (or expectation) of X is  xP {X = x}, E[X] = x∈S

provided the sum exists (it is absolutely convergent, meaning the sum of its absolute values is finite). The random variable X is !continuous if there is a (measurable) ! density ∞ function f : R → R+ such that −∞ f (x) dx = 1 and P {X ∈ A} = A f (x) dx, !x A ⊂ R. Then the distribution of X is F (x) = −∞ f (y) dy, and so f (x) = F  (x), the derivative of F . The mean (or expectation) of X is  ∞ E[X] = xf (x)dx, −∞

provided the integral exists (the integral of its absolute value is finite). There are random variables that are not discrete or continuous. Regardless of whether the random variable X is discrete, continuous or general, its mean will be denoted by  E[X] = xdF (x), R

a Riemann-Stieltjes integral defined in Section 6.4. In addition to its mean, other summary measures of a random variable X are as follows. The nth moment of X is E[X n ], and the nth moment about its mean μ = E[X] is E[(X − μ)n ]. The variance of X is Var[X] = E[(X − μ))2 ] = E[X 2 ] − μ2 . Whenever we refer to these moments, we assume they are finite.

6.2 Table of Distributions The following are tables of some standard distributions and their means, variances, and moment generating functions (which we discuss shortly).

408

6 Appendix

Discrete Random Variables Random Variable P {X = x} E[X] Var[X]

E[esX ]

Binomial n ≥ 1, p ∈ (0, 1)

n x n−x x p (1 − p) x = 0, 1, . . . , n

np

np(1 − p) (pes + (1 − p))n

Poisson λ>0

e−λ λx /n! x = 0, 1, . . .

λ

λ

e−λ(1−e

Geometric p ∈ (0, 1)

p(1 − p)x−1 x = 1, 2, . . . ,

1 p

1−p p2

pes 1−(1−p)es

r p

r(1−p) p2

pe r ( 1−(1−p)e s)

  r x−r Negative Binomial x−1 r−1 p (1 − p) r ≥ 1, p ∈ (0, 1) x = r, r + 1, . . .

−s

s

Continuous Random Variables Random Variable Density f (x) E[X] Var[X] 2

Normal μ ∈ R, σ > 0

2

λe−λx ,

Gamma∗ α ≥ 0, λ > 0

λα xα−1 e−λx , Γ (α)

Beta a, b > 0

1 b−a ,

x∈R

,

Exponential λ>0

Uniform on [a, b]

∗

/2σ e−(x−μ) √ σ 2π

x≥0

x≥0

a≤x≤b

Γ (a+b) a−1 (1 Γ (a)Γ (b) x

0≤x≤1

− x)b−1

)

E[esX ] 2 2

μ

σ2

eμs+σ

1 λ

1 λ2

λ λ−s

α λ

α λ2

λ α ( λ−s )

a+b 2

(b−a)2 12

ebs −eas s(b−a)

ab a (a+b) (a+b)2 (a+b+1)

n

n−1 −λx

s /2

...

e The gamma density with integer α = n ≥ 1 is λ x(n−1)! ; it is called an Erlang density. The gamma density with α = n/2 and λ = 1/2 is a ξ-squared density with n degrees of freedom.

6.3 Random Elements and Stochastic Processes

409

6.3 Random Elements and Stochastic Processes A unified way of discussing random vectors, stochastic processes and other random quantities is in terms of random elements. Suppose one is interested in a random element that takes values in a space S with a σ-field S. A random element in S, defined on a probability space (Ω, F, P ), is a measurable mapping X from Ω to S. The X is also called an S-valued random variable. For our purposes, the space S will be a countable set, a Euclidean space Rd , or a function space with a distance metric (for representing a stochastic process). To accommodate these and other spaces as well, we adopt the standard convention that (S, S) is a Polish space. That is, S is a metric space that is complete (each Cauchy sequence is convergent) and separable (there is a countable dense set in S); and S is the Borel σ-field generated by the open sets. A metric on S is a map d : S × S → R+ such that d(x, y) = d(y, x), d(x, y) = 0 if and only if x = y, d(x, z) ≤ d(x, y) + d(y, z), x, y, z ∈ S. Our discussion of functions, integrals, convergence, etc. on S does not require a familiarity of Polish spaces, since these concepts are understandable by interpreting them as being on Rd . We use terminology involving Polish spaces and random elements in this appendix because it allows for a rigorous and unified presentation of background material, but this terminology is not used throughout the book. The probability distribution of a random element X in S is the probability measure FX (B) = P {X ∈ B} = P ◦ X −1 (B), B ∈ S. If X and Y are random elements whose distributions are equal, we say that d X is equal in distribution to Y and denote this by X = Y . The underlying probability spaces for X and Y need not be the same. Loosely speaking, a stochastic process is a collection of random variables (or random elements) defined on a single probability space. Hereafter, we will simply use the term “random elements” (which includes random variables), and let (S, S) denote the Polish space where they reside. A discrete-time stochastic process (or random sequence) is a collection of random elements X = {Xn : n ≥ 0} in S defined on a probability space (Ω, F, P ). The nonnegative integer n is a time parameter and S is the state space of the process. The value Xn (ω) ∈ S is the state of the process at time n associated with the outcome ω. Note that X is also a random element in the infinite product space S ∞ with the product σ-field S ∞ : the smallest σ-field generated by sets B1 × · · · × Bn , Bj ’s ∈ S. Its distribution P {X ∈ B}, for B ∈ S ∞ , is uniquely defined in terms of its finite-dimensional distributions

410

6 Appendix

P {X1 ∈ B1 , . . . , Xn ∈ Bn },

Bj ∈ S, n ≥ 1.

Consequently, if Y is another random element in S ∞ whose finite-dimensional d distributions are equal to those of X, then X = Y . We sometimes refer to the process X = {Xn : n ≥ 0} by Xn . Stochastic processes in continuous time are defined similarly to those in discrete time, but their evolutions over time are typically more complicated. A continuous-time stochastic process is a collection of random elements {X(t) : t ≥ 0} in S defined on a probability space, where X(t, ω) is the state at time t associated with the outcome ω. The function t → X(t, ω) from R+ to S, for a fixed ω, is the sample path or trajectory associated with the outcome ω. Accordingly, X(t) is a random function from R+ to S. More precisely, the entire process X = {X(t) : t ≥ 0} is a random element in a space of functions from R+ to S. We sometimes refer to the process by X(t). A standard example is when the sample paths of X are in the set D(R+ ) of functions from R+ to S that are right-continuous with left hand limits — often called cadlag functions (from the French continu ` a droite, limites a ` gauche). Then X is a random element in D(R+ ), with an appropriate metric depending on one’s application (e.g., a uniform metric or a metric for the Skorohod topology [11, 60, 113]), and P {X ∈ B} is for a Borel set B ⊂ D(R+ ) of sample paths. The distribution of X is uniquely determined by its finitedimensional distributions P {X(t1 ) ∈ B1 , . . . , X(tn ) ∈ Bn },

t1 < · · · < tn , Bj ∈ S, n ≥ 1.

In summary, a stochastic process is a family of random variables or random elements defined on a probability space that contains all the probability information about the process. We will use the standard convention of suppressing the ω in random elements such as Xn or X(t), and not displaying the underlying probability space (Ω, F, P ), unless it is essential for the exposition. Also, all the functions appearing in this book are measurable, and we will mention this property only when it is needed.

6.4 Expectations as Integrals We defined the mean of discrete and continuous random variables above. These are special cases of the following definition for any real-valued random variable. Definition 1. Let X be a random variable with distribution function F (x) = P {X ≤ x}. The mean (or expectation) of X is defined by the RiemannStieltjes integral

6.4 Expectations as Integrals

411

 E[X] =

xdF (x), R

provided the integral exists. This general definition is needed for random variables that are not discrete or continuous. For instance, if X has positive probabilities at points in a countable set S, and also has a sub-density f (x) elsewhere on R, then   xP {X = x} + xf (x) dx. E[X] = R

x∈S

Riemann-Stieltjes integrals are similar to Riemann integrals in calculus. A Riemann-Stieltjes integral of a function g : [a, b] → R with respect to F is constructed by limits of the upper and lower Darboux sums Dχ and Dχ defined on the set of points χ = {a = x0 < x1 < . . . < xn = b} by D = χ

n 

sup{g(x) : xj−1 ≤ x ≤ xj }[F (xj ) − F (xj−1 )],

j=1

and Dχ is defined similarly with sup replaced by inf. The Riemann-Stieltjes integral of g exists if, for any ε > 0, there is a set χ depending on g and ε such that Dχ − Dχ < ε. When it exists, the integral has the form (e.g., see [53])  [a,b]

g(x)dF (x) = inf Dχ = sup Dχ . χ

χ

!

This integral is a Riemann integral [a,b] g(x)f (x)dx (as in calculus), when dF (x) = f (x)dx and dx is the Lebesgue measure on R. Riemann-Stieltjes integrals on infinite intervals are defined similar to Riemann integrals. For instance, for g : R → R,   g(x)dF (x) = lim g(x)dF (x), a,b→∞

R

[−a,b]

provided the limit exists and is finite. For a function g : S → R on a metric space S, its integral with respect to a measure μ on S is defined as a Lebesgue integral [53] denoted by  g(x)μ(dx). S

We denote the differential by μ(dx) instead of dμ(x) to emphasize that it is not a Riemann-Stieltjes integral. Another equivalent expression for the expectation of X in terms of the probability P is the Lebesgue integral

412

6 Appendix

 E[X] =

X(ω)P (dω). Ω

A few properties of the expectation operator are E[a] = a, for a ∈ R, E[X + Y ] = E[X] + E[Y ], E[

n 

aj X j ] =

j=1

E[X] ≤ E[Y ] if X ≤ Y , n 

aj E[Xj ].

j=1

6.5 Functions of Stochastic Processes Many features of a stochastic process, or related quantities of interest, are expressed as functions of the process. This section contains several examples and a formula for evaluating expectations of real-valued functions of random elements and processes. Suppose that X is a random element in a Polish space S, such as a discreteor continuous-time process X = {Xn : n ≥ 0} or X = {X(t) : t ≥ 0}, and denote its distribution by FX (B) = P {X ∈ B}, B ∈ S. Consider a measurable function g : S → S  and define Y = g(X). This Y is a random element in S  since it is a composition of X and g, which are measurable. When X is a continuous-time process, g(x) is a function on the space of sample paths x = {x(t) : t ≥ 0}. The distribution of Y is P {Y ∈ B} = P {g(X) ∈ B} = P {X ∈ g −1 (B)}, where g −1 (B) = {x ∈ S : g(x) ∈ B}. Then the distribution of Y as a function of FX is the probability measure FY (B) = FX ◦ g −1 (B) = FX (g −1 (B)),

B ∈ S.

In some cases, the function g is a standard measurable operation on real numbers such as addition, subtraction, maximum, etc. For instance, if X = {Xn : n ≥ 0} is a family of random variables, then Y = X1 + · · · + Xn is a random variable for fixed n < ∞, since the addition function g(x) = x1 + · · · + xn from R∞ to R is measurable. Other standard examples of random variables that are measurable functions of X include n  j=1

Xj ,

max Xj ,

1≤j≤n

n 

an (Xn − Xj ).

j=1

Examples based on multiple compositions of functions are

6.5 Functions of Stochastic Processes

413

sup Xn − inf Xn ,

n

  sup e−aXn (Yj − Yj−1 ) ,

n≥0

n≥0

n≥0

j=1

provided they exist. Examples of g(X) for a continuous-time process X include analogues of those above as well as  t  X(t − s) ds, e−a(t) inf X(s) dt. s≤t

R+

0

In modeling a stochastic system, the state of the system, or a performance measure for it, are often of the form Y (t) = g(t, X), where the process X represents the time-dependent system data, and the function g(t, x) represents the dynamics of the system. We will now describe a useful formula for the mean of a real-valued function Y = g(X) of the random element X. The mean of Y in terms of the distribution FX of X is the Lebesgue integral  g(x)FX (dx), (6.2) E[g(X)] = S

provided it exists. This follows since FY = FX ◦ g −1 and, by the change-ofvariable formula below, we have   yFX ◦ g −1 (dy) = g(x)FX (dx). E[Y ] = R

S

Change-of-variable Formula for Lebesgue integrals. Suppose F is a measure on S, and g : S → S  and h : S  → R are measurable. Then   h ◦ g(x)F (dx) = h(y)F ◦ g −1 (dy), (6.3) S

S

provided both integrals exist (one exists if and only if the other one does). Important functions of random variables are generating functions and transforms. They are tools for characterizing distributions and evaluating their moments. The moment generating function of a random variable X is  mX (s) = E[esX ] = esx dFX (x), R

provided the integral exists for s in some interval [0, a], where a > 0. A major property is that a moment generating function uniquely determines a distribution and vice versa (mX = mY if and only if FX = FY ). Also, the nth moment of X, when it exists, has the representation (n)

E[X n ] = mX (0),

414

6 Appendix

which is the nth derivative of mX at 0. Moment generating functions of some standard distributions are given in Section 6.2 below. Two variations of moment generating functions for special types of random variables are as follows. For a “nonnegative” random variable X, its Laplace transform (or the Laplace-Stieltjes transform of FX ) is  E[e−sX ] = e−sx dFX (x), s ≥ 0. R+

For a “discrete” random variable X whose range is contained in the nonnegative integers, its generating function is E[sX ] =

∞ 

sn P {X = n},

−1 ≤ s ≤ 1.

n=0

Laplace transforms and generating functions play the same role as moment generating functions in that they uniquely determine distribution functions and yield moments of random variables. Laplace transforms are also defined for increasing functions F that need not be bounded, such as renewal functions, and they can also be extended to the complex plane. A similar statement applies to generating functions. A generalization of a moment generating function is a characteristic function. The characteristic function of a random variable X (or the FourierStieltjes transform of FX ) is defined by  E[eisX ] = eisx dFX (x) s ∈ R, R

√ where i = −1 and eisx = cos sx + i sin sx. A characteristic function, which is complex-valued, exists for “any” random variable (or distribution function). In contrast, a moment generating function is real-valued, but it only exists for a random variable whose moments exist. There is a one-to-one correspondence between distribution functions and characteristic functions, and moments are expressible by derivatives of characteristic functions at 0. A characteristic function is typically used when the more elementary moment generating function, Laplace transform, or generating function are not applicable. The following are useful inequalities involving expectations of functions of random variables. For a random variable X and increasing g : R → R+ , P {X ≥ x} ≤ E[g(X)]/g(x),

Markov’s Inequality.

An example is P {|X − E[X]| ≥ x} ≤ Var[X]/x2 ,

Chebyshev’s Inequality.

6.6 Independence

415

For random variables X, Y with finite second moments, $ E|XY | ≤ E[X 2 ]E[Y 2 ], Cauchy-Buniakovsky-Schwarz. For random variables X1 , . . . , Xn and convex g : Rn → R, E[g(X1 , . . . , Xn )] ≥ g(E[X1 ], . . . , E[Xn ]),

Jensen’s Inequality.

6.6 Independence In this section, we define independent random variables and random elements, and describe several functions of them including summations. Random variables X1 , . . . , Xn are independent if, for Borel sets B1 , . . . , Bn in R, P {X1 ∈ B1 , . . . , Xn ∈ Bn } =

n  j=1

P {Xj ∈ Bj } =

n 

FXj (Bj ).

j=1

An infinite family of random variables are independent if any finite collection of the random variables are independent. The same definitions apply to independence of random elements such as the independence of stochastic processes. Many properties of functions of independent random elements can be analyzed in terms of their distributions. Here is an important formula for expectations. Suppose X and Y are independent random elements in S and S  , respectively, and g : S × S  → R is measurable. Then by (6.2) and Fubini’s theorem stated below, we have  g(x, y)FX (dx)FY (dy) E[g(X, Y )] = S×S     = g(x, y)FX (dx) FY (dy), (6.4) S

S

provided the integral exists. For the next result, we use the notion that a measure space (S, S, μ) is σ-finite if there is a partition B1 , B2 , . . . of S such that μ(Bn ) < ∞, for each n. A Polish space has this property. Theorem 2. (Fubini) Suppose μ and η are measures on the respective σfinite spaces (S,!S) and (S  , S  ), and g : S × S  → R is measurable. If g is nonnegative or S×S  |g(x, y)|μ(dx)η(dy) is finite, then

416

6 Appendix





 g(x, y)μ(dx)η(dy) =

S×S 



S  S =

 g(x, y)μ(dx) η(dy)  g(x, y)η(dy) μ(dx).

S

S

This says that if the integral exists on the product space, then it equals the single-space integrals done separately (in either order). A special case of (6.4) is   E[g(X)h(Y )] = g(x)FX (dx) h(y)FY (dy) = E[g(X)]E[h(Y )]. S

S

This generalizes, for independent X1 , . . . , Xn in S and gj : S → R, to E[

n 

gj (Xj )] =

j=1

n 

E[gj (Xj )].

j=1

provided the expectations exist. random variExample 3. Suppose X1 , . . . , Xn are independent nonnegative n ables. Then the moment generating function of Y = j=1 Xj is E[esY ] =

n 

E[esXj ].

j=1

Now, assume each Xj has an exponential distribution with rate λ, and so E[esXj ] = λ/(λ − s), 0 ≤ s < λ. Consequently, E[esY ] = [λ/(λ − s)]n , which is the moment generating function of a gamma (or Erlang) distribution with parameters λ and n (see Section 6.2). Hence Y has this gamma distribution. One can also determine distributions of sums of independent random variables by convolutions of their distributions. Specifically, if X and Y are independent random variables, then  P {X + Y ≤ z} = FY (z − x)dFX (x). (6.5) R

That is, FX+Y (z) = FX FY (z), where is the convolution operator defined below. To prove (6.5), first note that by (6.2) (even for dependent X and Y ), we have  FX,Y (dx × dy). P {X + Y ≤ z} = E[1(X + Y ≤ z)] = x+y≤z

Then applying FX,Y (dx × dy) = dFX (x)dFY (y) (from the independence) and Fubini’s theorem to this double integral yields (6.5). Properties of convolutions are as follows. The convolution of two distributions F and G is defined by

6.7 Conditional Probabilities and Expectations

417

 F G(z) = R

G(z − x)dF (x).

(6.6)

Note that F G = G F . Also, if F (0−) = G(0−) = 0, then  F G(z) = G(z − x)dF (x). (0,z]

If F and G have respective densities f and g, then the derivative of (6.6) yields the formula  f g(z) = g(z − x)f (x)dx. R

These formulas reduce to sums when F and G are discrete distributions. Convolutions of several distributions are defined in the obvious way, for instance F G H = F (G H), and if X, Y and Z are independent random variables, then FX+Y +Z = FX FY FZ . The nth fold convolutions F n (x) of a distribution F , for n ≥ 0, are defined recursively by F 0 (x) = 1(x ≥ 0) and, for n ≥ 1, F n (x) = F (n−1) F (x) = F · · · F (x),

n convolutions.

If Tn = X1 + · · · + Xn where the Xj are independent with distribution F , then FTn = F n . The definition (6.6) of a convolution also extends to more general functions μ h, where μ is a measure on R and h : R → R is such that the integral exists. For example, renewal theory involves convolutions of the form  U h(t) = h(t − s)dU (s), [0,t]

∞

where U (t) = n=0 F n (t), F (0) = 0 and h(t) is bounded on compact sets and is 0 for t < 0.

6.7 Conditional Probabilities and Expectations Section 1.22 in Chapter 1 reviewed fundamentals of conditional probabilities and expectations for discrete random variables. This section continues the review for random elements as well as non-discrete random variables. Suppose X and Y , defined on a common probability space (Ω, F , P ), are random elements in Polish spaces S and S  , respectively. A probability kernel from S  to S is a function p : S  × S → [0, 1] such that p(y, ·) is a probability measure on S for each y ∈ S  , and p(·, B) is a measurable function for each B ∈ S. There exists a probability kernel p(y, B) from S  to S such that

418

6 Appendix

 P {X ∈ B} =

S

B ∈ S.

p(y, B)FY (dy),

(6.7)

The kernel p is unique in the sense that if p is another such kernel, then p(Y, ·) = p (Y, ·) a.s. (e.g., see [60]). Definition 4. Using the preceding notation, the (random) probability measure P {X ∈ B|Y } = p(Y, B), B∈S is the conditional probability measure of X given Y . When X is a random variable with a finite mean, the conditional expectation of X given Y is  xp(Y, dx). (6.8) E[X|Y ] = R

Conditional probabilities and expectations — which are random quantities — are sometimes written as non-random quantities  P {X ∈ B|Y = y} = p(y, B), E[X|Y = y] = xp(y, dx), y ∈ S . R

An important property of conditional expectations, which follows from the definition, is  E[X|Y = y]FY (dy). E[X] = E[E[X|Y ]] = S

Similarly, P {X ∈ B} = E[P {X ∈ B|Y }]. These formulas are useful tools for evaluating the mean or distribution of X in terms of the conditional means or distributions. Another important property is that E[X|Y ] is a measurable function of ! Y , because E[X|Y ] = h(Y ), where h(y) = E[g(X, y)] = S xp(y, dx) is measurable. Since Definition 4 is for random elements, Y may represent several random elements. For instance E[X|Y, Z] is a measurable function of Y, Z and  

 E[X|Z] = E E[X|Y, Z]Z . Definition 4 is consistent with the definition used for elementary random variables. For instance, in case Y is a discrete random variable, the probability kernel that satisfies (6.7) is p(y, B) = P {X ∈ B|Y = y} = P {X ∈ B, Y = y}/P {Y = y}. Next suppose X and Y are continuous random variables such that  f (x, y) dx dy, P {X ∈ A, Y ∈ B} = A×B

6.8 Existence of Stochastic Processes

419

where f (x, y) is their joint density. Then the probability kernel satisfying (6.7) is  f (x, y)/f (y) dx. p(y, B) = B

The preceding notions extend to conditional probabilities P {X ∈ B|F } and expectations E[X|F] for conditioning on a σ-field F instead of a random element. Definition 4 includes these cases when F = σ(Y ), the smallest σfield generated by Y . We will only use this general notation only in Chapter 5. Here are some more properties of conditional expectations (assuming they exist) for measurable f : S  → R and g : S × S  → R: E[Xf (Y )|Y ] = f (Y )E[X|Y ], E[g(X, Y )|Y = y] = E[g(X, y)|Y = y]. Furthermore, if X and Y are independent, then E[g(X, Y )|Y = y] = E[g(X, y)], or equivalently E[g(X, Y )|Y ] = E[h(Y )], where h(y) = E[g(X, y)]. Most of the standard properties of expectations extend to conditional expectations. For instance, P {X ∈ B|Y } = E[1(X ∈ B)|Y ], E[X|Y ] ≤ E[Z|Y ]

if X ≤ Z,

E[f (X) + g(Z)|Y ] = E[f (X)|Y ] + E[g(Z)|Y ].

6.8 Existence of Stochastic Processes A stochastic process is commonly defined by designating a set of properties that its distribution and sample paths must satisfy. The existence of the process amounts to showing that there exist a probability space and functions on it (the sample paths) that satisfy the designated properties of the process. This section describes Kolmogorov’s theorem that is used for such a task. For a stochastic process {X(t) : t ∈ R+ } on a Polish space S, we mentioned that any probability for it can be expressed in terms of its finite-dimensional probability measures μI on S I . For any finite set I in R+ , and At ∈ S, t ∈ I, μI (×t∈I At ) = P {X(t) ∈ At , t ∈ I}.

(6.9)

Here S is the Borel σ-field on S, and the product σ-field SI is used on S I . These probability measures satisfy the consistency condition μJ (· × S J\I ) = μI (·),

I ⊂ J.

(6.10)

420

6 Appendix

That is, for At ∈ S, t ∈ J, μJ (×t∈J At ) = μI (×t∈I At ),

if At = S for t ∈ J\I.

Theorem 5. (Kolmogorov Extension Theorem) For probability measures μI that satisfy the consistency condition (6.10), there exists a stochastic process {X(t) : t ∈ R+ } on S that is defined on a probability space (Ω, F, P ) such that μI are its finite-dimensional distributions as in (6.9). The proof of this result in [60] defines (Ω, F ) = (S R+ , SR+ ), and defines P by

c

P ( · × S J ) = μJ (·). Here μJ are probability measures for countable sets J that are extensions, defined by (6.10), of the probability measures μI for finite sets I. Also, for each t, the function X(t) from Ω to S is defined as the projection map X(t) = X(t, ω) = ωt ,

ω = {ωt : t ∈ R+ } ∈ Ω.

The preceding theorem also applies to processes in which the time parameter t takes values in other time sets such as R or the nonnegative integers. Here is an important example. Corollary 6. (Existence of Independent Random Elements) For probability measures μ1 , μ2 , . . . on a Polish space S, there exist independent S-valued random elements X1 , X2 , . . . defined on a probability space (Ω, F , P ) such that P {Xj ∈ ·} = μj (·), j ≥ 1. d

Recall that processes X and X  are equal in distribution, denoted by X =  X , if their finite-dimensional probability measures are equal. However, their sample paths need not be the same. For instance, suppose X(t) = 0,

X  (t) = 1(ξ = t),

t ∈ R+ , d

where ξ is exponentially distributed with rate λ. Clearly X = X  since P {ξ = t} = 0 for any t. On the other hand, their sample paths are not the same: P {X(t) = 0, t ∈ R+ } = 1, but P {X  (t) = 0, t ∈ R+ } = 0. Although the consistency condition on finite-dimensional probability measures μI guarantees the existence of a stochastic process, additional conditions on the μI are needed to infer that the sample paths of the process have certain properties (e.g., that the paths are continuous, or right-continuous).

6.9 Convergence Concepts

421

6.9 Convergence Concepts Many properties of stochastic processes are expressed in terms of convergence of random variables and elements. There are several types of convergence, but for our purposes we primarily use convergence with probability one and convergence in distribution, which we now describe. We begin with a review of convergence of real numbers. A sequence of real numbers xn converges to some x ∈ R, denoted by limn→∞ xn = x, if, for any ε > 0, there exists a number N such that |xn − x| < ε,

n ≥ N.

We sometimes refer to this convergence as xn → x. One often establishes convergence with the quantities lim inf xn = lim inf xk , n→∞

lim sup xn = lim sup xk .

n→∞ k≥n

n→∞ k≥n

n→∞

(6.11)

This limit inferior and limit superior clearly satisfy −∞ ≤ lim inf xn ≤ lim sup xn ≤ ∞. n→∞

n→∞

If both of these quantities are equal to a finite x, then xn → x. For insight into this result, let a and b denote the lim inf and lim sup in (6.11) and assume they are finite. By its definition, a is the lower “cluster value” of the xn ’s in that xn is in the interval [a, a + ε] infinitely often (i.o.), for fixed ε > 0. Similarly, xn is in the interval [b − ε, b] i.o. Now xn does not converge to a limit if a < b (since xn is arbitrarily close to both a and b i.o.). On the other hand, xn → x if and only if a = b = x. The preceding properties of real numbers readily extend to random variables. Suppose X, X1 , X2 , . . . are random variables on a probability space. The sequence Xn converges with probability one to X if lim Xn (ω) = X(ω),

n→∞

ω ∈ Ω  ⊂ Ω,

where P (Ω  ) = 1. We denote this convergence by Xn → X,

a.s.

as n → ∞.

Now, the quantities lim inf Xn , n→∞

lim sup Xn n→∞

are random variables, since the functions lim inf n→∞ xn and lim supn→∞ xn are measurable. If there is a random variable X such that

422

6 Appendix

lim inf Xn = lim sup Xn = X n→∞

a.s.,

n→∞

then Xn → X, a.s. as n → ∞. This follows by the analogous property for a sequence of real numbers. Next, we consider the convergence in probability as well as convergence a.s. of random elements. Definition 7. Let X, X1 , X2 , . . . be random elements in a metric space S, where d(x, y) denotes the metric distance between x and y. The sequence Xn converges a.s. to X in S, denoted by Xn → X a.s., if d(Xn , X) → 0 a.s. The P Xn converges in probability to X, denoted by Xn → X, if lim P {d(Xn , X) > ε} = 0,

n→∞

ε > 0.

Many applications involve establishing the convergence of a function f (Xn ), when Xn converges. A useful tool for this is the following. Proposition 8. (Continuous-mapping Property) Suppose X, X1 , X2 , . . . are random elements in a metric space S, and f : S → S  where S  is a metric space. Assume f is continuous on S, or on a set B such that X ∈ B a.s. If Xn → X a.s. in S, then f (Xn ) → f (X) a.s. The same statement is true for convergence in probability. For instance, if (Xn , Yn ) → (X, Y ) a.s. in R2 , then Xn + Yn → X + Y,

Xn Yn → XY

a.s. in R .

This follows since the addition and multiplication functions from R2 to R are continuous. Similarly, Xn /Yn → X/Y if Y = 0 a.s. These statements also hold for convergence in probability. Another important mode of convergence of random variables and random elements is convergence in distribution or weak convergence. A sequence of w distributions Fn on R converges weakly to a distribution F , denoted by Fn → F , if lim Fn (x) = F (x), n→∞

for all continuity points x of F . A sequence of random variables Xn converges w d in distribution to a random variable X, denoted by Xn → X, if FXn → FX , as n → ∞. This notion extends to metric spaces as follows. Definition 9. A sequence of probability measures Pn on a metric space S w converge weakly to a probability measure P , denoted by1 Pn → P , if   f (x)Pn (dx) → f (x)P (dx), S 1

S w

d

Some authors use Pn ⇒ P instead of Pn → P , and Xn ⇒ X instead of Xn → X.

6.9 Convergence Concepts

423

for any bounded continuous f : S → R. Suppose that X and Xn are random elements in S, possibly defined on different probability spaces. The Xn w d converges in distribution to X, denoted by Xn → X, if FXn → FX , or equivalently, lim E[f (Xn )] = E[f (X)], n→∞

for any bounded continuous f : S → R. This definition is for random elements in the same space S; we do not need the more general convergence for random elements in different spaces. The preceding modes of convergence for random elements in a metric space S have the hierarchy Xn → X a.s.

=⇒

P

Xn → X

=⇒

d

Xn → X.

Here are several characterizations of convergence in distribution. Theorem 10. (Portmanteau theorem) For random elements X, X1 , X2 , . . . in a metric space S, the following statements are equivalent. d (i) Xn → X. (ii) lim inf n→∞ P {Xn ∈ G} ≥ P {X ∈ G}, for any open set G. (iii) lim supn→∞ P {Xn ∈ F } ≤ P {X ∈ F }, for any closed set F. (iv) P {Xn ∈ B} → P {X ∈ B}, for any Borel set B with X = ∂B a.s.2 The convergence in distribution of random variables is equivalent to the d convergence of their characteristic functions. Specifically, Xn → X in R if isXn isX and only if E[e ] → E[e ], s ∈ R. Similar statements hold for moment generating functions, Laplace transforms or generating functions. The continuous-mapping property in Proposition 8 extends to convergence in distribution as follows. Theorem 11. (Continuous Mappings; Mann and Wald, Prohorov, Rubin) d

Suppose Xn → X in a metric space S, and B ∈ S is such that X ∈ B a.s. Let f, f1 , f2 , . . . be measurable functions from S to a metric space S  . d (a) If f is continuous on B, then f (Xn ) → f (X). d (b) If fn (xn ) → f (x), for any xn → x ∈ B, then fn (Xn ) → f (X). d

As an example, for random variables Xn and Yn , suppose (Xn , Yn ) → (X, Y ) as n → ∞. Then d

Xn + Yn → X + Y,

d

Xn Yn → XY.

This follows by the continuous-mapping theorem since addition and multiplication are continuous functions from R2 to R. Caution: Such results may not be true, however, when Xn and Yn are real-valued stochastic processes. 2

∂B denotes the boundary of B.

424

6 Appendix

The next results address the following question for random variables Xn . If Xn → X a.s. (or in probability or distribution), what are the additional conditions under which E[Xn ] → E[X] or E|Xn − X| → 0? Lemma 12. (Fatou) If Xn are nonnegative random variables (or are bounded from below), then lim inf n→∞ E[Xn ] ≥ E[lim inf n→∞ Xn ]. Theorem 13. (Monotone Convergence) If Xn are nonnegative random variables (or are bounded from below) and Xn ↑ X a.s., then E[Xn ] ↑ E[X] as n → ∞, where E[X] = ∞ is possible. Theorem 14. (Dominated Convergence) If Xn are random variables such that Xn → X a.s., where |Xn | ≤ Y and E[Y ] < ∞, then E|X| exists and E[Xn ] → E[X] as n → ∞. ! These results describing the convergence of E[Xn ] = Ω Xn (ω)P (dω), are random-variable versions of basic theorems for Lebesgue integrals (and for summations as well). We will use the results a few times for real-valued functions f, fn on a measurable space (S, S) with a measure μ. For instance, the monotone convergence says that if 0 ≤ fn ↑ f , then   fn (x)μ(dx) → f (x)μ(dx). S

S

The next results address the convergence of E[Xn ] when Xn converges in probability. P

Theorem 15. (Convergence in mean or in L1 ) Suppose Xn → X in R, and E|Xn | and E|X| are finite. Then the following statements are equivalent. (Xn converges to X in L1 ). (i) E|Xn − X| → 0 (ii) E|Xn | → E|X|. (iii) The Xn are uniformly integrable in the sense that

 lim sup E |Xn |1(|Xn | ≥ x) → 0. x→∞ n≥0

Some convergence theorems such as the preceding result and the dominated convergence theorem are also true when the assumption that Xn → X d in probability or a.s. is replaced by the weaker assumption Xn → X. This is due to the following a.s. representation for convergence in distribution of random elements. d

Theorem 16. (Coupling; Skorohod, Dudley) Suppose Xn → X in a Polish space S. Then there exist random elements Yn and Y in S, defined on a single d d probability space, such that Yn = Xn , Y = X, and Yn → Y a.s. Loosely speaking, coupling is a method for comparing random elements on different probability spaces, usually to prove convergence theorems or

6.9 Convergence Concepts

425

stochastic ordering properties. For instance, suppose Xn is a random element in Sn defined on a probability space (Ωn , Fn , Pn ), for n ≥ 0. Random elements Yn in Sn defined on a single probability space (Ω, F, P ) form a coupling of d

Xn if Yn = Xn , n ≥ 0. Theorem 16 and the classical dominated convergence yield the following dominated convergence for convergence in distribution. d

Theorem 17. Suppose Xn → X in R and there exists a random variable Y with finite mean such that |Xn | ≤d Y , meaning P {|Xn | > x} ≤ P {Y > x}, Then E|X| exists and E[Xn ] → E[X] as n → ∞.

x ≥ 0.

Bibliographical Notes

The history of theoretical stochastic processes can be found in books on theoretical probability or stochastic processes. Extensive notes on this are in [46, 61], and miscellaneous references are in the miscellaneous books [10, 18, 26, 37, 38, 41, 42, 62, 63, 84, 96, 102, 106, 114]. The history of applied stochastic processes parallels that of theoretical work. The developments in applied stochastic processes stem from advances in theoretical probability as well as from problems that cry out for solutions. Standard texts on applied stochastic processes are [10, 27, 42, 46, 62, 63, 76, 78, 93, 94, 98], and those with a more specialized focus are [2, 54, 59, 109, 116]. Much of the material on actual applications has been presented in technical reports or in specialized journals that is not conducive to a unified review. So I will only comment on some of the main themes and representative references while discussing the chapters. Most of Chapter 1 on Markov chains is standard. Exceptions are the reflected random walk (a framework for several models), which is a discretetime version of the Skorohod equation for reflected Brownian motion [64], and the network models are discrete-time versions of the continuous-time Jackson network models in Chapter 4. Further background and more intricate branching process models are discussed in [49, 58, 62, 63]. References on general Markov processes are the early work [75] and the more current books [38, 37, 41, 42, 61, 84, 96, 106]. Phase-type distributions and computational algorithms, which were not covered, are discussed in [85]. Chapter 2 on Renewal and Regenerative Processes is a fairly thorough coverage of the literature in the 1950s and 1960s, including the works [3, 42, 81, 107] and ending with [69] (only a few articles have appeared after this one). Regenerative processes were popularized with the work of Smith [107]; many applications continue to be based on analyzing processes at embedded times that may or may not be regeneration times. The examples here and in other chapters on reliability and maintenance, production systems, and insurance are indicative of those fields; see for instance [4, 5, 8] and [22, 48, 94].

427

428

Bibliographical Notes

The richness of the applications of Poisson processes led me to devote an entire chapter to it. Chapter 3 covers classical Poisson processes, but goes further to show the variety of Poisson processes in space and time and their transformations that yield tractable models for systems. Standard references on point processes are the early work of Poisson [87] and later works are [16, 30, 60, 66]; further examples are in [6, 40, 44, 73, 101, 103]. The section on batch-service queueing systems describes a classic Markov decision model [33] (or stochastic dynamic programming model). Dynamic programming is described in [77, 90, 109]. The Markov/Poisson particle model is an elementary example of independent particles [34]; interacting particle systems are discussed in [80]. The Grigelionis theorem [45] showing that Poisson processes are natural limits for sparse point processes, is analogous to the central limit theory for summation processes converging to Brownian motion covered in Chapter 5. Queueing processes, input-output systems, and stochastic networks have been a main part of applied stochastic processes. This is reflected in Chapter 4 on Continuous-Time Markov Chains. In addition to covering the basics of CTMCs, the chapter provides numerous queueing and network models. Sample references on early work on queueing processes are [20, 28, 40, 44, 57, 68, 70, 71, 72, 82, 91], books on queueing are [2, 6, 12, 13, 15, 28, 29, 39, 47, 59, 74, 88, 108, 116], and works on stochastic networks are [7, 9, 23, 25, 67, 92, 95, 99, 101, 110, 111]. Palm probabilities, that began with Palm [86], are introduced to describe certain PASTA (Poisson actions see time averages) properties [116] of queues and other processes. The final chapter on Brownian motion covers many of its properties, without getting into more advanced stochastic integration. Books that discuss Brownian motion include [10, 18, 46, 61, 64]. Several key works on Donsker’s functional central limit theorem [35, 36] and based on ideas of Prohorov and Skorohod, followed by Billingsley, are [11, 35, 36, 104, 105]. Major applications in queueing and elsewhere are nicely reviewed in Whitt [55, 113]; also see the early work of Kingman 1961 and Iglehart and Whitt 1970 on heavy-traffic queues. During the last three decades, there has been a substantial stream of articles on heavy-traffic systems, Brownian approximations and fluid queues. Much of this research has been done by Harrison and his colleagues Bramson, Chen, Dai, Mandelbaum, Reiman, and Williams [14, 16, 24, 25, 30, 31, 50, 51, 52, 77, 83, 92, 115].

References

1. Ash, R. B. and C. A. Dol´eans-Dade (2000). Probability and Measure Theory, 2nd Ed. Academic Press, San Diego. 2. Asmussen, S. (2003). Applied Probability and Queues. Springer, New York. 3. Athreya, K., McDonald, D. and P. Ney (1972). Coupling and the renewal theorem. Amer. Math. Monthly, 85, 809–814. 4. Arjas, E. and I. Norros (1989). Change of life distribution via hazard transformation: An inequality with application to minimal repair. Math Oper. Res, 14, 355–361. 5. Aven, R. and U. Jensen (1999). Mathematical Theory of Reliability. Springer, New York. 6. Baccelli, F. and P. Br´emaud (1994). Elements of Queueing Theory. Springer, New York. 7. Baccelli, F. and S. Foss (1994). Ergodicity of Jackson-type queueing networks. Queueing Systems, 17, 5–72. 8. Barlow, R. and F. Proschan (1995). Mathematical Theory of Reliability. John Wiley & Sons, New York. 9. Baskett, F., Chandy, K. M., Muntz, R. R. and F. G. Palacios (1975). Open, closed and mixed networks of queues with different classes of customers. J. Assoc. Comput. Mach., 22, 248–260. 10. Bhattacharya, R. N. and E. C. Waymire (1990). Stochastic Processes with Applications. John Wiley & Sons, New York. 11. Billingsley, P. (1968). Convergence of Probability Measures. John Wiley & Sons, New York. 12. Borovkov, A. A. (1976). Stochastic Processes in Queueing theory. Springer, New York. 13. Borovkov, A. A. (1984). Asymptotic Methods in Queueing theory. John Wiley & Sons, New York. 14. Boxma, O. J. and J. W. Cohen (1999). Heavy-traffic analysis for the GI/G/1 queue with heavy-tailed distributions. Queueing Systems, 33, 177–204. 15. Brandt, A., Franken, P. and B. Lisek (1990). Stationary Stochastic Models. John Wiley & Sons, New York. 16. Bramson, M. (1998). State space collapse with application to heavy traffic limits for multiclass queueing networks. Queueing Systems, 30, 89–148. 17. Brandt, A. and G. Last (1995). Marked Point Processes on the Real Line: The Dynamic Approach. Springer, New York. 18. Breiman, L. (1968). Probability. Addison-Wesley, Reading, Mass. 19. Br´ emaud, P. (1999). Markov Chains. Gibbs Fields, Monte Carlo Simulation, and Queues. Springer, New York. 20. Burke, P. J. (1956). The output of a queuing system. Operations Res., 4, 699–704.

429

430

References

21. Burman, D. Y., Lehoczky, J. P. and Y. Lim (1984). Insensitivity of blocking probabilities in a circuit-switched network. J. Appl. Prob., 21, 853–859. 22. Buzzacott, J. A. and J. G. Shanthikumar (1993). Stochastic Models of Manufacturing Systems. Prentice–Hall, Englewood Cliffs, New Jersey. 23. Chao, X., Pinedo, M. and M. Miyazawa (1999). Queueing Networks: Negative Customers, Signals and Product Form. John Wiley & Sons, New York. 24. Chen, H. and A. Mandelbaum (1994). Heirarchical modeling of stochastic networks, part II: strong approximations. Stochastic Modeling and Analysis of Manufacturing Systems. D. D. Yao ed. Springer, New York, 107–131. 25. Chen, H. and D. D. Yao (2001). Fundamentals of Queueing Networks: Performance, Asymptotics and Optimzation. Springer, New York. 26. Chung, K. L. (1974). A Course in Probability Theory, 2nd ed. Academic Press, New York. 27. C ¸ inlar, E. (1975). Introduction to Stochastic Processes. Prentice–Hall, Englewood Cliffs, New Jersey. 28. Cohen, J. W. (1982). The Single Server Queue. revised ed. North-Holland, Amsterdam. 29. Cooper, R. B. (1982). Introduction to Queueing Theory. 2nd ed. North-Holland, Amsterdam. 30. Dai, J. G. (1994). On positive recurrence of multiclass queueing networks: a unified approach via fluid limit models. Ann. Applied Prob., 4, 49–77. 31. Dai, J. G. and J. M. Harrison (1991). Steady-state analysis of RBM in a rectangle: numerical methods and a queueing application. Ann. Applied Prob., 1, 16–35. 32. Daley, D. J. and D. Vere-Jones (1988). An Introduction to the Theory of Point Processes. Springer, New York. 33. Deb, R. K. and R. Serfozo (1973). Optimal control of batch service queues. Adv. Appl. Prob., 5, 340–361. 34. Derman, C. (1955). Some contributions to the theory of denumerable Markov chains. Trans. Amer. Math. Soc., 79, 541–555. 35. Donsker, M. (1951). An invariance principle for certain probability limit theorems. Mem. Amer. Math. Soc. 6. 36. Donsker, M. (1952). Justification and extension of Doob’s heuristic approach to the Kolmogorov-Smirnov theorems. Ann. Math. Statist., 23, 277–281. 37. Durrett, R. (2005). Probability: Theory and Examples, 2nd ed. Duxbury Press, Belmont, CA. 38. Dynkin, E. B. (1965). Markov Processes, I II. Springer, New York. 39. El-Taha, M. and S. Stidham Jr. (1999). Sample-Path Analysis of Queueing Systems. Kluwer Academic Publishers, Boston. 40. Erlang, A. K. (1909). The theory of probabilities and telephone conversations. Nyt. Tidskr. Mat. B, 20, 33–41. 41. Ethier, S. N. and T. G. Kurtz (1986). Markov Processes: Characterization and convergence. John Wiley & Sons, New York. 42. Feller, W. (1968, 1971). An Introduction to Probability Theory and its Applications, Vol I (3rd ed.), Vol. II (2nd ed.) John Wiley & Sons, New York. 43. Fishman, G. S. (1996). Monte Carlo: Concepts, Algorithms, and Applications. Springer, New York. 44. Franken, P., K¨ oenig, D., Arndt, U. and V. Schmidt (1981). Queues and Point Processes. Akademie Verlag, Berlin. 45. Grigelionis, B. (1963). On the convergence of sums of random step processes to a Poisson process. Theory Probab. Appl., 8, 172–182. 46. Grimmett, G. R. and D. R. Stirzaker (2001). Probability and Random Processes, 3rd ed. Oxford University Press, Oxford. 47. Gross, D. and C. M. Harris (1985). Fundamentals of Queueing Theory. John Wiley & Sons, New York.

References

431

48. Hackman, S. T. (2007). Production Economics. Springer, New York. 49. Harris, T. E. (1963). The Theory of Branching Processes. Springer, New York. 50. Harrison, J. M. (1985). Brownian Motion and Stochastic Flow Systems. John Wiley & Sons, New York. 51. Harrison, J. M. (2000). Brownian models of queueing networks: canonical representation of workload. Ann. Appl. Prob., 10, 75–103. 52. Harrison, J. M. and R. J. Williams (1996). A multiclass closed queueing network with unconventional heavy traffic behavior. Ann. Appl. Prob., 6, 1–47. 53. Hewitt, E. and K. Stromberg (1965). Real and Abstract Analysis. Springer, New York. 54. Heyman, D. P. and M. J. Sobel (1982, 1984). Stochastic models in Operations Research, Vol I, Vol II. McGraw-Hill, New York. 55. Iglehart, D. L. and W. Whitt (1970). Multiple channel queues in heavy traffic, I, II. Adv. Appl. Prob., 2, 150–177, 355–369. 56. Jacod, J. and A. N. Shiryaev (1987). Limit Theorems for Stochastic processes. Springer, New York. 57. Jackson, J. R. (1957). Networks of waiting lines. Operations Res., 5, 518–552. 58. Jagers, P. (1975). Branching Processes with Biological Applications. John Wiley & Sons, New York. 59. Kalashnikov, V. V. (1994). Mathematical Methods in Queueing Theory. Kluwer, Dordrecht. 60. Kallenberg, O. (1986). Random Measures, 4th ed. Academic Press, London. 61. Kallenberg, O. (2004). Foundations of Modern Probability, 2nd ed. Springer, New York. 62. Karlin, S. and H. M. Taylor (1975). A First Course in Stochastic Processes, 2nd ed. Academic Press, New York. 63. Karlin, S. and H. M. Taylor (1981). A Second Course in Stochastic Processes, Academic Press, New York. 64. Karatzas, I. and S. Shreve (1991). Brownian Motion and Stochastic Calculus, 2nd ed. Springer, New York. 65. Karatzas, I. and S. Shreve (1998). Methods of Mathematical Finance, Springer, New York. 66. Karr, A. (1991). Point Processes and Their Statistical Inference. 2nd ed. Marcel Dekker, New York. 67. Kelly, F. P. (1979). Reversibility and Stochastic Networks. John Wiley & Sons, New York. 68. Kendall, D. G. (1953). Stochastic processes occuring in the theory of queues and their analysis by means of the imbedded Markov chain. Ann. Math. Statist., 24, 338–354. 69. Kesten, H. (1974). Renewal theory for Markov chains. Ann. Probab., 3, 355–387. 70. Khintchine, A. Y. (1960). Mathmatical Methods in the Theory of Queueing. Griffin, London. 71. Kiefer, J. and J. Wolfowitz (1956). On the characteristics of the general queueing process with applications to random walks. Ann. Math. Statist., 27, 147–161. 72. Kingman, J. F. C. (1961). The single server queue in heavy traffic. Proc. camb. Phil. Soc., 57, 902–904. 73. Kingman, J. F. C. (1993). Poisson Processes. Oxford University Press, Oxford. 74. Kleinrock, L. (1975, 1976). Queueing Systems. Volume 1 Theory, Volume 2 Computer Applications. John Wiley & Sons, New York. 75. Kolmogorov, A. N. (1936). Zur Theorie der Markoffschen Ketten. Math. Ann., 112, 155–160. 76. Kulkarni, V. G. (1995). Modeling and Analysis of Stochastic Systems. Chapman and Hall, London. 77. Kushner, H. J. (2001). Heany Traffic Analysis of Controlled Queueing and Communication Networks. Springer, New York.

432

References

78. Lefebvre, M. (2007). Applied Stochastic Processes. Springer, New York. 79. Lewis, P. A. W. and G. S. Shedler (1979). Simulation of nonhomogeneous Poisson processes by thinning. Navak Res, Losust. Quart., 29, 403–413. 80. Liggett, T. (1972). Interacting Particle Systems. Springer, New York. 81. Lindvall, T. (1992). Lectures on the Coupling Method. John Wiley & Sons, New York. 82. Little, J. D. C. (1961). A proof for the queueing formula: L = λW . Operations Res., 9, 383–387. 83. Mandelbaum, A. and W. A. Massey (1995). Strong approximation for time-dependent queues. Math. Oper. Res., 20, 33–64. 84. Meyn, S. and R. L. Tweedie (1993). Stationary Markov Chains and Stochastic Stability. Springer, New York. 85. Neuts, M. F. (1994). Matrix-Geometric Solutions in Stochastic Models. An algorithmic approach. Corrected reprint of the 1981 original. Dover Publications, Inc., New York. 86. Palm, C. (1943). Intensit¨ atsschwankungen in Fernsprechverkehr. Ericsson Technics, 44, 1-189. 87. Poisson, S. D. (1837). Recherches sur la Probabilit´ e des Jugements en Mati` ere Criminelle et en Mati` ere Civile, Pr´ ed´ ed´ ees des R´ egles G´ en´ erales du Calcul des Probabiliti´ es. Bachelier, Paris. 88. Prabhu, N. U. (1998). Queues, Insurance, Dams, and Data, 2nd ed. Springer, New York. 89. Prokhorov, Yu. V. (1956). Convergence of random processes and limit theorems in probability. Theory Probab. Appl., 1, 157–214. 90. Putterman, M. L. (1994). Markov decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, New York. 91. Reich, E. (1957). Waiting times when queues are in tandem. Ann. Math. Statist., 28, 768–773. 92. Reiman, M. I. (1984). Open queuing networks in heavy traffic. Math. Oper. Res., 9, 441–458. 93. Resnick, S. I. (1992). Adventures in Stochastic Processes. Birkh¨ auser, Boston. 94. Rolski, T., Schmidli, H., Schmidt, V. and J. Teugels. (1999). Stochastic Processes for Insurance and Finance. John Wiley & Sons, New York. 95. Robert, P. (2003). Stochastic Networks and Queues. Springer-Verlag, Berlin. 96. Rogers, L. C. G. and D. Williams (1979-94; 1987). Diffusions, Markov Processes, and Martingales, Vol. 1 (2dn ed.), and Vol. 2. John Wiley & Sons, Chichester. 97. Rolski, T. (1981). Stationary Random Processes Associated with Point Processes. Lecture Notes in Statistics, 5. Springer, New York. 98. Ross, S. (1996). Stochastic Processes, 2nd ed. John Wiley & Sons, New York. 99. Roussas, G. (1973). A First Course in Mathematical Statistics, Addison-Wesley, Reading, MA. 100. Ryll-Nardzewski, C. (1961). Remarks on process of calls. Proc. 4th Berkeley Symposium, vol 2. University of California Press, Berkeley, 455–465. 101. Serfozo, R. F. (1999). Introduction to Stochastic Networks. Springer, New York. 102. Shiryaev, A. N. (1995). Probability, 2nd ed. Springer, New York. 103. Sigman, K. (1995). Stationary Marked Point Processes: An Intuitive Approach. Chapman and Hall, New York. 104. Skorohod, A. V. (1956). Limit theorems for stochastic processes. Theory Probab. Appl., 1, 261–290. 105. Skorohod, A. V. (1957). Limit theorems for stochastic processes with independent increments. Theory Probab. Appl., 2, 122–142. 106. Stroock, D. W. (2005). An Introduction to Markov Processes. Graduate Texts in Mathematics 230, Springer, Berlin. 107. Smith, W. L. (1955). Regenerative stochastic processes. Proc. Roy. Soc., Ser. A, 232, 6–31.

References

433

108. Tak´ acs, L. (1967). Combinatorial Methods in the theory of Stochastic Processes. John Wiley & Sons, New York. 109. Tijms, H. (1986). Stochastic Modelling and Analysis: A Computational Approach. John Wiley & Sons, New York. 110. Walrand, J. (1988). Introduction to Queueing Networks. Prentice–Hall, Englewood Cliffs, New Jersey. 111. Whittle, P. (1986). Systems in Stochastic Equilibrium. John Wiley & Sons, New York. 112. Whitt, W. (1983). The queueing network analyzer. AT&T Bell Labs. Tech. J., 62, 2779–2815. 113. Whitt, W. (2002). Stochastic Process Limits. Springer, New York. 114. Williams, D. (1991). Probability with Martingales. Cambridge University Press, Cambridge. 115. Williams, R. J. (1998). Diffusion approximations for open multiclass queueing networks: sufficient conditions involving state space collapse. Queueing systems, 30, 27–88. 116. Wolff, R. W. (1989). Stochastic Modelling and the Theory of Queues. Prentice-Hall, New York.

Notation

1(·)

The indicator function that is 1 or 0 when (·) is true or false Backward recurrence time A(t) = t − TN (t) B(t) = TN (t)+1 − t Forward recurrence time B, B(t) Standard Brownian motion process + CK (S) Set of continuous f : S → R+ with compact support Dirac measure with unit mass at 1 δx (A) = 1(x ∈ A) D Set of right-continuous, piece-wise constant functions x : R → S with left-hand limits, and finite number of jumps in finite intervals D = D[0, 1], D[0, T ], D(R) Set of real valued functions on [0, 1], [0, T ], R that are right continuous with left-hand limits DRI Directly Riemann integrable E Expectation operator E[X|Y ], E[X|A] Conditional expectations E[esX ] Moment generating function of X ! −αt ˆ Laplace transform of F F (α) = R+ e dF (t) ei = (0, . . . , 0, 1, 0 . . . , 0) ith unit vector F , Ft , Fn σ-field of events σ-field of events up to stopping time τ Fτ FtY σ-field of events generated by process Y f (t), g(t), h(t), H(t) Functions f equals its positive part minus its negative part f (t) = f + (t) − f − (t) F (t), G(t) ! Distribution functions x Fe (x) = μ1 0 [1 − F (s)]ds Equilibrium distribution of F Invariant measure of a CTMC γi H(t) = h(t) + F H(t) Renewal equation λ, λ(A) Arrival rate, and rate of entering A μ, μ(A) Service rate or mean, and measure of A M (t), M (A × B) Counting process, or Poisson process

435

436

Notation

M (t) = maxs≤t B(s) N (t) NT N (μ, σ 2 )

ξn = Tn+1 − Tn W (t), W (A) Z, Z+

Maxima of Brownian motion B Counting process, or renewal process Point process of T -transitions of a CTMC Normal random variable with mean μ and variance σ 2 Matrix of Markov chain transition probabilities X is measurable map from a probability space to S Palm probability of a T -transition of a CTMC Transition probability of a CTMC Stationary distributions of a CTMC Exponential sojourn rate in state i of a CTMC Transition rates of a CTMC Transition rate matrix of a CTMC The real numbers and nonnegative real numbers A countable state space for Markov chains in Chapter 1 Polish space and its Borel sets and bounded Borel sets Stopping time, and entrance times to state i Subset of sample paths in D A jump time t of a CTMC X with S t X ∈ T Time of nth event occurrence, or nth renewal time, or time of nth jump in a CTMC Renewal function Markov chains or sequences of random variables Continuous-time stochastic processes nth inter-renewal time, or time between event occurrences Sojourn time in a CTMC Waiting time process or sojourn time in set A The integers and nonnegative integers

a.s. x ∨ y = max{x, y} x ∧ y = min{x, y}

Almost surely, meaning with probability one Maximum of x and y Minimum of x and y

P = (pij ) Random element X in S PT (·) pij (t) pi , p(x) qi qij , q(x, y) Q = (qij ) R, R+ S S, S, Sˆ τ, τi , τi (n) T T -transition Tn  n (t) U (t) = ∞ n=0 F Xn , Yn X(t), Y (t), Z(t) ξn = Tn − Tn−1

d

X(t) → Y d

X =Y x+ = max{0, x} x− = − min{0, x} N (t) xn=1 (·) = 0 k=1 (·) = 1 x x f (t) = o(g(t)) as t → t0 f (t) = O(g(t)) as t → t0 a ⇒ b, a ⇔ a

X(t) converges in distribution to Y as t → ∞ The distributions of X and Y are equal Positive part of x Negative part of x and x = x+ − x− When N (t) = 0 When x = 0 Largest integer ≤ x, or the integer part of x Smallest integer ≥ x limt→t0 f (t)/g(t) = 0 lim supt→t0 |f (t)|/|g(t)| < ∞ a implies b, and a is equivalent to b

Index

T -transitions of CTMC Palm probability, 299–314 PASTA, 303–306 Poisson, 291–299 Arithmetic and non-arithmetic distributions, 108, 117 Asymptotic stationarity CTMC, 329 Markov chain – discrete time, 41 Bernoulli process, 93, 156, 216 Borel sets, 122, 145, 406, 409 bounded, 170, 181 Brownian bridge, 380 as Gaussian process, 380 empirical distribution, 380 Kolmogorov-Smirnov statistic, 381 probability, 404 relation to Brownian motion, 403 Brownian motion, 341–404 arc sine law, 352 as diffusion process, 344 as Gaussian process, 347 Brownian/Poisson particle system, 387, 403 CTMC approximation, 376 definition, 342 Donsker’s theorem, 370 existence, 348 functions that are Brownian, 394 geometric Brownian motion, 383 heavy-traffic queueing approximation, 389 hitting times, 350, 361 in random environment, 393 law of iterated logarithm, 364

Markov chain – discrete time approximation, 403 maximum process, 349 SLLN, 365 minimum process, 351 multidimensional, 385 Bessel process, 385, 404 optional stopping, 360 peculairities of sample paths, 377–379 quadratic variation, 400 random walk approximation, 372 reflection principle, 350, 398 regenerative-increment approximation, 373 related martingales, 356 relation to Brownian bridge, 380, 403 renewal approximation, 376 Skorohod embedding, 371 SLLN, 364 strong Markov property, 344 symmetry property, 343 with drift, 343 Campbell formula, 110, 307 Central limit theorems Anscombe theorem, 136, 375 CLT – renewal and regenerative, 135–139 CLT, Markov chains, 138, 170, 333 CLT, random walk, 166 Donsker’s FCLT, 368–373 FCLT – regenerative-increment and Markov chains, 374 FCLT – renewal process, 376 Chebyshev’s inequality, 415 Composition mapping, 374, 394, 406, 412 Conditional expectation, 81, 355, 358, 417

437

438 Conditional probabilities, 81–83, 417 Conditionally independent, 47 Confidence interval for mean, 137, 166 Continuous-mapping theorem, 369, 370, 382, 393, 402, 422, 423 Convergence lim inf, 421 lim sup, 421 a.s., 47, 421 asymptotic stationarity, 41 coupling, 73 in distribution, 41, 42, 422 Donsker’s theorem, 370 FCLT, 370 in D(R+ ), 371 in D[0, 1], 369 invariance principle, 370 point processes, 218 queues in heavy traffic, 389 random walk, 372 in probability, 421, 422 in total variation, 74 martingales and submartingales, 357 of partitions, 224 rate for CTMC, 266 real numbers, 421 sums of point processes, 220 vague, 218 weak, 218, 422 Convolution, 416 inter-renewal time, 110 networks, 57, 227, 334 renewal function, 114, 167 sums, 101 Coupling, 24, 40, 73, 424 Covariance Brownian bridge, 380 Brownian motion, 343, 347, 386 Gaussian process, 346 Jackson networks, 334 moving average, 396 Ornstein-Uhlenbeck, 348 renewal limits, 164 weakly stationary, 396 CTMC, 241–340 as a Markov jump process, 242 asymptotic stationarity, 329 Chapman Kolmogorov equations, 250 CLT, 333 compound Poisson process, 323 defining parameters, 243 existence, 253 finite dimensional distributions, 250

Index formulated by clock times, 246, 323 Jackson networks, 282–287, 298, 331, 334 Kolmogorov differential equations, 251 L´ evy formula, 270 Markov property, 248, 325 Markov-Renewal process, 321 Markov/Poisson particle system, 339 multiclass networks, 287–291, 336 P-regular transition rates, 244, 245, 253, 322, 331 Palm probabilities, 299–314 Poisson transition times, 291–299 regenerative property, cycle costs, 263 reversibility, 272–281, 333 SLLN, 264–269 stationary and limiting distributions, 258–262 transition rates, 251, 252 uniform transition rates, 256, 329 Diagonal selection principle, 75 Distribution function, 406 χ-squared, 386 arc sine and arc cosine, 352, 353, 397 arithmetic and non-arithmetic, 108, 262 beta, 146, 178 exponential, 102, 125, 225, 366 geometric, 5, 12, 21, 85, 96 Gumbel, 338 multivariate normal, 345 table of continuous distributions, 408 table of discrete distributions, 407 Dominated convergence theorem, 424 Empirical distribution, 380, 402 Equilibrium distribution, see Stationary distribution Estimation consistent estimator, 68, 96, 105, 380 CTMC stationary distribution, 330 empirical distribution, 380 Poisson rate, 105 simulation, 68 transition probabilities, 97 unbiased estimator, 105, 380 Extreme-value process, 87, 338 Fatou’s lemma, 424 Finite-dimensional distributions, 2, 410 Brownian motion, 343 convergence, 368 Gaussian, 347 independent random elements, 420

Index Kolmogorov consistency condition, 419 Markov chain discrete time, 6 point process, 182 random element, 409 First entrance time, see Stopping time Fubini’s theorem, 415 Hitting time, see Stopping time Integral convergence, 219, 424 directly Riemann integrable, 118, 151 expectation, 407, 410 Fubini theorem, 416 Laplace transform, 184 Lebesgue, 413 point process, 184, 187 renewal process, 159 Riemann integral, 118, 151 Riemann-Stieltjes, 109, 379, 410 Intensity measure, see Poisson process Invariant measure, 260 CTMC, 260, 272 Jackson network, 284 Markov chain – discrete time, 35 Inventory model, 10, 43, 85, 87, 399 (s, S), 10, 43, 53 production-inventory, 161, 330 reflected random walk, 87 Jensen’s inequality, 415 Kolmogorov extension theorem, 420 Kolmogorov reversibility criterion, 64 L´ evy formula, 270, 271 Laplace functional, 184 convergence, 219 moments, 238 Poisson, 185 Laplace transform, 109, 414 Limiting Distributions, 122 crude regenerations, 120 cyclic renewal process, 165 Kolmogorov-Smirnov statistic, 381 Markov chain in discrete time, 40–42, 74, 126 Markov-renewal process, 322 regenerative process, 121 renewal process, 124 SLLN, 47 stationary distribution, 90, 258 waiting times in G/G/1 queue, 315

439 Lindley recursion, 15, 314, 390 Locally finite measure, 182, 183, 186, 189, 190, 211, 218 M/M/1, see Queueing Process, M/M/s Machine maintenance, see Production model Markov chain transition probabilities, 3 Markov chain – discrete time, 1–98 absorbing state, 22 aperiodic state, 23 Chapman-Kolmogorov equations, 75 classification of states, 19–26 closed class, 22 communication graph, 73 construction, 9 coupling, 73 definition, 2 ergodic, 26, 36 existence, 9 first-step analysis, 20 Foster criterion, 76, 77 hitting probabilities, 26–30 hitting time, 16, 43 invariant measure, 35, 38 irreducible set, 22 limiting distribution, 40 maxima, 8 Monte Carlo, 68–71 non-homogeneous, 2, 87 null-communication relation, 22 null-recurrent state, 20 on subspace, 71 optimal design, 53 Pake criterion, 78 periodic state, 23 positive recurrent state, 20 rate of transitions, 50 regenerative property, 18, 43 reversible, 61–67 sample path probabilities, 6 simulation of, 9 SLLN, 46, 47 sojourn time, 51 stationary distribution, 35, 36 stopping time, 16, 43, 73, 88 strong Markov property, 17 taboo probability, 7, 28, 39 transient state, 20 transition graph, 23, 26 transition probabilities, 2, 6 two state, 86

440 Markov property continuous time, 242, 248 Markov chain – discrete time, 2 strong – Brownian motion, 344, 350 strong – continuous time, 250, 262 strong – discrete time, 16 Markov’s inequality, 414 Martingales and submartingales, 354–357 Brownian motion, 356 Brownian motion hitting times, 363, 364 convergence theorem, 357 optional stopping, 358, 360 stationary independent increments, 356 Mean measure, see Intensity measure Measurable function, 109, 145, 369, 406, 407, 410 composition, 412 point process, 182 probability kernel, 417 random element, 409 Memoryless property exponential distribution, 125, 225, 248 geometric distribution, 85 Markov property, 13 Modulus of continuity, 372 Moments characteristic function, 414 generating function, 414 Laplace transform, 414 moment generating function, 413 random variable, 407 Monotone convergence theorem, 424 Moving average, 93 Optional stopping of martingales, 358–361 Order statistics, 178, 179, 229 P-regular transition rates, see CTMC Palm probabilities for CTMC, 299–314 Campbell formula, 307 definition, 300 PASTA, 303 SLLN, 311 sojourn and travel times, 312 stationarity property, 310 time dependent, 302 Parallel processing, 79, 93, 157, 203, 230, 245 Particle system Brownian/Poisson, 387 Ehrenfest, 92 Markov/Poisson, 201, 339 Point process, 100 Campbell formula, 110 cluster process, 238

Index compound, 214 convergence to Poisson, 218–225 counting process, 170 Cox process, 211 delayed renewal, 160 finite-dimensional distributions, 182 general space, 182 infinitely divisible, 234 integral, 184, 187 intensity measure, 182 Laplace transform, 185 marked, 192 Markov/Poisson, 201 mixed sample process, 188 moments, 238 multiple points, 161 non-homogeneous renewals, 162 order statistics, 229 partition, 197 Poisson, 170 Poisson cluster process, 215 Poisson process on general space, 183 queues, 129 regeneration times, 147 renewal, 100 sample process, 188 simple, 170 SLLN, 104, 160 space-time Poisson process, 194 stationary, 145 stationary renewal process, 145 sum of sparse processes, 220 thinning, 222, 239 transformation, 191 Poisson process, 169–239 Mt /Gt /∞ systems, 203, 206 approximation for partitions, 224 approximation for point-process sums, 220 as renewal process, 173 as sample process, 189 classical, 170 compound Poisson, 214 convergence to, 219 Cox process relative, 211 existence, 190 general space, 183 infinitesimal properties, 173 intensity measure, 182, 185 Laplace functional, 185 marks and p-marks, 194, 195, 197, 200 multinomial point locations, 176 order statistic property, 179 rare event approximation, 216

Index rate function, 183, 190, 198, 231 simulation of, 198 space-time, 194, 203, 205, 207, 215, 236 splitting and merging, 199 thinning and partitioning, 197 transformations, 190–200 translations, 200 Polish space, see State space Portmanteau theorem for convergence, 423 Probability distribution, see Distribution function Probability kernel, 417 Probability space, 2, 405 σ-field, 2, 405, 406 filtration, 354 probability measure, 2, 405 Processor-sharing, 291 Production model dynamic servicing, 230 flexible manufacturing, 4, 37 fork-join network, 80, 98 job processing, 230 line interruptions, 186 machine availability, 86 machine deterioration, 10, 37, 52, 257 machine network, 11 machine replacement, 91 optimal machine replacement, 54 production-inventory, 161 production-maintenance network, 286 Pull-through property, 44, 77, 83, 259, 263 Queueing network, see Stochastic network Queueing process G/G/1 heavy traffic, 389–393 Little law, 131, 165 waiting time limits, 315 M/G/1 and G/M/1 waiting time and queue limits, 317–321 M/G/∞ and Mt /Gt /∞, 203–211 bounded queue, 278 departure process, 232 in space, 236 multiclass, 235 stationary distribution, 235 M/M/s arrival process, 296 as birth-death process, 247 balking and reneging, 326 departure process, 297 production-inventory system, 330 stationary distribution, 275 waiting times, 304, 312

441 with feedbacks, 324 acyclic network, 208 batch service, 327, 338 optimal batch size, 132 merging process, 325 optimal dispatching, 172, 228 regenerative process Little law, 129, 130 with blocking, 326 Queueing process in discrete time M/M/1 system, 12 buffer sharing, 94 busy period, 19, 94 stationary distribution, 44 with costs, 94 closed and open networks, 55–61 fork-join network, 80 optimal design, 54 perishable service, 85 reflected random walk, 13 Random walk, 3, 13 approximated by Brownian motion, 372 continuous time, 370 Donsker’s FCLT, 368 gambler’s ruin, 4, 29, 92 in G/G/1 queues, 389, 403 multidimensional, 386 on circles, 65, 73, 95 on graphs, 92, 95 period, 95 range of, 403 reflected, 13, 86, 88, 323 reversible, 62 Skorohod embedding, 371 Rate function, see Poisson process Record value, 87 Reflection Brownian motion, 343, 367, 394 mapping, 392 principle of Brownian motion, 350, 351, 398 random walk, 13, 86–88 Regenerative process, 121–125 batch-service queue, 132–135 CLT, 136 crude regenerations, 120 definition, 121 inheritance, 122 limiting distribution, 123 Little law, 129 Regenerative-increment process, 126–128 CLT, 136

442 definition, 127 FCLT, 374 SLLN, 128 Wald identity, 127 Renewal Process, 99–167 alternating, 104 Blackwell’s theorem, 116 cyclic, 103 definition, 100 delayed, 103 direct Riemann integrable, 119 elementary renewal theorem, 116 key renewal theorem, 119 renewal equation, 115 renewal function, 108 SLLN, 105 with rewards, 107 Renewal process backward and forward recurrence time, 124 key renewal theorem proof, 151 refined limit laws, 148 SLLN for Markov chain – discrete time, 126 stationary, 144–148 stopping time, 112 terminating, 139–144 Reversible CTMC, 272–281 Markov chain in discrete time, 61–67 simulation, 68 Simulation, 68–71 Gibbs sampler model, 69 Hastings-Metropolis model, 69 Markov chain in discrete time, 10 Poisson process, 198 random variable, 9, 86 Skorohod embedding theorem, 371 SLLN, 104–107 Brownian motion, 364–366 CTMC, 264–269 Markov chain – discrete time, 45–53 Palm probability, 311 renewal and regeneration, 105, 127, 148 Sojourn time Markov chain in discrete time, 85 CTMC, 243, 246, 254, 257, 265, 312, 323, 327 cyclic renewal process, 107 highway, 233 Little law, 130 Markov chain in discrete time, 51 queues, 131, 207, 209, 304, 336, 338

Index random walk on graph, 95 regenerative process, 129–132 uniform rates in CTMC, 255 State space, 2, 181, 409, 424 Stationary distribution backward recurrence time, 125, 148 forward recurrence time, 125, 145, 156 Markov chain – continuous time, 258–262 Markov chain – discrete time, 33–42 regenerative cycle costs, 42, 263 Stochastic network BCMP, 291 fork-join, 337 Jackson, 282–287 departure process, 298 multiclass, 336 star-shaped, 335 tandem, 331 variable waiting space, 333 Kelly, 288, 336 multiclass, 287–291 Stochastic process, 2, 409 Bernoulli, 3 Bessel, 385, 404 birth-death, 247, 275, 278, 296, 297, 331 branching, 30–33 Brownian bridge, 379–383 Brownian motion, 341–404 compound Poisson, 181, 214–216, 238, 384, 398 diffusion, 343 finite-dimensional distributions, 2 Gaussian, 346–349, 379, 396 geometric Brownian motion, 383–385 Markov chain – continuous time, 241–340 Markov chain – discrete time, 1–98 martingale, 354–361 multidimensional Brownian motion, 385–387 Ornstein-Uhlenbeck, 348 point process, 100, 104, 110, 129, 145, 160, 161 Poisson, 169–239 random walk, 3 renewal and regenerative, 99–167 stationary, 33, 144, 155, 167, 260 submartingale, 355, 357, 401, 402 supermartingale, 355, 402 weakly stationary, 396 Stopping times Brownain motion, 362, 371, 386 Brownian motion, 344, 350 coupling, 73 criterion, 88

Index CTMC, 250, 254, 330 entrance time, 18, 20 filtration, 344 hitting time, 16, 20 Markov chain in discrete time, 16, 17 Markov chain regenerations, 18, 43 martingales, 358–361, 400 reflection principle, 398 renewal, 112 Skorohod embedding, 371 stopped martingale, 359 sums, 400 Subadditive renewal function, 162 Subordinated Brownian motion, 394 Subordinated Markov chain, 255, 292, 295 Success runs, 11 Supremum mapping, 391 Supremum norm on D(0,1], 369–371, 373, 374 Total variation distance, 74 Traffic equations, 289, 290 Traffic intensity, 45, 314

443 Transforms characteristic function, 414 Laplace, 414 moment generating function, 414 Truncation of state space, 278, 279 Uniform norm, see Supremum norm on D[0,1] Uniformly integrable, 357, 424 Vague convergence, 218 Variance, 407 Brownian motion, 342 movimg average, 396 renewal integral, 187 renewal process, 102 shot-noise, 229 table of distributions, 407 Waiting time, see Sojourn time Wald identity, 112, 116, 127, 360, 400 Weakly stationary, 396 Wiener process, see Brownian motion