8385 - APPLIED PROBABILITY MODELS AND MARKOV PROCESSES

Markov chains are the simplest mathematical models for random phenomenon evolving in time. Their simple structure makes it possible to say a great deal about their behaviour. At the same time, the class of Markov chains is rich enough to serve in many applications. This makes Markov chains the first and most important examples of random processes.
Indeed, the whole of the mathematical study of random processes can be regarded as a generalization in one way or another of the theory of Markov chains.
The course is an account of the elementary theory of Markov chains, with applications.

• Probability and Measure
  Probability spaces and Expectations; Monotone convergence and Fubini's theorem
• Discrete-time Markov chains
  Definitions and basic properties; Class structure; Hitting time and absorption probabilities; Strong Markov property; Recurrence and transience; Recurrence and transience of random walks; Invariant distributions; Convergence to equilibrium; Time reversal; Ergodic theorem
• Continuous-time Markov chains
  Q-matrices and their exponential; Continuous-time random processes; Some properties of the exponential distribution; Poisson processes; Jump chain in and holding times; Explosion; Forward and backward equations; Non-minimal chains; Continuous-time properties
• Further Theory
• Martingales; Brownian motion
• Applications
  Markov chains in Finance; Markov chains in Biology; Applications in Genetics; Queues and queuing networks; Markov chains in resource management; Markov chains and decision processes; Markov chains Monte Carlo

The  exam consists of an essay on a topic chosen by the student, that is discussed with the professor.

• J.R. NORRIS, Markov Chains, Cambridge University Press, 1998