Continuous-time controlled Markov chains with safety upper bound

Continuous-time controlled Markov chains with safety upper bound

For access to this article, please select a purchase option:

Buy article PDF
(plus tax if applicable)
Buy Knowledge Pack
10 articles for $120.00
(plus taxes if applicable)

IET members benefit from discounts to all IET publications and free access to E&T Magazine. If you are an IET member, log in to your account and the discounts will automatically be applied.

Learn more about IET membership 

Recommend Title Publication to library

You must fill out fields marked with: *

Librarian details
Your details
Why are you recommending this title?
Select reason:
IET Control Theory & Applications — Recommend this title to your library

Thank you

Your recommendation has been sent to your librarian.

This study introduces the notion of safety for the controlled Markov chains in the continuous-time horizon. The concept is a non-trivial extension of safety control for stochastic systems modelled as discrete-time Markov decision processes, where the safety means that the probability distributions of the system states will not visit the given forbidden set at any time. In this paper study a unit-interval-valued vector that serves as an upper bound on the state probability distribution vector characterises the forbidden set. A probability distribution is then called safe if it does not exceed the upper bound. Under mild conditions the author derives two results: (i) the necessary and sufficient conditions that guarantee the all-time safety of the probability distributions if the starting distribution is safe, and (ii) the characterisation of the supreme set of safe initial probability vectors that remain safe as time passes. In particular, study the paper identifies an upper bound on time and shows that if a distribution is always safe before that time, the distribution is safe at all times. Numerical examples are provided to illustrate the two results.


    1. 1)
      • E. Altman . (1999) Constrained Markov Decision Processes.
    2. 2)
      • Feinberg, E.A.: `Constrained finite continuous-time Markov decision processes with average rewards', Proc. IEEE Conf. on Decisions and Control, 2002, p. 3805–3810.
    3. 3)
    4. 4)
      • M.L. Puterman . (1994) Markov Decision Process: discrete stochastic dynamic programming.
    5. 5)
      • Arapostathis, A., Kumar, R., Hsu, S.-P.: `State-feedback control of Markov chains with safety bounds', Proc. IEEE Conf. on Decision and Control, 2003, Maui, Hawaii, p. 6283–6288.
    6. 6)
    7. 7)
    8. 8)
      • E. Seneta . (1981) Non-negative Matrices and Markov Chains.

Related content

This is a required field
Please enter a valid email address