Hostname: page-component-74656f9884-j7cmm Total loading time: 0 Render date: 2026-11-10T09:32:53.831Z Has data issue: false hasContentIssue false
  • English
  • Français

Control of singularly perturbed Markov chains: A numerical study

Published online by Cambridge University Press:  17 February 2009

H. Yang ,
G. Yin ,
K. Yin  and
Q. Zhang
H. Yang
Affiliation:
Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455, USA; e-mail: hyang@ece.umn.edu.
G. Yin
Affiliation:
Department of Mathematics, Wayne State University, Detroit, MI 48202, USA; e-mail: gyin@math.wayne.edu.
K. Yin
Affiliation:
Department of Wood and Paper Science, University of Minnesota, St. Paul, MN 55108, USA.
Q. Zhang
Affiliation:
Department of Mathematics University of Georgia, Athens, GA 30602, USA.
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the 'Save PDF' action button.

This work is devoted to numerical studies of nearly optimal controls of systems driven by singularly perturbed Markov chains. Our approach is based on the ideas of hierarchical controls applicable to many large-scale systems. A discrete-time linear quadratic control problem is examined. Its corresponding limit system is derived. The associated asymptotic properties and near optimality are demonstrated by numerical examples. Numerical experiments for a continuous-time hybrid linear quadratic regulator with Gaussian disturbances and a discrete-time Markov decision process are also presented. The numerical results have not only supported our theoretical findings but also provided insights for further applications.

Information

Type
Research Article
Information
The ANZIAM Journal , Volume 45 , Issue 1 , July 2003 , pp. 49 - 74
Copyright
Copyright © Australian Mathematical Society 2003

References

[1]Abbad, M., Filar, J. A. and Bielecki, T. R., “Algorithms for singularly perturbed limiting average Markov control problems”, IEEE Trans. Automat. Control AC-37 (1992) 1421–1425.CrossRefGoogle Scholar
[2]Bertsekas, D., Dynamic programming: deterministic and stochastic models (Prentice-Hall, Englewood Cliffs, New Jersey, 1987).Google Scholar
[3]Blankenship, G., “Singularly perturbed difference equations in optimal control problems”, IEEE Trans. Automat. Control T-AC 26 (1981) 911–917.CrossRefGoogle Scholar
[4]Courtois, P. J., Decomposability: queuing and computer system applications (Academic Press, New York, 1977).Google Scholar
[5]Davis, M. H. A., Markov models and optimization (Chapman and Hall, London, 1993).CrossRefGoogle Scholar
[6]Delebecque, F. and Quadrat, J., “Optimal control for Markov chains admitting strong and weak interactions”, Automatica 17 (1981) 281–296.CrossRefGoogle Scholar
[7]Ethier, S. N. and Kurtz, T. G., Markov processes: characterization and convergence (J. Wiley, New York, 1986).CrossRefGoogle Scholar
[8]Fleming, W. H. and Rishel, R. W., Deterministic and stochastic optimal control (Springer, New York, 1975).CrossRefGoogle Scholar
[9]Hoppensteadt, F. C. and Miranker, W. L., “Multitime methods for systems of difference equations”, Studies Appl. Math. 56 (1977) 273–289.CrossRefGoogle Scholar
[10]Khasminskii, R. Z., Yin, G. and Zhang, Q., “Asymptotic expansions of singularly perturbed systems involving rapidly fluctuating Markov chains”, SIAM J. Appl. Math. 56 (1996) 277–293.Google Scholar
[11]Khasminskii, R. Z., Yin, G. and Zhang, Q., “Constructing asymptotic series for probability distribution of Markov chains with weak and strong interactions”, Quart. Appl. Math. 55 (1997) 177–200.CrossRefGoogle Scholar
[12]Kushner, H. J., Approximation and weak convergence methods for random processes, with applications to stochastic systems theory (MIT Press, Cambridge, MA, 1984).Google Scholar
[13]Kushner, H. J. and Yin, G., Stochastic approximation algorithms and applications (Springer, New York, 1997).CrossRefGoogle Scholar
[14]Liu, R. H., Zhang, Q. and Yin, G., “Nearly optimal control of singularly perturbed Markov decision processes in discrete time”, Appl. Math. Optim. 44 (2001) 105–129.Google Scholar
[15]Pan, Z. G. and Basar, T., “H ∞-control of Markovian jump linear systems and solutions to associated piecewise-deterministic differential games”, in New trends in dynamic games and applications (ed. Olsder, G. J.),(Birkhäuser, Boston, 1995) 61–94.CrossRefGoogle Scholar
[16]Pervozvanskii, A. A. and Gaitsgori, V. G., Theory of suboptimal decisions: decomposition and aggregation (Kluwer, Dordrecht, 1988).CrossRefGoogle Scholar
[17]Phillips, R. G. and Kokotovic, P. V., “A singular perturbation approach to modelling and control of Markov chains”, IEEE Trans. Automat. Control 26 (1981) 1087–1094.Google Scholar
[18]Ross, S., Introduction to stochastic dynamic programming (Academic Press, New York, 1983).Google Scholar
[19]Sethi, S. P. and Zhang, Q., Hierarchical decision making in stochastic manufacturing systems (Birkhäuser, Boston, 1994).CrossRefGoogle Scholar
[20]Simon, H. A. and Ando, A., “Aggregation of variables in dynamic systems”, Econometrica 29 (1961) 111–138.Google Scholar
[21]Thompson, W. A. Jr., Point process models with applications to safety and reliability (Chapman and Hall, New York, 1988).CrossRefGoogle Scholar
[22]Tse, D. N. C., Gallager, R. G. and Tsitsiklis, J. N., “Statistical multiplexing of multiple time-scale Markov streams”, IEEE J. Selected Areas Comm. 13 (1995) 1028–1038.CrossRefGoogle Scholar
[23]White, D. J., Markov decision processes (Wiley, New York, 1992).Google Scholar
[24]Yin, G. and Zhang, Q., Continuous-time Markov chains and applications: a singular perturbation approach (Springer, New York, 1998).CrossRefGoogle Scholar
[25]Yin, G. and Zhang, Q., “Singularly perturbed discrete-time Markov chains”, SIAM J. Appl. Math. 61 (2000) 834–854.CrossRefGoogle Scholar
[26]Yin, G., Zhang, Q. and Badowski, G., “Asymptotic properties of a singularly perturbed Markov chain with inclusion of transient states”, Ann. Appl. Probab. 10 (2000) 549–572.Google Scholar
[27]Yin, G., Zhang, Q. and Badowski, G., “Discrete-time singularly perturbed Markov chains: aggregation, occupation measures, and switching diffusion limit”, to appear in Adv. Appl. Probab.Google Scholar
[28]Zhang, Q. and Yin, G., “On nearly optimal controls of hybrid LQG problems”, IEEE Trans. Automat. Control 44 (1999) 2271–2282.CrossRefGoogle Scholar