Hostname: page-component-7b7567c8c-z4zkx Total loading time: 0 Render date: 2026-11-16T23:23:30.676Z Has data issue: false hasContentIssue false

Uniformization for semi-Markov decision processes under stationary policies

Published online by Cambridge University Press:  14 July 2016

Frederick J. Beutler*
Affiliation:
University of Michigan, Ann Arbor
Keith W. Ross*
Affiliation:
University of Pennsylvania
*
∗ Postal address: Department of Electrical Engineering and Computer Science, EECS, University of Michigan, Ann Arbor, MI 48109, USA.
∗∗ Postal address: Department of Systems Engineering, University of Pennsylvania, Philadelphia, PA 19104, USA.

Abstract

Uniformization permits the replacement of a semi-Markov decision process (SMDP) by a Markov chain exhibiting the same average rewards for simple (non-randomized) policies. It is shown that various anomalies may occur, especially for stationary (randomized) policies; uniformization introduces virtual jumps with concomitant action changes not present in the original process. Since these lead to discrepancies in the average rewards for stationary processes, uniformization can be accepted as valid only for simple policies.

We generalize uniformization to yield consistent results for stationary policies also. These results are applied to constrained optimization of SMDP, in which stationary (randomized) policies appear naturally. The structure of optimal constrained SMDP policies can then be elucidated by studying the corresponding controlled Markov chains. Moreover, constrained SMDP optimal policy computations can be more easily implemented in discrete time, the generalized uniformization being employed to relate discrete- and continuous-time optimal constrained policies.

Information

Type
Research Papers
Copyright
Copyright © Applied Probability Trust 1987 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable