Hostname: page-component-74656f9884-c52kr Total loading time: 0 Render date: 2026-11-09T04:23:50.896Z Has data issue: false hasContentIssue false
  • English
  • Français

Markov renewal theory

Published online by Cambridge University Press:  01 July 2016

Erhan Çinlar
Erhan Çinlar*
Affiliation:
Northwestern University, Evanston, Illinois
Get access Rights & Permissions [Opens in a new window]

Extract

Consider a stochastic process X(t) (t ≧ 0) taking values in a countable state space, say, {1, 2,3, …}. To be picturesque we think of X(t) as the state which a particle is in at epoch t. Suppose the particle moves from state to state in such a way that the successive states visited form a Markov chain, and that the particle stays in a given state a random amount of time depending on the state it is in as well as on the state to be visited next. Below is a possible realization of such a process.

Information

Type
Research Article
Information
Advances in Applied Probability , Volume 1 , Issue 2 , Autumn 1969 , pp. 123 - 187
Copyright
Copyright © Applied Probability Trust 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Anderson, L. B. (1967) Filtered semi-Markov processes. Unpublished Master's thesis, Northwestern University.Google Scholar
Anselone, P. M. (1960) Ergodic theory for discrete semi-Markov chains. Duke Math. J. 27, 33–40.CrossRefGoogle Scholar
Barlow, R. E. (1962) Applications of semi-Markov processes to counter problems. In Studies in Applied Probability and Management Science. Arrow, , Karlin, , Scarf, , Editors, Stanford University Press, Stanford.Google Scholar
Barlow, R. E. and Proschan, F. (1964) Mathematical Theory of Reliability. Wiley, New York.Google Scholar
Bartlett, M. S. (1953) Recurrence and first passage times. Proc. Camb. Phil. Soc. 49, 263–275.CrossRefGoogle Scholar
Beneš, V. E. (1962) A renewal limit theorem for general stochastic processes. Ann. Math. Statist. 33, 98–113.CrossRefGoogle Scholar
Blackwell, D. (1948) A renewal theorem. Duke Math. J. 15, 145–150.Google Scholar
Çinlar, E. (1967a) Decomposition of a semi-Markov process under a state dependent rule. SIAM J. Appl. Math. 15, 252–263.Google Scholar
Çinlar, E. (1967b) Time dependence of queues with semi-Markovian services. J. Appl. Prob. 4, 356–364.Google Scholar
Çinlar, E. (1967C) Queues with semi-Markovian arrivals. J. Appl. Prob. 4, 365–379.Google Scholar
Çinlar, E. (1968a) Some joint distributions for Markov renewal processes. Aust. J. Statist. 10, 8–20.CrossRefGoogle Scholar
Çinlar, E. (1968b) On semi-Markov process on arbitrary spaces. Proc. Camb. Phil. Soc. (to appear).CrossRefGoogle Scholar
COx, D. R. (1962) Renewal Theory. Wiley, New York.Google Scholar
Derman, C. (1954) A solution to a set of fundamental equations in Markov chains. Proc. Amer. Math. Soc. 5, 332–334.CrossRefGoogle Scholar
Derman, C. (1955) Some contributions to the theory of denumerable Markov chains. Trans. Amer. Math. Soc. 79, 541–555.Google Scholar
Derman, C. (1961) Remark concerning two state semi-Markov processes. Ann. Math. Statist. 32, 615–616.CrossRefGoogle Scholar
Dynkin, E. B. (1965) Markov Processes, 2 vols. Springer Verlag, Berlin.Google Scholar
Feller, W. (1956) Boundaries induced by non-negative matrices. Trans. Amer. Math. Soc. 83, 19–54.Google Scholar
Feller, W. (1957), (1966) An Introduction to Probability Theory and Its Applications 1 and 2. Wiley, New York.Google Scholar
Feller, W. (1964) On semi-Markov processes. Proc. Nat. Acad. Sci. USA. 51, 653–659.Google Scholar
Jewell, W. S. (1963) Markov renewal programming, I: formulation, finite return models. Operat. Res. 11, 938–948.CrossRefGoogle Scholar
Kemeny, J. G., Snell, J. L. and Knapp, A. W. (1966) Denumerable Markov chains. Van Nostrand, Princeton.Google Scholar
Kesten, H. (1962) Occupation times for Markov and semi-Markov chains. Trans. Amer. Math. Soc. 103, 82–112.Google Scholar
Lévy, P. (1954) Processus semi-Markoviens. Proc. Int. Congress Math. (Amsterdam) 3, 416–426.Google Scholar
McLean, R. A. and Neuts, M. F. (1967) The integral of a step function defined on a semi-Markov process. SIAM J. Appl. Math. 15, 726–738.CrossRefGoogle Scholar
Miller, H. D. (1961) A convexity property in the theory of random variables defined on a finite Markov chain. Ann. Math. Statist. 32, 1260–1270.Google Scholar
Moore, E. H. and Pyke, R. (1964) Estimation of the transition distributions of a Markov renewal process. Boeing Sci. Res. Labs., Tech. Rep. D1–82–0371.Google Scholar
Neuts, M. F. (1964) Generating functions for Markov renewal processes. Ann. Math. Statist. 35, 431–434.Google Scholar
Neuts, M. F. (1966) The single server queue with Poisson input and semi-Markov service times. J. Appl. Prob. 3, 202–230.CrossRefGoogle Scholar
Pyke, R. (1958) On renewal processes related to type I and type II counter models. Ann. Math. Statist. 29, 737–754.CrossRefGoogle Scholar
Pyke, R. (1961a) Markov renewal processes: definitions and preliminary properties. Ann. Math. Statist. 32, 1231–1242.CrossRefGoogle Scholar
Pyke, R. (1961b). Markov renewal processes with finitely many states. Ann. Math. Statist. 32, 1243–1259.CrossRefGoogle Scholar
Pyke, R. (1962) Markov renewal processes of zero order and their application to counter theory. In Studies in Applied Probability and Management Science, Arrow, , Scarf, , Karlin, , Editors, Stanford University Press, Stanford.Google Scholar
Pyke, R. and Schaufele, R. A. (1964) Limit theorems for Markov renewal processes. Ann. Math. Statist. 35, 1746–1764.CrossRefGoogle Scholar
Pyke, R. and Schaufele, R. A. (1966) The existence and uniqueness of stationary measures for Markov renewal processes. Ann. Math. Statist. 37, 1439–1462.Google Scholar
Schaufele, R. A. (1966) Potentials for recurrent Markov renewal processes. J. Math. Anal. Appl. 13, 303–336.Google Scholar
Smith, W. L. (1954) Asymptotic renewal theorems. Proc. Roy. Soc. Edinburgh Sect. A, 64, 9–48.Google Scholar
Smith, W. L. (1955) Regenerative stochastic processes. Proc. Roy. Soc. London, Ser. A, 232, 6–31.Google Scholar
Smith, W. L. (1958) Renewal theory and its ramifications. J. R. Statist. Soc. B, 20, 243–302.Google Scholar
Smith, W. L. (1961) Remarks on the paper ‘Regenerative stochastic processes’. Proc. Roy. Soc. London, Ser. A, 256, 496–501.Google Scholar
Smith, W. L. (1962) On necessary and sufficient conditions for the convergence of the renewal density. Trans. Amer. Math. Soc. 104, 79–100.Google Scholar
Smith, W. L. (1965) Some peculiar semi-Markov processes. Proc. Fifth Berkeley Symp. Math. Statist. Prob., vol. 2, part 2, 255–263.Google Scholar
Taga, Y. (1963) On the limiting distributions in Markov renewal processes with finitely many states. Ann. Inst. Statist. Math. 15, 1–10.Google Scholar
Takács, L. (1954) Some investigations concerning recurrent stochastic processes of a certain type. Magyar Tud. Akad. Mat. Kutatô Int. Közl. 3, 115–128.Google Scholar
Takács, L. (1956) On secondary stochastic processes generated by recurrent processes. Acta Math. Acad. Sci. Hung. 7, 17–29.Google Scholar
Takács, L. (1957) On certain sojourn time problems in the theory of stochastic processes. Acta Math. Acad. Sci. Hung. 8, 169–191.CrossRefGoogle Scholar
Takács, L. (1962) An Introduction to Queueing Theory. New York, Oxford University Press.Google Scholar
Volkov, I. S. (1958) On the distribution of sums of random variables defined on a homogeneous Markov chain with a finite number of states. Teor. Veroyat. Primen. 3, 413–429. (In Russian).Google Scholar
Weiss, G. H. and Maradudin, A. A. (1962) Some problems in traffic delay. Operat. Res. 10, 74–104.CrossRefGoogle Scholar
Yackel, J. (1966) Limit theorems for semi-Markov processes. Trans. Amer. Math. Soc. 123, 402–424.CrossRefGoogle Scholar
Yackel, J. (1967) A random time change relating semi-Markov and Markov processes. Ann. Math. Statist. 93, 358–364.Google Scholar