Sufficiency of Markov policies for continuous-time Markov decision processes and solutions to Kolmogorov's forward equation for jump Markov processes

In continuous-time Markov decision processes (CTMDPs) with Borel state and action spaces, unbounded transition rates, for an arbitrary policy, we construct a relaxed Markov policy such that the marginal distribution on the state-action pairs at any time instant is the same for both the policies. This result implies the existence of a relaxed Markov policy that performs equally to an arbitrary policy with respect to expected discounted and non-discounted total costs as well as average costs per unit time. The proof consists of two steps. The first step describes the properties of solutions to Kolmogorov's forward equation for jump Markov Processes. The second step applies these results to CTMDPs.