Search

MiS Preprint Repository

Delve into the future of research at MiS with our preprint repository. Our scientists are making groundbreaking discoveries and sharing their latest findings before they are published. Explore repository to stay up-to-date on the newest developments and breakthroughs.

MiS Preprint
22/2016

Geometry and Determinism of Optimal Stationary Control in Partially Observable Markov Decision Processes

Guido Montúfar, Keyan Ghazi-Zahedi and Nihat Ay

Abstract

It is well known that for any finite state Markov decision process (MDP) there is a memoryless deterministic policy that maximizes the expected reward. For partially observable Markov decision processes (POMDPs), optimal memoryless policies are generally stochastic. We study the expected reward optimization problem over the set of memoryless stochastic policies. We formulate this as a constrained linear optimization problem and develop a corresponding geometric framework. We show that any POMDP has an optimal memoryless policy of limited stochasticity, which allows us to reduce the dimensionality of the search space. Experiments demonstrate that this approach enables better and faster convergence of the policy gradient on the evaluated systems.

Received:
Mar 3, 2016
Published:
Mar 3, 2016
MSC Codes:
93E20, 90C40
Keywords:
MDP, POMDP, partial observability, memoryless stochastic policy, average reward, policy gradient, Reinforcement Learning

Related publications

Preprint
2015 Repository Open Access
Guido Montúfar, Keyan Ghazi-Zahedi and Nihat Ay

Geometry and determinism of optimal stationary control in partially observable Markov decision processes