Hierarchical Reinforcement Learning for Continuous State and Action Spaces
Abstract Value Iteration for Hierarchical Reinforcement Learning
We propose a novel hierarchical reinforcement learning framework for control with continuous state and action spaces.In our framework, the user specifies subgoal regions which are subsets of states ; then, we (i)learn options that serve as transitions between these subgoal regions, and (ii)construct a high-level plan in the resulting abstract decision process (adp).A key challenge is that the resulting decision process may not be a markov process, which we address by proposing two algorithms for planning in the resulting abstract decision process.Our first algorithm is conservative, allowing us to prove theoretical guarantees on its performance, which help inform the design of subgoal regions.Our second algorithm is a practical one that interweaves planning at the abstract level and learning at the concrete level.We demonstrate that our approach outperforms state-of-the-art hierarchical reinforcement learning algorithms on several challenging benchmarks.
