Policy Gradient for Continuous Time and Space Reinforcement Learning

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and more.Subscribe

CC BY

Source arXiv

Computational Engineering

Machine Learning

Computational Finance

Portfolio Management

Policy Gradient for Continuous Time and Space Reinforcement Learning

Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms

Policy gradient for reinforcement learning is studied in continuous time and space under the regularized exploratory formulation developed by wang et al.

We represent the gradient of the value function with respect to a given parameterized stochastic policy as the expected integration of an auxiliary running reward function that can be evaluated using samples and the current value function.

This effectively turns policy gradient into a policy evaluation (pe)problem, enabling us to apply the martingale approach recently developed by jia and zhou (2021) for pe to solve our policy gradient problem.

We propose two types of the actor-critic algorithms for reinforcement learning, where we learn and update value functions and policies simultaneously and alternatingly.

The first type is based directly on the aforementioned representation which involves future trajectories and hence is offline.

The second type, designed for online learning, employs the first-order condition of the policy gradient and turns it into martingale orthogonality conditions.

These conditions are then incorporated using stochastic approximation when updating policies.

Finally, we demonstrate the algorithms by simulations in two concrete examples.

Authors

Yanwei Jia, Xun Yu Zhou

◐ Latest Activity

◐ Recommended Follows