Accelerated Continuous-Time Approximate Dynamic Programming via Data-Assisted Hybrid Control
Daniel E. Ochoa, Jorge I. Poveda
We introduce a new closed-loop architecture for the online solution of
approximate optimal control problems in the context of continuous-time systems.
Specifically, we introduce the first algorithm that incorporates dynamic
momentum in actor-critic structures to control continuous-time dynamic plants
with an affine structure in the input. By incorporating dynamic momentum in our
algorithm, we are able to accelerate the convergence properties of the
closed-loop system, achieving superior transient performance compared to
traditional gradient-descent based techniques. In addition, by leveraging the
existence of past recorded data with sufficiently rich information properties,
we dispense with the persistence of excitation condition traditionally imposed
on the regressors of the critic and the actor. Given that our continuous-time
momentum-based dynamics also incorporate periodic discrete-time resets that
emulate restarting techniques used in the machine learning literature, we
leverage tools from hybrid dynamical systems theory to establish asymptotic
stability properties for the closed-loop system. We illustrate our results with
a numerical example.