Fast Algorithms and Performance Guarantee for Safe Reinforcement Learning

Reinforcement learning (RL) has achieved tremendous success in solving sequential decision making problems, typically with the aim of finding a policy that maximizes an expected total reward. In practice, it is often critical to further ensure the satisfaction of various safety constraints (for example, a robot working in a warehouse should not hit the arm on the shelf), which thus motivates an emerging field of safe RL. Despite the empirical success, existing safe RL algorithms do not achieve the best possible convergence rate, and often fail to converge to the globally optimal policy.

In this talk, I will present our recent studies that provably improve the existing two major safe RL approaches. I will first present a new primal-dual safe RL algorithm and show that such an algorithm attains a globally optimal policy and improves the best known computational complexity of the existing algorithms. I will then present another new primal-type safe RL algorithm and further provide the characterization of its global optimality guarantee and convergence rate. Our experiments demonstrate that the primal safe RL algorithm substantially outperforms the primal-dual algorithm. I will conclude my talk with further intuition about the comparison between these two approaches and future directions on the topic.