Run Time Assurance Filter Framework for Collision Avoidance of Unmanned Aerial Vehicles Using Reinforcement Learning

In recent years, Reinforcement Learning (RL) has been increasingly applied to the collision avoidance systems of Unmanned Aerial Vehicles (UAVs) due to its high performance in complex tasks and relatively low online computational cost. Despite these advantages, the lack of interpretability in RL and decision-making processes poses significant challenges to safety verification, which severely limits the practical deployment of RL-based autonomous flight systems in safety-critical scenarios. Ensuring safety during both training and testing remains the key for real-world applications. To address this issue, a Run Time Assurance (RTA) filter framework for a UAV collision avoidance system based on RL is proposed. The framework continuously monitors the avoidance actions generated by RL and ensures that unsafe actions can be identified and corrected in real time. The framework provides security while retaining the performance advantages of RL. First, a discrete kinematics model of the UAV and a collision avoidance environment for a three-dimensional multi-intruder UAV are established. The Proximal Policy Optimization (PPO) algorithm is used to train the UAV avoidance policy. Secondly, the RTA filter with safe constraints is designed based on this model. To reduce the position error between the discrete kinematics model and the continuous dynamics model of the UAV, optimization is performed using the Sequential Least Squares Programming Method (SLSQP). The RTA is applied to the training and testing phases of UAV collision avoidance systems to monitor and correct unsafe actions online. Finally, three training and testing methods are set: no RTA, RTA without punishment, and RTA with punishment. Comparing different relaxation coefficients allows for analyzing their impact on filter performance. Simulation results demonstrate that the RTA effectively enhances safety throughout training and testing. Compared with the no RTA method, the collision rate is reduced by 7.04%, and the success rate is increased by 2.68%. In extreme collision avoidance scenarios, the average collision rate obtained by different reinforcement learning algorithms is reduced by 21.46%, and the average success rate is increased by 15.54%.