Performance Evaluation of an Autonomous Vehicle Using Resilience Engineering

Standard operation of autonomous vehicles on public roads results in significant exposure to high levels of risk. There is a significant need to develop metrics that evaluate safety of an automated system without reliance on the rate of vehicle accidents and fatalities compared to the number of miles driven; a proactive rather than a reactive metric is needed. Resilience engineering is a new paradigm for safety management that focuses on evaluating complex systems and their interaction with the environment. This paper presents the overall methodology of resilience engineering and the resilience assessment grid (RAG) as an evaluation tool to measure autonomous systems' resilience. This assessment tool was used to evaluate the ability to respond to the system. A Pure Pursuit controller was developed and utilized as the path tracking control algorithm, and the Carla simulator was used to implement the algorithm and develop the testing environment for this methodology. The path tracking control algorithm was tested at different speeds and evaluated using RAG. Simulation results show that at higher speeds the vehicle demonstrated lower overall resilience and tells us the algorithm is less susceptible to overcome disturbances. We conclude that this metric can be successfully used to proactively evaluate the safety of automated vehicle subsystems or the system's overall performance and demonstrates a clear path to improve performance. In future work, we plan on expanding our evaluation to include commercially available products such as SuperCruise, BlueCruise, and the Full Self-Driving product and sensor fusion algorithms.