This content is not included in
your SAE MOBILUS subscription, or you are not logged in.
Performance Evaluation of an Autonomous Vehicle Using Resilience Engineering
Technical Paper
2022-01-0067
ISSN: 0148-7191, e-ISSN: 2688-3627
Annotation ability available
Sector:
Language:
English
Abstract
Standard operation of autonomous vehicles on public roads results in significant exposure to high levels of risk. There is a significant need to develop metrics that evaluate safety of an automated system without reliance on the rate of vehicle accidents and fatalities compared to the number of miles driven; a proactive rather than a reactive metric is needed. Resilience engineering is a new paradigm for safety management that focuses on evaluating complex systems and their interaction with the environment. This paper presents the overall methodology of resilience engineering and the resilience assessment grid (RAG) as an evaluation tool to measure autonomous systems' resilience. This assessment tool was used to evaluate the ability to respond to the system. A Pure Pursuit controller was developed and utilized as the path tracking control algorithm, and the Carla simulator was used to implement the algorithm and develop the testing environment for this methodology. The path tracking control algorithm was tested at different speeds and evaluated using RAG. Simulation results show that at higher speeds the vehicle demonstrated lower overall resilience and tells us the algorithm is less susceptible to overcome disturbances. We conclude that this metric can be successfully used to proactively evaluate the safety of automated vehicle subsystems or the system's overall performance and demonstrates a clear path to improve performance. In future work, we plan on expanding our evaluation to include commercially available products such as SuperCruise, BlueCruise, and the Full Self-Driving product and sensor fusion algorithms.
Authors
Topic
Citation
Fanas Rojas, J., Brown, N., Rupp, J., Bradley, T. et al., "Performance Evaluation of an Autonomous Vehicle Using Resilience Engineering," SAE Technical Paper 2022-01-0067, 2022, https://doi.org/10.4271/2022-01-0067.Also In
References
- Brodsky , J.S. How an Uncertain Legal Landscape May Hit the Brakes On Self-Driving Cars Berkeley Technol. Law J. 31 2 2016 851 878
- Schwarting , W. , Alonso-Mora , J. , and Rus , D. Planning and Decision-Making for Autonomous Vehicles Annu. Rev. Control Robot. Auton. Syst. 1 1 2018 187 210
- Singh , S. Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey 2015
- Walker Smith , B. http://cyberlaw.Stanford.edu/blog/2013/12/human-Error-Cause-Vehicle-Crashes 2013
- Taeihagh , A. and Lim , H.S.M. Governing Autonomous Vehicles: Emerging Responses for Safety, Liability, Privacy, Cybersecurity, and Industry Risks Null 39 1 2019 103 128
- Collingwood , L. Privacy Implications and Liability Issues of Autonomous Vehicles Null 26 1 2017 32 45
- Hollnagel , E. , Woods , D.D. , and Leveson , N. Resilience Engineering: Concepts and Precepts Ashgate Publishing, Ltd. 2006 9780754681366
- Madni , A.M. , Sievers , M.W. , Humann , J. , Ordoukhanian , E. et al. Model-Based Approach for Engineering Resilient System-of-Systems: Application to Autonomous Vehicle Networks Disciplinary Convergence in Systems Engineering Research Springer International Publishing 2018 365 380
- Marshall , C. , Roberts , B. , and Grenn , M. Intelligent Control amp; Supervision for Autonomous System Resilience in Uncertain Worlds 2017 3rd International Conference on Control, Automation and Robotics (ICCAR) 2017 438 443
- Koopman , P. and Wagner , M. Autonomous Vehicle Safety: An Interdisciplinary Challenge IEEE Intell. Transp. Syst. Mag. 9 1 2017 90 96
- Michelmore , R. , Kwiatkowska , M. , and Gal , Y. arXiv [cs.LG] 2018
- Zhang , F. , Martinez , C.M. , Clarke , D. , Cao , D. et al. Neural Network Based Uncertainty Prediction for Autonomous Vehicle Application Front. Neurorobot. 13 2019 12
- Xu , S. and Peng , H. Design, Analysis, and Experiments of Preview Path Tracking Control for Autonomous Vehicles IEEE Trans. Intell. Transp. Syst. 21 1 2020 48 58
- Henaff , M. , Canziani , A. , and LeCun , Y. arXiv [cs.LG] 2019
- Cosgun , A. , Ma , L. , Chiu , J. , Huang , J. et al. Towards Full Automated Drive in Urban Environments: A Demonstration in GoMentum Station, California 2017 IEEE Intelligent Vehicles Symposium (IV) 1811 1818 2017
- Koopman , P. and Wagner , M. Challenges in Autonomous Vehicle Testing and Validation SAE International Journal of Transportation Safety 4 1 2016 15 24
- Furuta , K. Resilience Engineering Reflections on the Fukushima Daiichi Nuclear Accident 2014 435 454
- Woods , D.D. Four Concepts for Resilience and the Implications for the Future of Resilience Engineering Reliab. Eng. Syst. Saf. 141 2015 5 9
- Wang , W.-J. , Hsu , T.-M. , and Wu , T.-S. The Improved Pure Pursuit Algorithm for Autonomous Driving Advanced System 2017 IEEE 10th International Workshop on Computational Intelligence and Applications (IWCIA) 2017 33 38
- Dosovitskiy , A. , Ros , G. , Codevilla , F. , Lopez , A. et al. arXiv [cs.LG] 2017
- Goberville , N. , Zoardar , M.M. , Rojas , J. , Brown , N. et al. Techno-Economic Analysis of Fixed-Route Autonomous and Electric Shuttles SAE Technical Paper 2021-01-0061 2021 10.4271/2021-01-0061
- Stević , S. , Krunić , M. , Dragojević , M. , and Kaprocki , N. Development and Validation of ADAS Perception Application in ROS Environment Integrated with CARLA Simulator 2019 27th Telecommunications Forum (TELFOR) 1 4 2019
- Dworak , D. , Ciepiela , F. , Derbisz , J. , Izzat , I. et al. Performance of LiDAR Object Detection Deep Learning Architectures Based on Artificially Generated Point Cloud Data from CARLA Simulator 2019 24th International Conference on Methods and Models in Automation and Robotics (MMAR) 600 605 2019
- Gómez-Huélamo , C. , Del Egido , J. , Bergasa , L.M. , Barea , R. et al. Train Here, Drive There: Simulating Real-World Use Cases with Fully-Autonomous Driving Architecture in CARLA Simulator Advances in Physical Agents II Springer International Publishing 2021 44 59
- Prescinotti Vivan , G. , Goberville , N. , Asher , Z. , Brown , N. et al. No Cost Autonomous Vehicle Advancements in CARLA through ROS SAE Technical Paper 2021-01-0106 2021 10.4271/2021-01-0106
- Carballo , A. , Wong , D. , Ninomiya , Y. , Kato , S. et al. Training Engineers in Autonomous Driving Technologies Using Autoware 2019 IEEE Intelligent Transportation Systems Conference (ITSC) 3347 3354 2019
- Zhang , Y. , Sun , H. , Zhou , J. , Pan , J. et al. Optimal Vehicle Path Planning Using Quadratic Optimization for Baidu Apollo Open Platform 2020 IEEE Intelligent Vehicles Symposium (IV) 978 984 2020
- Righi , A.W. , Saurin , T.A. , and Wachs , P. A Systematic Literature Review of Resilience Engineering: Research Areas and a Research Agenda Proposal Reliab. Eng. Syst. Saf. 141 2015 142 152
- Woods , D.D. http://ordvac.com/soro/library/Aviation/Aviation%20Safety/General%20Safety%20Articles/resilience%20engineering%20bulletin.pdf 2006
- Hoffman , R.R. and Woods , D.D. Beyond Simon’s Slice: Five Fundamental Trade-Offs That Bound the Performance of Macrocognitive Work Systems IEEE Intell. Syst. 26 6 2011 67 71
- Hollnagel , E. The ETTO Principle: Efficiency-Thoroughness Trade-Off : Why Things That Go Right Sometimes Go Wrong Ashgate Publishing, Ltd. 2009 9780754676775
- Ni , H. , Chen , A. , and Chen , N. Some Extensions on Risk Matrix Approach Saf. Sci. 48 10 2010 1269 1278
- Provan , D.J. , Woods , D.D. , Dekker , S.W.A. , and Rae , A.J. Safety II Professionals: How Resilience Engineering Can Transform Safety Practice Reliab. Eng. Syst. Saf. 195 2020 106740
- Liu , L. , Lu , S. , Zhong , R. , Wu , B. et al. arXiv [cs.DC], 2020
- Kukkala , V.K. , Tunnell , J. , Pasricha , S. , and Bradley , T. Advanced Driver-Assistance Systems: A Path Toward Autonomous Vehicles IEE 2018
- Hollnagel , E. RAG-Resilience Analysis Grid Introduction to the Resilience Analysis Grid (RAG) 2015