This content is not included in your SAE MOBILUS subscription, or you are not logged in.
Automated Driving System Safety: Miles for 95% Confidence in “Vision Zero”
ISSN: 0148-7191, e-ISSN: 2688-3627
Published April 14, 2020 by SAE International in United States
This content contains downloadable datasetsAnnotation ability available
Engineering reliability models from RAND, MobilEye, and Volvo concluded that billions of miles of on-road data were required to validate that the real-world fatality rate of an “Automated Driving System-equipped vehicle” (AV) fleet for an improvement over human-driven conventional vehicles (CV). RAND said 5 billion miles for 20%, MobileEye 30 billion for 99.9%, and Volvo 5 billion for 50% improvement. All these models used the Gaussian distribution, which is inaccurate for low crash numbers. The current study proposes a new epidemiologic method and criterion to validate real-world AV data with 95% confidence for zero to ten fatal crashes. The upper confidence limit (UL) of the AV fatal crash rate has to be lower than the CV fatal crash rate with 95% confidence. That criterion is met if the UL of the AV fatal crash incidence rate ratio estimate is below one. That UL was estimated using the mid-P exact method for calculating confidence limits for a dual Poisson process, using a one-tailed 95% confidence level. The required AV mileage was adjusted by trial and error until the UL was just below 1. The method was applied to the real-world fatal crash and mileage data from California (Waymo and GM Cruise), U.S. (Uber and Tesla), China (Tesla), and Sweden (Volvo) to calculate the AV miles required to validate a “Vision Zero” AV fatal crash objective with 95% confidence. For AV fleets in California, U.S., and China, the AV miles required are only in the millions. In Sweden, 1.9 billion AV miles are required because its CV fatal crash rates are already so low. In conclusion, epidemiologic analysis of real-world data finds that only millions of real-world AV data are typically required to validate AV safety with 95% confidence to a “Vision Zero” fatal crash criterion.
CitationYoung, R., "Automated Driving System Safety: Miles for 95% Confidence in “Vision Zero”," SAE Technical Paper 2020-01-1205, 2020, https://doi.org/10.4271/2020-01-1205.
Data Sets - Support Documents
|[Unnamed Dataset 1]|
|[Unnamed Dataset 2]|
|[Unnamed Dataset 3]|
|[Unnamed Dataset 4]|
|[Unnamed Dataset 5]|
|[Unnamed Dataset 6]|
|[Unnamed Dataset 7]|
|[Unnamed Dataset 8]|
|[Unnamed Dataset 9]|
|[Unnamed Dataset 10]|
|[Unnamed Dataset 11]|
|[Unnamed Dataset 12]|
|[Unnamed Dataset 13]|
|[Unnamed Dataset 14]|
|[Unnamed Dataset 15]|
|[Unnamed Dataset 16]|
|[Unnamed Dataset 17]|
- Kalra, N. and Paddock, S.M. , “Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?” Transportation Research Part A: Policy and Practice 94:182-193, 2016, doi:https://doi.org/10.1016/j.tra.2016.09.010.
- Shalev-Shwartz, S., Shammah, S. and Shashua, A. , “On a Formal Model of Safe and Scalable Self-Driving Cars,” arXiv preprint arXiv:1708. (Young et al., 201506374 [version 6], https://arxiv.org/abs/1708.06374, Oct. 27, 2018.
- Lindman, M., Isaksson-Hellman, I., and Strandroth, J. , “Basic Numbers Needed to Understand the Traffic Safety Effect of Automated Cars,” in Proceedings of IRCOBI 2017 Conference, 2017.
- Rothman, K., Greenland, S., and Lash, T. , Modern Epidemiology Third Edition (Philadelphia, PA: Lippincott Williams & Wilkins, 2008).
- Lord, D., Washington, S.P., and Ivan, J.N. , “Poisson, Poisson-Gamma and Zero-Inflated Regression Models of Motor Vehicle Crashes: Balancing Statistical Fit and Theory,” Accident Analysis & Prevention 37(1):35-46, 2005.
- Knipling, R. and Wåhlberg, A.E. , “The Heterogeneity Principle,” in Proceedings of the Tenth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Santa Fe, New Mexico, 2019.
- Greenland, S. , “A Counterexample to the Test-Based Principle of Setting Confidence Limits,” American Journal of Epidemiology 120(1):4-7, 1984.
- Waymo , “Waymo Safety Report: On the Road to Fully Self-Driving,” Mountain View, CA, 2018.
- GM , “Self-Driving Safety Report,” General Motors, Detroit, 2018, https://www.gm.com/content/dam/company/docs/us/en/gmcom/gmsafetyreport.pdf.
- NHTSA , “Voluntary Safety Self-Assessment,” 2018, https://www.nhtsa.gov/automated-driving-systems/voluntary-safety-self-assessment.
- https://en.wikipedia.org › wiki › Vision_Zero.
- Young, R.A. , “Epidemiological Study Designs Applied to Driving Safety: Definitions and Examples,” SAE Technical Paper 2018-01-0496, 2018, doi:10.4271/2018-01-0496.
- StatsDirect Statistical Software , Version 3.2.8, Sept. 16, 2019.
- OpenEpi , Version 3.01, Updated Apr. 6, 2013, https://www.openepi.com/PersonTime2/PersonTime2.htm.
- Stata , Version 13.1, https://www.stata.com/.
- California Vehicle Code (CVC) , Section 38750, https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?lawCode=VEH§ionNum=38750.
- SAE J-3016 , Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles (Warrendale, PA: SAE International, 2018).
- CA DMV, 2018. State of California Department of Motor Vehicles. Reports of Traffic Accidents Involving an Autonomous Vehicle (OL 316). Available from https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/autonomousveh_ol316+. Note: The California DMV has removed all OL-316 reports from its autonomous vehicle directory prior to January 1, 2019. They state, “Email AVarchive@dmv.ca.gov to request a digital copy of an archived report. Requests must include the year and manufacturer.”
- Autonomous Vehicle Disengagement Reports 2015-2018. See https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/testing. Note: The California DMV has removed all disengagement reports from its autonomous vehicle directory prior to January 1, 2019. They state, “Email AVarchive@dmv.ca.gov to request a digital copy of an archived report. Requests must include the year and manufacturer.”
- SWITRS , “California Highway Patrol Statewide Integrated Traffic Records System,” http://iswitrs.chp.ca.gov/Reports/jsp/RawData.jsp.
- Transportation Injury Mapping System (TIMS) , “Safe Transportation Research and Education Center,” University of California, Berkeley, 2019, https://tims.berkeley.edu/help/SWITRS.php.
- HPMS , https://dot.ca.gov/programs/research-innovation-system-information/highway-performance-monitoring-system.
- Young, R.A. , “Prototype Autonomous Vehicle Crash Rates: A Critical Review of the Scientific Literature,” PLOS-ONE, submitted.
- NHTSA , “Traffic Safety Facts 2013 Data,” National Highway Traffic Safety Administration, Department of Transportation, Washington, DC, 2015, https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812169.
- Walker, J. , “The Self-Driving Car Timeline - Predictions from the Top 11 Global Automakers,” 2019, https://emerj.com/ai-adoption-timelines/self-driving-car-timeline-themselves-top-11-automakers/.
- Liu, P., Yang, R., and Xu, Z. , “How Safe Is Safe Enough for Self-Driving Vehicles?” Risk Analysis 39(2):315-325, 2019.
- Young, R.A., Aryal, B., Muresan, M., Ding, X. et al. , “Road-to-Lab: Validation of the Static Load Test for Predicting on-Road Driving Performance While Using Advanced Information and Communication Devices,” in Proceedings of the Third International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Rockport, Maine, June 2005, https://drivingassessment.uiowa.edu/sites/drivingassessment.uiowa.edu/files/wysiwyg_uploads/35_dickyoungformat.pdf.
- Young, R.A., Hsieh, L., and Seaman, S. , “The Tactile Detection Response Task: Preliminary Validation for Measuring the Attentional Effects of Cognitive Load,” in Proceedings of 7th International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, Bolton Landing, NY, June 17-20, 2013, https://drivingassessment.uiowa.edu/sites/drivingassessment.uiowa.edu/files/wysiwyg_uploads/012_young_0.pdf.
- Young, R.A. , “Predicting Relative Crash Risk from the Attentional Effects of the Cognitive Demand of Visual-Manual Secondary Tasks,” SAE Technical Paper 2017-01-1384, 2017, doi:https://doi.org/10.4271/2017-01-1384.
- Shladover, S.E. and Nowakowski, C. , “Regulatory Challenges for Road Vehicle Automation: Lessons from the California Experience,” Transportation Research Part A: Policy and Practice, 2017.
- Claybrook, J. and Kildare, S. , “Autonomous Vehicles: No Driver… No Regulation?” Science 361(6397):36-37, 2018.
- Fusheng, L. “Country Issues National Standards for Autonomous Vehicle Testing,” China Daily, Aug. 13, 2018, http://english.www.gov.cn/state_council/ministries/2018/08/13/content_281476259760362.htm.
- Tesla. Tesla Model S Owner's Manual. May 16, 2019.
- Hwang, J.G. and Yang, M.-C. , “An Optimality Theory for Mid P-Values in 2 X 2 Contingency Tables,” Statistica Sinica 11(3):807-826, 2001.
- Rothman, K.J. , “Episheet: Spreadsheets for the Analysis of Epidemiologic Data,” 2015, http://krothman.org/Episheet.xls.
- Liu, H., Man, H., Cui, H., Wang, Y. et al. , “An Updated Emission Inventory of Vehicular VOCs and IVOCs in China,” Atmospheric Chemistry and Physics 17(20):12709, 2017. https://pdfs.semanticscholar.org/310d/d631ee760b262247a15082fe88dd087b0085.pdf.
- NHTSA , “Traffic Safety Facts 2015 Data,” National Highway Traffic Safety Administration, Department of Transportation, Washington, DC, 2017, https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812384.
- Traffic Management Bureau of Ministry of Public Security , “Road Traffic Accident Statistics’ Annual Report of People’s Republic of China 2015,” Traffic Management Bureau of Ministry of Public Security, Beijing, 2016 [in Mandarin].
- Hu, G., Baker, T., and Baker, S.P. , “Comparing Road Traffic Mortality Rates from Police-Reported Data and Death Registration Data in China,” Bulletin of the World Health Organization 89:41-45, 2011, doi:https://www.scielosp.org/scielo.php?pid=S0042-96862011000100011&script=sci_abstract&tlng=fr.
- Huang, H., Yin, Q., Schwebel, D.C., Li, L. et al. , “Examining Road Traffic Mortality Status in China: A Simulation Study,” PLoS One 11(4):e0153251, 2016.
- Huang, H., Yin, Q., Schwebel, D.C., Ning, P. et al. , “Availability and Consistency of Health and Non-Health Data for Road Traffic Fatality: Analysis of Data from 195 Countries, 1985-2013,” Accident Analysis & Prevention 108:220-226, 2017, doi:https://www.sciencedirect.com/science/article/abs/pii/S0001457517303135.
- Huang, H., Chang, F., Schwebel, D.C., Ning, P. et al. , “Improve Traffic Death Statistics in China,” Science 362(6415):650, Nov. 9, 2018, doi:10.1126/science.aav5117.
- World Health Organization , “Global Status Report on Road Safety 2018,”, https://www.who.int › violence_injury_prevention › 2018, 2018.
- Wang, L., Ning, P., Yin, P., Cheng, P. et al. , “Road Traffic Mortality in China: Analysis of National Surveillance Data from 2006 to 2016,” The Lancet Public Health 4(5):e245-e255, 2019.
- https://www.iihs.org/topics/fatality-statistics/detail/yearly-snapshot; IIHS (Insurance Institute for Highway Safety), “Real-Word Benefits of Crash Avoidance Technologies,” 2019, https://www.iihs.org/media/259e5bbd-f859-42a7-bd54-3888f7a2d3ef/EiyeBQ/Topics/ADVANCED%20DRIVER%20ASSISTANCE/IIHS-real-world-CA-benefits-0419.pdf.
- Cox, W. , “Average Chinese Car Travels as Much as American Car,” newgeography, 2019, https://www.newgeography.com/content/006420-average-chinese-car-travels-much-american-car.
- IRTAD (International Traffic Safety Data and Analysis Group) , “Speed and Crash Risk,” 2018, https://www.itf-oecd.org/sites/default/files/docs/speed-crash-risk.pdf.
- Porta, M. , A Dictionary of Epidemiology Sixth Edition (New York: Oxford University Press, 2014). http://irea.ir/files/site1/pages/dictionary.pdf.
- Yue, L., Abdel-Aty, M., Wu, Y., and Wang, L. , “Assessment of the Safety Benefits of Vehicles’ Advanced Driver Assistance, Connectivity and Low Level Automation Systems,” Accident Analysis & Prevention 117:55-64, 2018, doi:10.1016/j.aap.2018.04.002, available from https://www.researchgate.net/publication/324531299.