Open Access

Modes of Automated Driving System Scenario Testing: Experience Report and Recommendations

Journal Article
2020-01-1204
ISSN: 2641-9637, e-ISSN: 2641-9645
Published April 14, 2020 by SAE International in United States
Modes of Automated Driving System Scenario Testing: Experience Report and Recommendations
Sector:
Citation: Antkiewicz, M., Kahn, M., Ala, M., Czarnecki, K. et al., "Modes of Automated Driving System Scenario Testing: Experience Report and Recommendations," SAE Int. J. Adv. & Curr. Prac. in Mobility 2(4):2248-2266, 2020, https://doi.org/10.4271/2020-01-1204.
Language: English

References

  1. Czarnecki, K. , “Operational World Model Ontology for Automated Driving Systems - Part 1: Road Structure,” Waterloo Intelligent Systems Engineering Lab (WISE) Report, University of Waterloo, 2018, http://doi.org/10.13140/RG.2.2.15521.30568.
  2. Czarnecki, K. , “Operational World Model Ontology for Automated Driving Systems - Part 2: Road Users, Animals, Other Obstacles, and Environmental Conditions,” Waterloo Intelligent Systems Engineering Lab (WISE) Report, University of Waterloo, 2018, http://doi.org/10.13140/RG.2.2.11327.00165.
  3. Aptiv, Audi, Apollo, BMW, Continental, Daimler, FCA, Here, Infineon, Intel, Volkswagen , “Safety First for Automated Driving,” 2019, https://newsroom.intel.com/wp-content/uploads/sites/11/2019/07/Intel-Safety-First-for-Automated-Driving.pdf.
  4. WISE Lab , “Autonomoose,” https://uwaterloo.ca/wise-lab/autonomoose, accessed Jan. 2020.
  5. Singh, S. , “Critical Reasons for Crashes Investigated in the National Motor Vehicle Crash Causation Survey,” National Highway Traffic Safety Administration, Feb. 2015, https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812115.
  6. Winner, H., Lemmer, K., Form, T., and Mazzega, J. , “PEGASUS-First Steps for the Safe Introduction of Automated Driving,” Road Vehicle Automation 5, Lecture Notes in Mobility, Springer, 2019.
  7. “Deliverable D1.0: Final Project Results,” AdaptIVe, 2017, http://www.adaptive-ip.eu/files/adaptive/content/downloads/AdaptIVe-SP1-v1-0-DL-D1-0-Final_Report.pdf, accessed Jan. 2020.
  8. Thorn, E., Kimmel, S., and Chaka, M. , “A Framework for Automated Driving System Testable Cases and Scenarios,” National Highway Traffic Safety Administration, Blacksburg, 2018, https://rosap.ntl.bts.gov/view/dot/38824.
  9. National Highway Traffic Safety Administration , “Removing Regulatory Barriers for Vehicles with Automated Driving Systems,” 2018, https://www.federalregister.gov/documents/2019/05/28/2019-11032/removing-regulatory-barriers-for-vehicles-with-automated-driving-systems.
  10. ISO , “Road Vehicles - Safety of the Intended Functionality,” ISO/PAS 21448:2019, 2019, https://www.iso.org/standard/70939.html.
  11. Koopman, P., Ferrell, U., and Wagner, M. , “A Safety Standard Approach for Fully Autonomous Vehicles,” in Proceedings of WAISE, Computer Safety, Reliability, and Security, Springer, 2019.
  12. Li, L., Huang, W., Liu, Y., Zheng, N. et al. , “Intelligence Testing for Autonomous Vehicles: A New Approach,” IEEE Transactions on Intelligent Vehicles 1(2):158-166, 2016, doi:10.1109/TIV.2016.2608003.
  13. Huang, W., Wang, K., Yisheng Lv, Y., and Zhu, F. , “Autonomous Vehicles Testing Methods Review,” in IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, 163-168, 2016, doi:10.1109/ITSC.2016.7795548.
  14. Czarnecki, K. , “WISE Drive: Requirements Analysis Framework for Automated Driving Systems,” 2018, https://uwaterloo.ca/wise-lab/projects/wise-drive-requirements-analysis-framework-automated-driving, accessed Jan. 2020.
  15. Bender, P., Ziegler, J., and Stiller, C. , “Lanelets: Efficient Map Representation for Autonomous Driving,” in IEEE Intelligent Vehicles Symposium Proceedings, Dearborn, MI, 420-425, 2014.
  16. Open Street Map (OSM) , https://www.openstreetmap.org.
  17. Queiroz, R., Berger, T., and Czarnecki, K. , “GeoScenario: An Open DSL for Autonomous Driving Scenario Representation,” in IEEE Intelligent Vehicles Symposium (IV), Paris, 2019.
  18. Queiroz, R. , “GeoScenario User’s Manual,” 2019, https://geoscenario.readthedocs.io/, accessed Jan. 2020.
  19. Scenario Definitions from “Modes of Automated Driving System Scenario Testing: Experience Report and Recommendations,” 2020, https://git.uwaterloo.ca/wise-lab/modes-of-ads-scenario-testing-scenarios.
  20. Dillen, N., Ilievski, M., Law, E., Nacke, L. et al. , “Keep Calm and Ride Along: Passenger Comfort and Anxiety as Physiological Responses to Autonomous Driving Styles,” in CHI, 2020, doi:10.1145/3313831.3376247.
  21. ISO , “Intelligent Transport Systems - Low-Speed Automated Driving (LSAD) Systems for Limited Operational Design Domains - Performance Requirements, System Requirements and Performance Test Procedures,” ISO/AWI 22737, Under Development.
  22. National Highway Traffic Safety Administration , “Traffic Jam Assist System Confirmation Test,” Working Draft, National Highway Traffic Safety Administration, Washington, DC, June 2019, https://rosap.ntl.bts.gov/view/dot/41890/dot_41890_DS1.pdf.
  23. WISE Sim , 2020, https://uwaterloo.ca/waterloo-intelligent-systems-engineering-lab/projects/wise-sim, accessed Jan. 2020.
  24. Van Gennip, M. , “Vehicle Dynamic Modelling and Parameter Identification for an Autonomous Vehicle,” Master’s thesis, University of Waterloo, 2018, http://hdl.handle.net/10012/14260.
  25. Hosking, B.A. , “Modelling and Model Predictive Control of Power-Split Hybrid Powertrains for Self-Driving Vehicles,” Master’s thesis, University of Waterloo, 2018, http://hdl.handle.net/10012/14094.
  26. Alhaija, H.A., Mustikovela, S.K., Mescheder, L., Geiger, A. et al. , “Augmented Reality Meets Computer Vision,” International Journal of Computer Vision (IJCV), 2018, https://avg.is.tuebingen.mpg.de/publications/alhaija2018ijcv.
  27. Li, W., Pan, C., Zhang, R., Ren, J. et al. , “AADS: Augmented Autonomous Driving Simulation Using Data-driven Algorithms,” arXiv:1901.07849, https://ui.adsabs.harvard.edu/abs/2019arXiv190107849L/.
  28. Koduri, T., Bogdoll, D., Paudel, S., and Sholingar, G. , “AUREATE: An Augmented Reality Test Environment for Realistic Simulations,” SAE Technical Paper 2018-01-1080 , 2018, doi:https://doi.org/10.4271/2018-01-1080. https://saemobilus.sae.org/content/2018-01-1080/.
  29. Milgram, P. and Kishino, A.F. , “Taxonomy of Mixed Reality Visual Displays,” IEICE Transactions on Information and Systems 1321-1329, 1994.
  30. von Bernuth, A., Volk, G., and Bringmann, O. , “Simulating Photo-Realistic Snow and Fog on Existing Images for Enhanced CNN Training and Evaluation,” in IEEE Intelligent Transportation Systems Conference (ITSC), 2019.
  31. Fang, J. et al. , “Augmented Lidar Simulator for Autonomous Driving,” arXiv preprint, arXiv:1811.07112, 2019.
  32. Feng, Y., Yu, C., Xu, S., Liu, H.X. et al. , “An Augmented Reality Environment for Connected and Automated Vehicle Testing and Evaluation,” in 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, 1549-1554, 2018, doi:10.1109/IVS.2018.8500545.
  33. Khan, S. , “Towards Synthetic Dataset Generation for Semantic Segmentation Networks,” Master’s thesis, University of Waterloo, 2019, https://uwspace.uwaterloo.ca/handle/10012/15128.
  34. Waymo , “On the Road to Fully Self-Driving: Waymo Safety Report,” 2017, https://goo.gl/7HUiew, accessed Jan. 2020.
  35. Zoox , “Merging Simulation & Reality,” https://youtu.be/otmxoK4lCNw?t=23m14s.
  36. ASAM , “Open Curved Regular Grid,” 2018, https://www.asam.net/standards/detail/opencrg/.
  37. The Atlantic , “Inside Waymo’s Secret World for Training Self-Driving Cars,” https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/, accessed Aug. 2017.
  38. International Civil Aviation Organization , “Manual of Criteria for the Qualification of Flight Simulation Training Devices - Volume I - Aeroplanes (9625-1),” Fourth Edition, 2015, https://store.icao.int/products/manual-of-criteria-for-the-qualification-of-flight-simulation-training-devices-volume-i-aeroplanes-9625-1.
  39. Gechter, F., Dafflon, B., Gruer, P., and Koukam, A. , “Towards a Hybrid Real/Virtual Simulation of Autonomous Vehicles for Critical Scenarios,” in the Sixth International Conference on Advances in System Simulation (SIMUL 2014), 2014.
  40. Quinlan, M., Au, T.-C., Zhu, J., Stiurca, N. et al. , “Bringing Simulation to Life: A Mixed Reality Autonomous Intersection,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2010, doi:10.1109/iros.2010.5651993.
  41. “The European New Car Assessment Programme,” https://www.euroncap.com/.
  42. Knauss, A., Schröder, J., Berger, C., and Eriksson, H. , “Paving the Roadway for Safety of Automated Vehicles: An Empirical Study on Testing Challenges,” in IEEE Intelligent Vehicles Symposium, 1873-1880, 2017, doi:10.1109/IVS.2017.7995978.
  43. Koopman, P. and Wagner, M. , “Toward a Framework for Highly Automated Vehicle Safety Validation,” SAE Technical Paper 2018-01-1071 , 2018, doi:https://doi.org/10.4271/2018-01-1071.
  44. Wang, F.-Y., Zheng, N.-N., Cao, D., Martinez, C.M. et al. , “Parallel Driving in CPSS: A Unified Approach for Transport Automation and Vehicle Intelligence,” IEEE/CAA Journal of Automatica Sinica 4(4):577-587, 2017.
  45. ASAM, “OpenSCENARIO,” https://www.asam.net/standards/detail/openscenario/.

Cited By