This content is not included in your SAE MOBILUS subscription, or you are not logged in.

Virtual Assessment of Automated Driving: Methodology, Challenges, and Lessons Learned

Journal Article
ISSN: 2574-0741, e-ISSN: 2574-075X
Published December 18, 2019 by SAE International in United States
Virtual Assessment of Automated Driving: Methodology, Challenges, and Lessons Learned
Citation: Wagner, S., Knoll, A., Groh, K., Kühbeck, T. et al., "Virtual Assessment of Automated Driving: Methodology, Challenges, and Lessons Learned," SAE Intl. J CAV 2(4):263-277, 2019,
Language: English


  1. Kröger, F. , “Das automatisierte Fahren im gesellschaftsgeschichtlichen und kulturwissenschaftlichen Kontext,” In: Autonomes Fahren (Berlin, Germany: Springer, 2015), 41-67.
  2. ERTRAC - Automated Driving Roadmap , European Road Transport Research Advisory Council, Tech. Rep., Jul. 2015.
  3. Prodhan, G. , “BMW Says Self-Driving Car to be Level 5 Capable by 2021,” Mar. 2017. [Online], Available:
  4. Lambert, F. , “Tesla CEO Elon Musk: ‘Self-Driving will Encompass all Modes of Driving by the End of Next Year,” Mar. 2018. [Online]. Available:
  5. SAE International , “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Tech. Standard J3016, Sep. 2016.
  6. ISO/DIS 26262-1 , “Road Vehicles - Functional Safety,” 2011.
  7. Wachenfeld, W. and Winner, H. , “Die Freigabe des Autonomen Fahrens,” In: Autonomes Fahren (Berlin, Germany: Springer, 2015), 439-464, doi:10.1007/978-3-662-45854-9_21.
  8. Schuldt, F., Saust, F., Lichte, B., Maurer, M. et al. , “Effiziente systematische Testgenerierung für Fahrerassistenzsysteme in virtuellen Umgebungen,” Automatisierungssysteme, Assistenzsysteme und Eingebettete Systeme Für Transportmittel, 2013.
  9. Amersbach, C. and Winner, H. , “Functional Decomposition: An Approach to Reduce the Approval Effort for Highly Automated Driving,” in 8. Tagung Fahrerassistenz, 2017.
  10. Eckstein, L. and Zlocki, A. , “Safety Potential of ADAS-Combined Methods for an Effective Evaluation,” in 23rd International Technical Conference on the Enhanced Safety of Vehicles (ESV), Seoul, South Korea , 2013.
  11. “Self-Driving Safety Report,” General Motors Company, Detroit, Tech. Rep., 2018. [Online]. Available:
  12. Helmer, T., Kompaß, K., Wang, L., Kühbeck, T. et al. , “Safety Performance Assessment of Assisted and Automated Driving in Traffic: Simulation as Knowledge Synthesis,” In: Automated Driving (Cham, Switzerland: Springer, 2017), 473-494.
  13. Tatar, M. and Mauss, J. , “Systematic Test and Validation of Complex Embedded Systems,” in ERTS-2014 , Toulouse , 2014, 05-07.02.
  14. Groh, K., Kuehbeck, T., Schiementz, M., and Chibelushi, C. , “Towards a Scenario-Based Assessment Method for Highly Automated Driving Functions,” in 8th Conference on Driver Assistance , Munich, Sep. 2017.
  15. Murray-Smith, D.J. , Testing and Validation of Computer Simulation Models (Cham, Switzerland: Springer, 2015), doi:10.1007/978-3-319-15099-4.
  16. Notz, D., Sigl, M., Kühbeck, T., Wagner, S. et al. , “Methods for Improving the Accuracy of the Virtual Assessment of Autonomous Driving,” in Proceedings of the 8th International Conference on Connected Vehicles and Expo 2019, Graz, Austria, IEEE, Nov. 2019.
  17. Ulbrich, S., Menzel, T., Reschka, A., Schuldt, F. et al. , “Defining and Substantiating the Terms Scene, Situation, and Scenario for Automated Driving,” in Intelligent Transportation Systems (ITSC), 2015 IEEE 18th International Conference on. IEEE, Gran Canaria, Spain, 2015, 982-988.
  18. Aeberhard, M., Rauch, S., Bahram, M., Tanzmeister, G. et al. , “Experience, Results and Lessons Learned from Automated Driving on Germany’s Highways,” IEEE Intelligent Transportation Systems Magazine 7(1):42-57, 2015.
  19. Girardin, G. and Mounier, E. , “Sensors and Data Management for Autonomous Vehicles,” Yole Développement, Tech. Rep., 2015.
  20. Thomas, J. and Rojas, R. , “Sensor-Based Road Model Estimation for autonomous driving,” in 2017 IEEE Intelligent Vehicles Symposium (IV) , IEEE, Redondo Beach, CA, 2017, 1764-1769.
  21. Ziegler, J., Werling, M., and Schroder, J. , “Navigating Car-Like Robots in Unstructured Environments Using an Obstacle Sensitive Cost Function,” in 2008 IEEE Intelligent Vehicles Symposium , IEEE, Eindhoven, The Netherlands, 2008, 787-791.
  22. Rakowski, T. , “Informationstheoretische Änderungserkennung von hochgenauen Straßenmodellen als Grundlage für automatisierte Fahrfunktionen,” Master’s thesis, Freie Universität Berlin, Jul. 2013.
  23. Aeberhard, M. and Kaempchen, N. , “High-Level Sensor Data Fusion Architecture for Vehicle Surround Environment Perception,” in Proc. 8th Int. Workshop Intell. Transp, Las Vegas, NV, 2011.
  24. Banerjee, K., Notz, D., Windelen, J., Gavarraju, S.N. et al. , “Online Camera LiDAR Fusion and Object Detection on Hybrid Data for Autonomous Driving,” in 2018 IEEE Intelligent Vehicles Symposium, IV 2018, Changshu, China, June 26-29, 2018 , IEEE, 2018.
  25. Bock, T. , Vehicle in the loop: Test-und Simulationsumgebung für Fahrerassistenzsysteme (Göttingen, Germany: Cuvillier Verlag, 2008).
  26. Groh, K., Wagner, S., Kuehbeck, T., and Knoll, A. , “Simulation and its Contribution to evaluate Highly Automated Driving Functions,” in System Safety and Verification of Automated Driving Systems & ADAS, SAE WCX 2019, Detroit, SAE International, Apr. 2019, 11.
  27. Wagner, S., Groh, K., Kuehbeck, T., Doerfel, M. et al. , “Using Time-To-React Based on Naturalistic Traffic Object Behavior for Scenario-Based Risk Assessment of Automated Driving,” in Proceedings of the IEEE Intelligent Vehicle Symposium 2018 , Changshu, China, IEEE, Jun. 2018, 1521-1528.
  28. “OxTS RT3000 Datasheet,” Tech. Rep. [Online]. Available:
  29. Dupuis, M., Strobl, M., and Grezlikowski, H. , “OpenDRIVE 2010 and Beyond-Status and Future of the de facto Standard for the Description of Road Networks,” in Proceedings of the Driving Simulation Conference DSC Europe , Paris , 2010, 231-242.
  30. Hanke, T. , “Open Simulation Interface - Introduction and Overview,” Oct. 2017. [Online]. Available:
  31. Wagner, S., Groh, K., Kühbeck, T., and Knoll, A. , “Towards Cross-Verification and Use of Simulation in the Assessment of Automated Driving,” in Processings of the IEEE Intelligent Vehicle Symposium 2019 , Paris, France, IEEE, Jun. 2019, 1407-1414.
  32. Hillenbrand, J., Spieker, A.M., and Kroschel, K. , “A multilevel collision mitigation approach-Its situation assessment, decision making, and performance tradeoffs,” IEEE Transactions on Intelligent Transportation Systems 7(4):528-540, 2006.
  33. Wachenfeld, W., Junietz, P., Wenzel, R., and Winner, H. , “The Worst-Time-to-Collision Metric for Situation Identification,” in 2016 IEEE Intelligent Vehicles Symposium (IV) , Gothenburg, Sweden, Jun. 2016, 729-734.
  34. Kessler, C. and Etemad, A. , “European Large-Scale Field Operational Tests on In-Vehicle Systems - SP 6 D6.8 FOT Data,” Ford Research & Advanced Engineering Europe, Tech. Rep., Jun. 2012.
  35. Rocklage, E., Kraft, H., Karatas, A., and Seewig, J. , “Automated Scenario Generation for Regression Testing of Autonomous Vehicles,” in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) , Yokohama, Japan, Oct. 2017, 476-483.
  36. Abdessalem, R., Nejati, S., Briand, L., and Stifter, T. , “Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms,” in Proceedings—International Conference on Software Engineering , vol. Part F137142, Gothenburg, Sweden, 2018, 1016-1026.
  37. Sippl, C., Bock, F., Wittmann, D., Altinger, H. et al. , “From Simulation Data to Test Cases for Fully Automated Driving and ADAS,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9976 LNCS, 191-206, 2016.
  38. Andrews, A., Abdelgawad, M., and Gario, A. , “Towards World Model-Based Test Generation in Autonomous Systems,” in 2015 3rd International Conference on Model-Driven Engineering and Software Development (MODELSWARD) , Angers, France, Feb. 2015, 1-12.
  39. Krajewski, R., Bock, J., Kloeker, L., and Eckstein, L. , “The High D Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems,” in 2018 IEEE 21th International Conference on Intelligent Transportation Systems (ITSC) , Maui, HI, 2018.
  40. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V. et al. , “Scikit-Learn: Machine Learning in Python,” Journal of Machine Learning Research 12:2825-2830, 2011.
  41. Anguelov, D. , “MIT Self-Driving Cars,” Feb. 2019. [Online]. Available:
  42. Dobberstein, J., Bakker, J., Wang, L., Vogt, T. et al. , “The Eclipse Working Group openPASS-An Open Source Approach to Safety Impact Assessment via Simulation,” in 25th International Technical Conference on the Enhanced Safety of Vehicles (ESV) National Highway Traffic Safety Administration , Detroit, MI, 2017.
  43. “ALP.Lab GmbH - Austrian Light Vehicle Proving Region for Automated Driving.” [Online]. Available:
  44. “Providentia - Die intelligente Autobahn auf dem Digitalen Testfeld A9.” [Online]. Available:

Cited By