This content is not included in your SAE MOBILUS subscription, or you are not logged in.
A Comprehensive Testing and Evaluation Approach for Autonomous Vehicles
ISSN: 0148-7191, e-ISSN: 2688-3627
Published April 03, 2018 by SAE International in United States
This content contains downloadable datasetsAnnotation ability available
Performance testing and evaluation always plays an important role in the developmental process of a vehicle, which also applies to autonomous vehicles. The complex nature of an autonomous vehicle from architecture to functionality demands even more quality-and-quantity controlled testing and evaluation than ever before. Most of the existing testing methodologies are task-or-scenario based and can only support single or partial functional testing. These approaches may be helpful at the initial stage of autonomous vehicle development. However, as the integrated autonomous system gets mature, these approaches fall short of supporting comprehensive performance evaluation. This paper proposes a novel hierarchical and systematic testing and evaluation approach to bridge the above-mentioned gap. In this paper, firstly a three-dimensional evaluation model conforming to the functional architecture of autonomous vehicles was built, with each dimension representing one of the three key functional layers of autonomous vehicle including sensing & perception, decision-making ﹠ planning, control & execution. Each dimension has a set of metrics carefully defined with their weights fairly determined based on an entropy weights method. Then, considering environment effect on vehicle functions, we innovatively determine task-scenarios for testing the performance of each dimension. Besides, we design a hierarchical systematic testing method which could specially testing each function layer of autonomous vehicle. Fuzzy comprehensive and TOPSIS evaluation method was proposed to quantitatively evaluate the comprehensive performance of autonomous vehicles under defined task-scenarios. Finally, our methods are used to evaluate three candidate vehicles based on simulation scenario in PanoSim. Compared to traditional approach based on external task performance, the proposed approach can not only provide convincing results of the overall system performance but can also peek into each of the key functional layer and provide insights about their performance. Therefore, this approach provides better guidance for autonomous vehicle research.
CitationWang, G., Deng, W., Zhang, S., Wang, J. et al., "A Comprehensive Testing and Evaluation Approach for Autonomous Vehicles," SAE Technical Paper 2018-01-0124, 2018, https://doi.org/10.4271/2018-01-0124.
Data Sets - Support Documents
|[Unnamed Dataset 1]|
|[Unnamed Dataset 2]|
|[Unnamed Dataset 3]|
|[Unnamed Dataset 4]|
- Sun, Y., Tao, G., Xiong, G., and Chen, H. , “The Fuzzy-AHP Evaluation Method for Unmanned Ground Vehicles,” Applied Mathematics & Information Sciences 7(2):653-658, March 1, 2013, doi:10.12785/amis/070232.
- Macias F. “The Test and Evaluation of Unmanned and Autonomous Systems[R]. WHITE SANDS MISSILE RANGE NM,” 2008.
- Campbell, M., Egerstedt, M., How, J.P., and Murray, R.M. , “Autonomous Driving in Urban Environments: Approaches, Lessons and Challenges,” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 368(1928):4649-4672, September 6, 2010, doi:10.1098/rsta.2010.0110.
- Schneider, F.E. and Wildermuth, D. , “Results of the European Land Robot Trial and their Usability for Benchmarking Outdoor Robot Systems,” Lecture Notes in Computer Science 408-409, 2011, doi:10.1007/978-3-642-23232-9_51.
- Lesemann M . “Testing and evaluation methods for ICT-based safety systems[C]//15th World Congress on Intelligent Transport Systems and ITS America's 2008 Annual Meeting ITS AmericaERTICOITS JapanTransCore.” 2008.
- Kokogias S, Svensson L, Pereira G C, et al. “Development of Platform-Independent System for Cooperative Automated Driving Evaluated in GCDC 2016.” IEEE Transactions on Intelligent Transportation Systems, 2017, 1, 13.
- National Science Foundation of China, Intelligent vehicle future challenges : “2009-2017, Nat. Sci. Found. China, Beijing, China, Tech. Rep.,”2015.
- Sun, Y., Xiong, G., Ma, X., Gong, J., and Chen, H. , “Quantitative Evaluation of Unmanned Ground Vehicle Trajectory Based on Chaos Theory,” Advances in Mechanical Engineering 7(11), November 5, 2015): 168781401561692, doi:10.1177/1687814015616929.
- Huang, W.L., Wen, D., Geng, J., and Zheng, N.-N. , “Task-Specific Performance Evaluation of UGVs: Case Studies at the IVFC,” IEEE Transactions on Intelligent Transportation Systems 15(5):1969-1979, October 2014, doi:10.1109/tits.2014.2308540.
- Li, L., Huang, W.-L., Liu, Y., Zheng, N.-N., and Wang, F.-Y. , “Intelligence Testing for Autonomous Vehicles: A New Approach,” IEEE Transactions on Intelligent Vehicles 1(2):158-166, June 2016, doi:10.1109/tiv.2016.2608003.
- Li, L., Wen, D., Zheng, N.-N., and Shen, L.-C. , “Cognitive Cars: A New Frontier for ADAS Research,” IEEE Transactions on Intelligent Transportation Systems 13(1):395-407, March 2012, doi:10.1109/tits.2011.2159493.
- Meng, K.-W., Zhao, Y.-n., Gao, L., and Tan, H.-c. , “Evaluation of the Intelligent Behaviors of Unmanned Ground Vehicles Based on Information Theory,” CICTP 2015, July 13, 2015, doi:10.1061/9780784479292.037.
- Sun Y. “Quantitative Evaluation of Intelligent Levels for Unmanned Ground Vehicles [D].” Beijing Institute of Technology, 2014.
- Konrad . “Analog Bus Extension for PXI [EB/OL].” https://www.konradtechnologies.com/files/konrad/media/datasheet/ABex_Technology_Brochure_Rev120_web.pdf
- Rohde & Schwarz . “Automated Measurements of 77 GHz FMCW Radar Signals [EB/OL].” https://cdn.rohde-schwarz.com/pws/dl_downloads/dl_application/application_notes/1ef88/1EF88_0e_Automated_Measurements_of_77_GHz_FMCW_Radar.pdf
- Rohde & Schwarz . “Bring satellites into your lab: GNSS simulators from the T&M expert [EB/OL].” https://cdn.rohde-schwarz.com/pws/solution/wireless_and_mobile_communications/positioning___navigation_1/Rohde-Schwarz_GNSS_fly_en_5215_5042_32_v0100_web2.pdf
- Dspace . “Developing Advanced Driver Assistance Systems (ADAS) and Functions for Autonomous Driving.” https://www.dspace.com/shared/data/pdf/2017/ADAS_Brochure_2017_English.pdf
- Geiger, B.C. and Kubin, G. , “Relative Information Loss in the PCA,” 2012 IEEE Information Theory Workshop, September 2012, doi:10.1109/itw.2012.6404738.
- Bernardin, K. and Stiefelhagen, R. , “Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics,” EURASIP Journal on Image and Video Processing 2008:1-10, 2008, doi:10.1155/2008/246309.
- Van der Pol, E., Boing, A.N., Harrison, P., Sturk, A., and Nieuwland, R. , “Classification, Functions, and Clinical Relevance of Extracellular Vesicles,” Pharmacological Reviews 64(3):676-705, June 21, 2012, doi:10.1124/pr.112.005983.
- Geiger, A., P. Lenz, and R. Urtasun . “Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite.” 2012 IEEE Conference on Computer Vision and Pattern Recognition (June 2012). doi:10.1109/cvpr.2012.6248074.
- Sun H. “Studies on Trajectory Planning Considering Motion Uncertainties of Traffic Vehicles [D]. Jilin University,” 2016
- Zhao P. “Research on Motion Control Approaches of Autonomous Vehicle in Urban Environments [D].” University of Science and Technology of China, 2012
- Karayalcin, I.I. , “The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation,” European Journal of Operational Research 9(1):97-98, January 1982, doi:10.1016/0377-2217(82)90022-4.
- Hwang, C.L., Lai, Y.J., and Liu, T.Y. , “A New Approach for Multiple Objective Decision making,” Computers & Operations Research 20(8):889-899, 1993.
- “PanoSim an Integrated Simulation Platform for Intelligent Driving” [Online] Available: http://www.panosim.com/