This content is not included in
your SAE MOBILUS subscription, or you are not logged in.
A Comprehensive Testing and Evaluation Approach for Autonomous Vehicles
Technical Paper
2018-01-0124
ISSN: 0148-7191, e-ISSN: 2688-3627
This content contains downloadable datasets
Annotation ability available
Sector:
Language:
English
Abstract
Performance testing and evaluation always plays an important role in the developmental process of a vehicle, which also applies to autonomous vehicles. The complex nature of an autonomous vehicle from architecture to functionality demands even more quality-and-quantity controlled testing and evaluation than ever before. Most of the existing testing methodologies are task-or-scenario based and can only support single or partial functional testing. These approaches may be helpful at the initial stage of autonomous vehicle development. However, as the integrated autonomous system gets mature, these approaches fall short of supporting comprehensive performance evaluation. This paper proposes a novel hierarchical and systematic testing and evaluation approach to bridge the above-mentioned gap. In this paper, firstly a three-dimensional evaluation model conforming to the functional architecture of autonomous vehicles was built, with each dimension representing one of the three key functional layers of autonomous vehicle including sensing & perception, decision-making ﹠ planning, control & execution. Each dimension has a set of metrics carefully defined with their weights fairly determined based on an entropy weights method. Then, considering environment effect on vehicle functions, we innovatively determine task-scenarios for testing the performance of each dimension. Besides, we design a hierarchical systematic testing method which could specially testing each function layer of autonomous vehicle. Fuzzy comprehensive and TOPSIS evaluation method was proposed to quantitatively evaluate the comprehensive performance of autonomous vehicles under defined task-scenarios. Finally, our methods are used to evaluate three candidate vehicles based on simulation scenario in PanoSim. Compared to traditional approach based on external task performance, the proposed approach can not only provide convincing results of the overall system performance but can also peek into each of the key functional layer and provide insights about their performance. Therefore, this approach provides better guidance for autonomous vehicle research.
Recommended Content
Authors
Citation
Wang, G., Deng, W., Zhang, S., Wang, J. et al., "A Comprehensive Testing and Evaluation Approach for Autonomous Vehicles," SAE Technical Paper 2018-01-0124, 2018, https://doi.org/10.4271/2018-01-0124.Data Sets - Support Documents
Title | Description | Download |
---|---|---|
Unnamed Dataset 1 | ||
Unnamed Dataset 2 | ||
Unnamed Dataset 3 | ||
Unnamed Dataset 4 |
Also In
References
- Sun , Y. , Tao , G. , Xiong , G. , and Chen , H. The Fuzzy-AHP Evaluation Method for Unmanned Ground Vehicles Applied Mathematics & Information Sciences 7 2 653 658 March 1, 2013 10.12785/amis/070232
- Macias F. 2008
- Campbell , M. , Egerstedt , M. , How , J.P. , and Murray , R.M. Autonomous Driving in Urban Environments: Approaches, Lessons and Challenges Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 368 1928 4649 4672 September 6, 2010 10.1098/rsta.2010.0110
- Schneider , F.E. and Wildermuth , D. Results of the European Land Robot Trial and their Usability for Benchmarking Outdoor Robot Systems Lecture Notes in Computer Science 408 409 2011 10.1007/978-3-642-23232-9_51
- Lesemann M 2008
- Kokogias S , Svensson L , Pereira G C , et al. 2017
- National Science Foundation of China, Intelligent vehicle future challenges 2015
- Sun , Y. , Xiong , G. , Ma , X. , Gong , J. , and Chen , H. Quantitative Evaluation of Unmanned Ground Vehicle Trajectory Based on Chaos Theory Advances in Mechanical Engineering 7 11 November 5, 2015 10.1177/1687814015616929
- Huang , W.L. , Wen , D. , Geng , J. , and Zheng , N.-N. Task-Specific Performance Evaluation of UGVs: Case Studies at the IVFC IEEE Transactions on Intelligent Transportation Systems 15 5 1969 1979 October 2014 10.1109/tits.2014.2308540
- Li , L. , Huang , W.-L. , Liu , Y. , Zheng , N.-N. , and Wang , F.-Y. Intelligence Testing for Autonomous Vehicles: A New Approach IEEE Transactions on Intelligent Vehicles 1 2 158 166 June 2016 10.1109/tiv.2016.2608003
- Li , L. , Wen , D. , Zheng , N.-N. , and Shen , L.-C. Cognitive Cars: A New Frontier for ADAS Research IEEE Transactions on Intelligent Transportation Systems 13 1 395 407 March 2012 10.1109/tits.2011.2159493
- Meng , K.-W. , Zhao , Y.-n. , Gao , L. , and Tan , H.-c. Evaluation of the Intelligent Behaviors of Unmanned Ground Vehicles Based on Information Theory CICTP 2015 July 13, 2015 10.1061/9780784479292.037
- Sun Y. 2014
- Konrad https://www.konradtechnologies.com/files/konrad/media/datasheet/ABex_Technology_Brochure_Rev120_web.pdf
- Rohde & Schwarz https://cdn.rohde-schwarz.com/pws/dl_downloads/dl_application/application_notes/1ef88/1EF88_0e_Automated_Measurements_of_77_GHz_FMCW_Radar.pdf
- Rohde & Schwarz https://cdn.rohde-schwarz.com/pws/solution/wireless_and_mobile_communications/positioning___navigation_1/Rohde-Schwarz_GNSS_fly_en_5215_5042_32_v0100_web2.pdf
- Dspace https://www.dspace.com/shared/data/pdf/2017/ADAS_Brochure_2017_English.pdf
- Geiger , B.C. and Kubin , G. Relative Information Loss in the PCA 2012 IEEE Information Theory Workshop September 2012 10.1109/itw.2012.6404738
- Bernardin , K. and Stiefelhagen , R. Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics EURASIP Journal on Image and Video Processing 2008 1 10 2008 10.1155/2008/246309
- Van der Pol , E. , Boing , A.N. , Harrison , P. , Sturk , A. , and Nieuwland , R. Classification, Functions, and Clinical Relevance of Extracellular Vesicles Pharmacological Reviews 64 3 676 705 June 21, 2012 10.1124/pr.112.005983
- Geiger , A. , P. Lenz , and R. Urtasun Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite 2012 IEEE Conference on Computer Vision and Pattern Recognition June 2012 10.1109/cvpr.2012.6248074
- Sun H. 2016
- Zhao P. 2012
- Karayalcin , I.I. The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation European Journal of Operational Research 9 1 97 98 January 1982 10.1016/0377-2217(82)90022-4
- Hwang , C.L. , Lai , Y.J. , and Liu , T.Y. A New Approach for Multiple Objective Decision making Computers & Operations Research 20 8 889 899 1993
- http://www.panosim.com