This content is not included in
your SAE MOBILUS subscription, or you are not logged in.
Validating an Approach to Assess Sensor Perception Reliabilities Without Ground Truth
Technical Paper
2021-01-0080
ISSN: 0148-7191, e-ISSN: 2688-3627
This content contains downloadable datasets
Annotation ability available
Sector:
Event:
SAE WCX Digital Summit
Language:
English
Abstract
A reliable environment perception is a requirement for safe automated driving. For evaluating and demonstrating the reliability of the vehicle’s environment perception, field tests offer testing conditions that come closest to the vehicle’s driving environment. However, establishing a reference ground truth in field tests is time-consuming. This motivates the development of a procedure for learning the vehicle’s perception reliability from fleet data without the need for a ground truth, which would allow learning the perception reliability from fleet data.
In Berk et al. (2019), a method based on Bayesian inference to determine the perception reliability of individual sensors without the need for a ground truth was proposed. The model utilizes the redundancy of sensors to learn the sensor’s perception reliability. The method was tested with simulated data. In this contribution, we further explore and validate the method by utilizing real data, including ground truth data based on high-resolution LIDAR and human labeling. An area with overlapping field of view from five sensors is selected for the analysis. A basic association method is used to compare the object data obtained from the different sensors. Finally, we compare the sensor perception reliabilities learned from the Bayesian inference model with the sensor perception reliabilities determined from the labeled ground truth.
In this paper, it is shown that the model introduced in Berk et al. (2019) can approximate the reference data based on the provided ground truth. The estimated parameters of the model do not perfectly correspond to the sensor reliabilities but are of the same order of magnitude as when derived from the ground truth.
Authors
Citation
Kryda, M., Berk, M., Buschardt, B., and Straub, D., "Validating an Approach to Assess Sensor Perception Reliabilities Without Ground Truth," SAE Technical Paper 2021-01-0080, 2021, https://doi.org/10.4271/2021-01-0080.Data Sets - Support Documents
Title | Description | Download |
---|---|---|
Unnamed Dataset 1 | ||
Unnamed Dataset 2 | ||
Unnamed Dataset 3 | ||
Unnamed Dataset 4 | ||
Unnamed Dataset 5 |
Also In
References
- Berk , M. , Schubert , O. , Kroll , H. , Buschardt , B. et al. Exploiting Redundancy for Reliability Analysis of Sensor Perception in Automated Driving Vehicles IEEE Transactions on Intelligent Transportation Systems https://doi.org/10.1109/TITS.2019.2948394
- Salvatier , J. , Wiecki , T.V. , and Fonnesbeck , C. Probabilistic Programming in Python using PyMC3 PeerJ Computer Science 2 e55 2016 https:doi:10.7717/peerj-cs.55
- Hoffmann , M.D. and Gelman , A. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo Journal of Machine Learning Research 15 1 1593 1623 2014
- Dunnet , C. and Sobel , M. Approximations to the Probability Integral and Certain Percentage Points of a Multivariate Analogue of Student’s t-Distribution Biometrika 42 1 2 258 260 1955 https://doi.org/10.2307/2333441
- Teoh , E.R. and Kidd , D.G. Rage Against the Machine? Google’s Self-Driving Cars versus Human Drivers Journal of Safety Research 63 57 60 2017 https://doi.org/10.1016/j.jsr.2017.08.008
- Schubert , O. , Kroll , H. , Buschardt , B. et al. Reliability Assessment of Safety-Critical Sensor Information: Does One Need a Reference Truth? IEEE Transactions on Reliability 68 4 1227 1241 2019 https://doi.org/10.1109/TR.2019.2923735
- Berk , M. , Schubert , O. , Kroll , H. , Buschardt , B. et al. Assessing the Safety of Environment Perception in Automated Driving Vehicles SAE Int. J. Trans. Safety 8 1 49 74 2020 https://doi.org/10.4271/09-08-01-0004
- Kalra , N. and Paddock , S.M. 2016 https://doi.org/10.7249/RR1478
- Li , L. , Huang , W. , Liu , Y. , Zheng , N. et al. Intelligence Testing for Autonomous Vehicles: A New Approach IEEE Transactions on Intelligent Vehicles 1 2 158 166 2016 https://doi.org/10.1109/TIV.2016.2608003
- Alawadhi , M. , Almazrouie , J. , Kamil , M. , and Khalil , K.A. A Systematic Literature Review of the Factors Influencing the Adoption of Autonomous Driving Int J Syst Assur Eng Manag 2020 https://doi.org/10.1007/s13198-020-00961-4
- Wachenfeld , W. and Winner , H. The Release of Autonomous Vehicles Maurer , M. , Gerdes , J. , Lenz , B. , and Winner , H. Autonomous Driving Berlin, Heidelberg Springer 2016 https://doi.org/10.1007/978-3-662-48847-8_21
- Wachenfeld , W. and Winner , H. The New Role of Road Testing for the Safety Validation of Automated Vehicles Watzenig , D. , Horn , M. Automated Driving Cham Springer 2017 https://doi.org/10.1007/978-3-319-31895-0_17
- Mullins , G.E. , Stankiewicz , P.G. , and Gupta , S.K. Automated Generation of Diverse and Challenging Scenarios for Test and Evaluation of Autonomous Vehicles 2017 IEEE International Conference on Robotics and Automation (ICRA) 1443 1450 https://doi.org/10.1109/ICRA.2017.7989173
- Huang , W. , Wang , K. , Lv , Y. , and Zhu , F. Autonomous Vehicles Testing Methods Review 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC) 163 168 https://doi.org/10.1109/ITSC.2016.7795548
- Tuncali , C.E. , Fainekos , G. , Ito , H. , and Kapinski , J. Simulation-Based Adversarial Test Generation for Autonomous Vehicles with Machine Learning Components CoRR 2018
- Koopman , P. Practical Experience Report: Automotive Safety Practices vs. Accepted Principles Preprint Computer Safety, Reliability, and Security 2018
- Koopman , P. and Wagner , M. Toward a Framework for Highly Automated Vehicle Safety Validation SAE International 2018 https://doi.org/10.4271/2018-01-1071
- Koopman , P. and Osyk , B. Safety Argument Considerations for Public Road Testing of Autonomous Vehicles SAE Int. J. Adv. & Curr. Prac. in Mobility 1 2 512 523 2019 https://doi.org/10.4271/2019-01-0123
- Åsljung , D. , Nilsson , J. , and Fredriksson , J. Using Extreme Value Theory for Vehicle Level Safety Validation and Implications for Autonomous Vehicles IEEE Transactions on Intelligent Vehicles 2 4 288 297 2017 https://doi.org/10.1109/TIV.2017.2768219
- Koopman , P. and Wagner , M. Autonomous Vehicle Safety: An Interdisciplinary Challenge IEEE Intelligent Transportation Systems Magazine 9 1 90 96 2017 https://doi.org/10.1109/MITS.2016.2583491
- Corso , A. and Kochenderfer , M.J. 2020
- Takács , Á. , Drexler , D.A. , Galambos , P. , Rudas , I.J. et al. Assessment and Standardization of Autonomous Vehicles 2018 IEEE 22nd International Conference on Intelligent Engineering Systems (INES) 2018 https://doi.org/10.1109/INES.2018.8523899
- ISO/PAS 21448 Road Vehicles - Safety of the Intended Functionality Geneva, Switzerland International Organization for Standardization 2019
- Marco Kryda marco.kryda@tum.de