This content is not included in
your SAE MOBILUS subscription, or you are not logged in.
The Science of Testing: An Automotive Perspective
Technical Paper
2018-01-1070
ISSN: 0148-7191, e-ISSN: 2688-3627
This content contains downloadable datasets
Annotation ability available
Sector:
Language:
English
Abstract
Increasing automation in the automotive systems has re-focused the industry’s attention on verification and validation methods and especially on the development of test scenarios. The complex nature of Advanced Driver Assistance Systems (ADASs) and Automated Driving (AD) systems warrant the adoption of new and innovative means of evaluating and establishing the safety of such systems. In this paper, the authors discuss the results from a semi-structured interview study, which involved interviewing ADAS and AD experts across the industry supply chain.
Eighteen experts (each with over 10 years’ of experience in testing and development of automotive systems) from different countries were interviewed on two themes: test methods and test scenarios. Each of the themes had three guiding questions which had some follow-up questions. The interviews were transcribed and a thematic analysis via coding was conducted on the transcripts. A two-stage coding analysis process was done to first identify codes from the transcripts and subsequently, the codes were grouped into categories.
The analysis of transcripts for the question about the biggest challenge in the area of test methods revealed two specific themes. Firstly, the definition of pass/fail criteria and secondly the quality of requirements (completeness and consistency). The analysis of the questions on test scenarios revealed that “good” scenario is one that is able to test a safety goal and ways in which a system may fail. Based on the analysis of the transcripts, the authors propose two types of testing for ADAS and AD systems: Requirements-Based Testing (traditional method) and Hazard Based Testing. The proposed approach not only generates test scenarios for testing how the system works, but also how the system may fail.
Recommended Content
| Technical Paper | A System Safety Perspective into Chevy Bolt’s One Pedal Driving |
| Technical Paper | Hazard Analysis and Risk Assessment beyond ISO 26262:Management of Complexity Via Parametrization |
Authors
Citation
Khastgir, S., Birrell, S., Dhadyalla, G., and Jennings, P., "The Science of Testing: An Automotive Perspective," SAE Technical Paper 2018-01-1070, 2018, https://doi.org/10.4271/2018-01-1070.Data Sets - Support Documents
| Title | Description | Download |
|---|---|---|
| Unnamed Dataset 1 | ||
| Unnamed Dataset 2 |
Also In
References
- Carbaugh , J. , Godbole , D.N. , and Sengupta , R. Safety and Capacity Analysis of Automated and Manual Highway Systems Transp. Res. Part C Emerg. Technol. 6 1-2 69 99 1998 10.1016/S0968-090X(98)00009-6
- Fagnant , D.J. and Kockelman , K. Preparing a Nation for Autonomous Vehicles: Opportunities, Barriers and Policy Recommendations 3 SAGE 2015 0965-8564 10.1016/j.tra.2015.04.003
- van Arem , B. , Cornelie , J.G.V.D. , and Visser , R. The Impact of Co-Operative Adaptive Cruise Control on Traffic Flow Characteristics IEEE Trans. Intell. Transp. Syst. 7 4 429 436 2005
- Shladover , S.E. Cooperative (Rather than Autonomous) Vehicle-Highway Automation Systems IEEE Intell. Transp. Syst. Mag. 1 1 10 19 2009 10.1109/MITS.2009.932716
- Le Vine , S. , Zolfaghari , A. , and Polak , J. Autonomous Cars: The Tension between Occupant Experience and Intersection Capacity Transp. Res. Part C Emerg. Technol. 52 1 14 2015 10.1016/j.trc.2015.01.002
- Schoettle , B. and Sivak , M. 2014
- Le Coze , J.C. New Models for New Times. An Anti-Dualist Move Saf. Sci. 59 200 218 2013 10.1016/j.ssci.2013.05.010
- Malek , S. http://it-cisq.org/what-software-developers-can-learn-from-the-latest-car-recalls/ 2017
- Altinger , H. , Wotawa , F. , and Schurius , M. Testing Methods Used in the Automotive Industry: Results from a Survey Proc. of the 2014 Workshop on Joining AcadeMiA and Industry Contributions to Test Automation and Model-Based Testing-JAMAICA 2014 2014 9781450329330 10.1145/2631890.2631891
- Whittaker , J.A. What Is Software Testing? And why Is it So Hard? IEEE Softw. 17 1 70 79 2000 10.1109/52.819971
- Lions , J.L. 1996
- Weyuker , E.J. Testing Component-Based Software: A Cautionary Tale IEEE Softw. 15 5 54 59 1998 10.1109/52.714817
- NHTSA 2017
- Leveson , N.G. New Safety Technologies for the Automotive Industry 3 SAGE Detroit 2006
- ISO Road Vehicles-Functional Safety (ISO 26262) SAGE 2011
- Yu , H. , Lin , C.-W. , and Kim , B. Automotive Software Certification: Current Status and Challenges SAE Int. J. Passeng. Cars - Electron. Electr. Syst. 9 1 74 80 10.4271/2016-01-0050
- Khastgir , S. , Birrell , S. , Dhadyalla , G. , Sivencrona , H. et al. Towards Increased Reliability by Objectification of Hazard Analysis and Risk Assessment (HARA) of Automated Automotive Systems Saf. Sci. 99 166 177 2017 10.1016/j.ssci.2017.03.024
- Charette , R.N. This Car Runs on Code IEEE Spectr. 46 3 2009
- Khastgir , S. , Birrell , S. , Dhadyalla , G. , and Jennings , P. Identifying a Gap in Existing Validation Methodologies for Intelligent Automotive Systems: Introducing the 3xD Simulator 3 Sage 2015 9781467372664 10.1109/IVS.2015.7225758
- Kalra , N. and Paddock , S.M. Driving to Safety: How Many Miles of Driving would it Take to Demonstrate Autonomous Vehicle Reliability? Transp. Res. Part A Policy Pract. 94 182 193 2016 10.1016/j.tra.2016.09.010
- Transport Systems Catapult 2017
- Koopman , P. and Wagner , M. Challenges in Autonomous Vehicle Testing and Validation SAE Int. J. Transp. Saf. 4 1 15 24 2016 10.4271/2016-01-0128
- Wachenfeld , W. and Winner , H. The New Role of Road Testing for the Safety Validation of Automated Vehicles Automated Driving 419 435 2017 978-3-319-31893-6 10.1007/978-3-319-31895-0_17
- Campbell , R.L. Will the Real Scenario Please Stand Up? ACM SIGCHI Bull. 24 2 6 8 1992 10.1145/142386.1054872
- Cockburn , A. and Fowler , M. Question Time! About Use Cases ACM SIGPLAN Not. 33 10 226 229 1998 10.1145/286942.286960
- Cockburn , A. Structuring Use Cases with Goals J. Object Oriented Program 1997 5 1 16 1997
- Flage , R. and Aven , T. Emerging Risk-Conceptual Definition and a Relation to Black Swan Type of Events Reliab. Eng. Syst. Saf. 144 61 67 2015 10.1016/j.ress.2015.07.008
- Ergai , A. , Cohen , T. , Sharp , J. , Wiegmann , D. et al. Assessment of the Human Factors Analysis and Classification System (HFACS): Intra-Rater and Inter-Rater Reliability Saf. Sci. 82 393 398 2016 10.1016/j.ssci.2015.09.028
- Aven , T. On the Meaning of a Black Swan in a Risk Context Saf. Sci. 57 44 51 2013 10.1016/j.ssci.2013.01.016
- Khastgir , S. , Dhadyalla , G. , Birrell , S. , Redmond , S. et al. Test Scenario Generation for Driving Simulators Using Constrained Randomization Technique SAE Technical Paper 2017-01-1672 2017 10.4271/2017-01-1672.Copyright
- Griessnig , G. and Schnellbach , A. Development of the 2nd Edition of the ISO 26262 Stolfa J. , Stolfa S. , O’Connor R. , and Messnarz R. Systems, Software and Services Process Improvement: EuroSPI 2017: Communications in Computer and Information Science Cham Springer 2017 535 546 10.1007/978-3-319-64218-5 978-3-319-64217-8
- Louise Barriball , K. and While , A. Collecting Data Using a Semi-Structured Interview: A Discussion Paper J. Adv. Nurs. 19 2 328 335 1994 10.1111/j.1365-2648.1994.tb01088.x
- Robson , C. and McCartan , K. Real World Research: A Resource for Users of Social Research Methods in Applied Settings 4 Wiley 2016 9781405182416
- Saldaña , J. The Coding Manual for Qualitative Researchers 3 SAGE 2016
- Robinson-Mallett , C.L. An Approach on Integrating Models and Textual Specifications Proceedings of the 2nd IEEE International Workshop on Model Requirement Engineering MoDRE 2012 2012 92 96 10.1109/MoDRE.2012.6360079
- Robinson-Mallett , C. , Grochtmann , M. , Köhnlein , J. , Wegener , J. et al. Modelling Requirements to Support Testing of Product Lines ICSTW 2010-3rd International Conference on Software Testing, Verification and Validation Work 2010 11 18 10.1109/ICSTW.2010.65
- Alexander , I. Misuse Cases: Use Cases with Hostile Intent IEEE Softw. 20 1 58 66 2003 10.1109/MS.2003.1159030
- Stamatis , D.H. Failure Mode and Effect Analysis: FMEA from Theory to Execution 2 Milwaukee ASQ Quality Press 2003 9780521190817
- Hoffman , R.R. , Lintern , G. , and Eitelman , S. The Janus Principle IEEE Intell. Syst. 19 2 78 80 2004 10.1109/MIS.2004.1274915
- Salmon , P.M. , Cornelissen , M. , and Trotter , M.J. Systems-Based Accident Analysis Methods: A Comparison of Accimap, HFACS, and STAMP Saf. Sci. 50 4 1158 1170 2012 10.1016/j.ssci.2011.11.009
- Chen , S.T. , Wall , A. , Davies , P. , Yang , Z. et al. A Human and Organisational Factors (HOFs) Analysis Method for Marine Casualties Using HFACS-Maritime Accidents (HFACS-MA) Saf. Sci. 60 105 114 2013 10.1016/j.ssci.2013.06.009
- Baysari , M.T. , Caponecchia , C. , McIntosh , A.S. , and Wilson , J.R. Classification of Errors Contributing to Rail Incidents and Accidents: A Comparison of two Human Error Identification Techniques Saf. Sci. 47 7 948 957 2009 10.1016/j.ssci.2008.09.012
- Wiegmann , D. and Shappell , S. Applying the human factors analysis and classification system (HFACS) to the analysis of commercial aviation accident data Proc. of the 11th International Symposium on Aviation Psychology Columbus, Ohio 2001
- Lee , W.S. , Grosh , D.L. , Tillman , F.A. , and Lie , C.H. Fault Tree Analysis, Methods, and Applications - a Review IEEE Trans. Reliab. R-34 3 194 203 1985 10.1109/TR.1985.5222114
- Reay , K.A. and Andrews , J.D. A Fault Tree Analysis Strategy Using Binary Decision Diagrams Reliab. Eng. Syst. Saf. 78 1 45 56 2002 10.1016/S0951-8320(02)00107-2
- Abimbola , M. , Khan , F. , and Khakzad , N. Risk-Based Safety Analysis of Well Integrity Operations Saf. Sci. 84 149 160 2016 10.1016/j.ssci.2015.12.009
- Leveson , N.G. Applying Systems Thinking to Analyze and Learn from Events Saf. Sci. 49 1 55 64 2011 10.1016/j.ssci.2009.12.021
- Leveson , N. A New Accident Model for Engineering Safer Systems Saf. Sci. 42 4 237 270 2004 10.1016/S0925-7535(03)00047-X
- Leveson , N.G. Engineering a Safer World The MIT Press 2011 9780262016629
- Fleming , C.H. , Spencer , M. , Thomas , J. , Leveson , N. , and Wilkinson , C. Safety Assurance in NextGen and Complex Transportation Systems Saf. Sci. 55 173 187 2013 10.1016/j.ssci.2012.12.005
- Khastgir , S. , Birrell , S. , Dhadyalla , G. , and Jennings , P. Calibrating Trust on Automation in Vehicles through Knowledge: Introducing the Concept of Informed Safety Transp. Res. Part C Emerg. Technol.