This document provides preliminary1 safety-relevant guidance for in-vehicle fallback test driver training and for on-road testing of vehicles being operated by prototype conditional, high, and full (Levels 3 to 5) ADS, as defined by SAE J3016. It does not include guidance for evaluating the performance of post-production ADS-equipped vehicles. Moreover, this guidance only addresses testing of ADS-operated vehicles as overseen by in-vehicle fallback test drivers (IFTD).
These guidelines do not address:
Remote driving, including remote fallback test driving of prototype ADS-operated test vehicles in driverless operation. (Note: The term “remote fallback test driver” is included as a defined term herein and is intended to be addressed in a future iteration of this document. However, at this time, too little is published or known about this type of testing to provide even preliminary guidance.)
Testing of driver support features (i.e., Levels 1 and 2), which rely on a human driver to perform part of the dynamic driving task (DDT) and to supervise the driving automation feature’s performance in real time. (Refer to SAE J3016.)
Closed-course testing.
Simulation testing (except for training purposes).
Component-level testing.
These guidelines also do not address prototype vehicle and IFTD performance data collection and retention. The collection of data invokes various legal and risk management considerations that users of this document should nevertheless bear in mind, such as:
Maintaining auditable procedures and documentation.
Adhering to applicable privacy laws and principles.
Ensuring adequate data collection and recording integrity to support post-crash forensic analysis.
Rationale
This document provides safety-relevant guidance for in-vehicle fallback test driver training and for testing prototype automated driving systems (ADS) equipped on test vehicles operated in mixed-traffic environments on public roads (hereafter, prototype ADS-operated vehicles). This document is being substantially updated in order to incorporate content from Automated Vehicle Safety Consortium (AVSC) publication 00001201911: “AVSC Best Practice for In-Vehicle Fallback Test Driver Selection, Training, and Oversight Procedures for Automated Vehicles Under Test” and to re-classify this document as an SAE Recommended Practice, rather than an SAE Information Report.
It is assumed that the prototype ADS-operated vehicles that are the subject of this guidance have been developed using standardized methods for safer product development including, but not limited to:
A systems engineering approach (i.e., V-model).
Adherence to a recognized system safety process(es) for identifying hazards and implementing strategies for mitigating them.
Implementation of an electrical/electronic (E/E) architecture (system/hardware/software levels) capable of implementing hazard mitigation concepts and strategies.
Analysis and testing of identified hazard mitigation strategies (hardware and software).
Prototype ADS-operated vehicles that are based on existing production vehicles rely on the existing vehicle’s E/E architecture, as adapted for ADS. Prototype ADS technology provided via added hardware and software modules that are not integrated according to the vehicle manufacturer’s specifications, should be checked to ensure that they do not interfere with base vehicle hardware or software systems. As such, they should abide by the following general principles:
All hardware and software interfaces between production- and development-level hardware and software should be analyzed and tested for operational integrity, including analysis of failure modes and effects.
Developmental software added to a vehicle (including that equipped on added hardware modules) should be monitored and/or include self-diagnostics for safety-critical functions, which should be verified for efficacy prior to on-road testing. Alternatively, system-level approaches to ensuring developmental software safety (e.g., shadow mode testing) is also acceptable.
Test program/operations management plays a key role in helping to maintain safety while conducting on-road testing of prototype ADS-operated vehicles. Unexpected behaviors (including incidents) should be reported accurately and consistently for later root-cause analysis and resolution. A manager in charge of prototype ADS-operated vehicle testers should explain to them the organization’s specific rules about testing and documentation, as well as any hardware/software updates that impact the performance of the ADS-operated vehicles. Novice testers should be paired with more experienced testers to learn the appropriate reactions in various situations.
Real-time calibration/tuning of ADS software during testing should be allowed only after evaluation by qualified personnel (e.g., development engineer, lead calibrator, and/or designated safety engineer), indicating that the change does not pose unacceptable risk for on-road testing.
The On-Road Automated Driving (ORAD) committee reports to the Driver Assistance Systems Steering Committee of the Motor Vehicle Council. The ORAD committee is responsible for developing and maintaining SAE standards, recommended practices, and information reports related to motor vehicle driving automation system features across the full range of levels of driving automation. "On-road" refers to publicly accessible roadways that provide driving environments for the users of motor vehicles of all classes and all levels of driving automation. However, the ORAD committee will focus primarily on Automated Driving Systems (ADSs) as defined by SAE J3016, namely, levels 3-5 which are capable of performing the complete DDT on a sustained basis while engaged. This includes cooperative automation at these levels. While automated subsystems from other committees will be integrated with automated driving technology, the ORAD committee does not focus on specific subsystems like short-range communication, active safety such as electronic stability control and automated emergency braking or other driver assistance such as lane keeping assistance and cooperative adaptive cruise control. The ORAD committee will coordinate with and contribute to committees and task forces in these distinct areas of expertise.
* Redlines comparisons are available for those standards
listed in the Revision History that contain a radio button. A
redline comparison of the current version against a revision is
accomplished by selecting the radio button next to the standard and
then selecting 'compare'. At this time, Redline versions only exist
for some AMS standards. SAE will continue to add redline versioning
with ongoing updates to SAE MOBILUS.
View/Annotate
Content Loading, Please Wait
Document Preview
Content Loading, Please Wait
Document Comparison
Content Loading, Please Wait
Download
A list of downloadable items.
Product Code
Document Title
File Type
Download
J3018_202012
Safety-Relevant Guidance for On-Road Testing of Prototype Automated Driving System (ADS)-Operated Vehicles
Export Metadata
This modal will allow the user to download citation information or export specific fields of data into one of three file formats: Excel, comma-separated values, or text.
You are currently not logged into your Dashboard account, which is required for annotating.
Please select Login to login or register for the Dashboard. Select Continue to view the content without the ability to annotate.
Redline Warning
You are currently not logged into your Dashboard account, which is required for annotating.
Please select Login to login or register for the Dashboard. Select Continue to view the content without the ability to annotate.
Data Set Warning
Safety-Relevant Guidance for On-Road Testing of Prototype Automated Driving System (ADS)-Operated Vehicles
SAE Journal Subscriptions are available in a variety of options, which include electronic, electronic with back file, and archive. For detailed information and options, please complete the following:
Content Request
Check My Access!
You may already have access to the Ground Vehicle Standard, Safety-Relevant Guidance for On-Road Testing of Prototype Automated Driving System (ADS)-Operated Vehicles
Subscription Access
If you have Subscription Access with a username and password, please login below.
Institutional Login
If you have Institutional Access to SAE MOBILUS, using a predetermined organizational login utilizing single-sign-on (Shibboleth), please login below. To request institutional access to SAE MOBILUS, please complete the institutional subscription information form.
Purchase
If you do not have access to SAE MOBILUS via username/password or institutional access, you can still purchase the Ground Vehicle Standard, Safety-Relevant Guidance for On-Road Testing of Prototype Automated Driving System (ADS)-Operated Vehicles.
Search Tips: Search Operators:AND, OR, NOT; Wild Cards; Using Quotes
SAE MOBILUS® Search Tips
Please note: All terms not contained within quotes, will be evaluated using the OR operator. This means that your search results will contain all results that contain ANY of the entered terms.
AND, OR, NOT
SAE MOBILUS supports the use of the AND, OR and NOT BOOLEAN operators ONLY function with the Advanced Search.
Wild Cards
Using the asterisk, ( * ) allows you to search for a partial word. For example, entering a keyword search of aero* will search for any word beginning with aero.
Using the question mark, ( ? ) allows you to search for a word with a single character wildcard. For example, searching for t?re would match both tire and tyre
Using Quotes
Words in double quotes " " are treated as a phrase. Conducting a search for "heat model" will return all results in which the exact phrase "heat model" is found, while "heat modeling" would NOT be included.
A term combined with a phrase will be joined with an OR operator.
Searching for Standards
When searching for a standard please use the entire Document Number assigned, which includes the abbreviation. A space can be used, or omitted between the abbreviation and number.
