Incorporating the Scenario-Based Framework Underpinned by Operational Design Domain (ODD) for Automated Driving Stack (ADS) Evaluation

The rapid development of open-source Automated Driving System (ADS) stacks has created a pressing need for clear guidance on their evaluation and selection for specific use cases. This paper introduces a scenario-based evaluation framework combined with a modular simulation framework, offering a scalable methodology for assessing and benchmarking ADS solutions, including but not limited to off-the-shelf designs. The study highlights the lack of clear Operational Design Domain (ODD) descriptions in such systems. Without a common understanding, users must rely on subjective assumptions, which hinders the process of accurate system selection. To address this gap, the study proposes adopting a standardised ISO 34503 ODD description format within the ADS stacks. The application of the proposed framework is showcased through a case study evaluating two open-source systems, Autoware and Apollo. By first defining the assumed system's ODD, then selecting a relevant scenario, and establishing pass/fail criteria, the framework provides objective data to highlight performance differences between both systems. The proposed methodology provides an unbiased foundation for informed decision-making, promoting the safe and effective integration of ADS technologies.