The Need for Dependency Coverage for Verification of Autonomous Vehicles

Scenario-based testing is one of the widely used approaches to verify the intended functionality of autonomous vehicles. In this approach, the vehicle is tested by exposing it to various known scenarios and then analyzed based on corresponding results. While scenarios can be constructed in simulation, they are hard to control in the real-world. Hence, a coverage metric such as scenario coverage is used to understand how many of the known scenarios has the vehicle been exposed to in simulation or real world. However, scenarios can be detailed at various levels of abstraction and many situations are possible to occur within a scenario. Therefore, considering only scenario coverage may not ensure that all the dependencies among components in the system and the ODD factors within a scenario are taken into account and verified sufficiently. To address this limitation, and to ensure that dependencies are not overlooked as a part of testing, we propose a new coverage metric called dependency coverage. This metric quantifies how many known relationships among components and ODD factors are considered to ensure the triggering conditions and functional insufficiencies are sufficiently covered during testing. The metric helps to evaluate the health of testing strategy and refine the strategy to increase the coverage. We explain with an example architecture on the drawbacks of solely relying on scenario coverage and how dependency coverage can be more beneficial.