Video 2 Scenario: A Novel Method to Generate Edge Case Traffic Scenarios from Dashcam Videos

This paper presents Video 2 Scenario, a systematic and cost-effective method to generate edge case scenarios derived from road events captured in dash camera videos. Current methods for obtaining real-world scenarios rely on supplementary sensor data like LiDAR, GPS and radar, which require expensive sensor setups for data collection. Other scenario generation methods rely on manual effort by human experts, which can be laborious and time-consuming. Our proposed methodology utilizes state-of-the-art deep learning models to extract vehicle trajectories, road layouts and other scenario-specific elements and reconstructs a scenario in the format of simulation scripts. The scripts are then refined by replaying the scenario multiple times in a simulation with attached virtual sensors to minimize the discrepancy between the subsequently obtained simulated video data and the original dash camera video data. The refined simulation scripts ensure that critical edge case scenarios are replicated with high accuracy. Our initial experiments show promising results, as our methodology is able to replicate simple to moderately complex videos in simulation. From just 20 dash camera videos (each video being around 30 seconds in length), we were able to generate over 2000 scenarios with simple variations in under 6 hours with a simple hardware setup, which would have otherwise taken multiple days for human experts to manually create from scratch.