Quality Evaluation of Synthetic Images for Camera Object Detection Model

The realism of simulation scenarios significantly impacts the performance of object detection models. The confidence of synthetic images generated by current simulators remains controversial. Verifying the quality of synthetic image generation methods is a prerequisite and foundation for using these synthetic images for robustness testing and optimization of object detection models. Therefore, this paper aims to propose a quality quantification evaluation scheme for synthetic images based on the object detection model. This scheme serves as a reference for ensuring the reliability of simulation testing and for optimizing the quality of synthetic images. Firstly, a synthetic image dataset for quality evaluation is created based on real driving scenarios from the Waymo open dataset. The 3D Gaussian Splatting (3D GS) method is employed to reconstruct high-fidelity scenarios and generate images from new viewpoints, thereby producing a static scenario synthetic image dataset. The approach of reconstructing static backgrounds with 3D GS and defining dynamic objects in the simulator using the OpenSCENARIO standard language can efficiently generate diverse dynamic testing scenarios, thereby producing a dynamic scenario synthetic image dataset. Then, the quality evaluation indicators system and the comprehensive evaluation model for synthetic images are established based on object detection models. The experimental results show that the static scenario synthetic images already have high quality, while the dynamic scenario synthetic images still require further optimization in terms of object quality. Furthermore, when optimizing the scenario generation methods, only referring to the image quality evaluation will have limitations. Given the application, it is more crucial to focus on whether the simulation confidence has met the test requirements, thereby achieving a comprehensive balance of other performances. This experiment validates the necessity and application value of the proposed evaluation scheme.