Validation and Exploration of Multimodal Deep-Learning Camera–LiDAR Calibration Models

This paper presents an innovative study in exploring, evaluating, and implementing deep-learning architectures for the calibration of multimodal sensor systems. The aim of this paper is to leverage the use of sensor fusion to achieve dynamic, real-time alignment between 3D LiDAR and 2D camera sensors. Static calibration methods are tedious and time-consuming, which is why we propose utilizing conventional neural networks (CNNs) coupled with geometrically informed learning to solve this issue. We leverage the foundational principles of extrinsic LiDAR–camera calibration tools such as RegNet, CalibNet, and LCCNet by exploring open-source models that are available online and compare our results with their corresponding research papers. Requirements for extracting these visual and measurable outputs involved tweaking source code, fine-tuning, training, validation, and testing of each of these frameworks for equal comparisons. This approach aims to investigate which of these advanced networks produces the most accurate and consistent predictions. Through a series of experiments, we reveal some of their shortcomings and areas for potential improvements. We find that LCCNet yields the best results among all the models that we validated.