Sensor-Fused Low Light Pedestrian Detection System with Transfer Learning

Objection detection using a camera sensor is essential for developing Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD) vehicles. Due to the recent advancement in deep Convolution Neural Networks (CNNs), object detection based on CNNs has achieved state-of-the-art performance during daytime. However, using an RGB camera alone in object detection under poor lighting conditions, such as sun flare, snow, and foggy nights, causes the system's performance to drop and increases the likelihood of a crash. In addition, the object detection system based on an RGB camera performs poorly during nighttime because the camera sensors are susceptible to lighting conditions. This paper explores different pedestrian detection systems at low-lighting conditions and proposes a sensor-fused pedestrian detection system under low-lighting conditions, including nighttime. The proposed system fuses RGB and infrared (IR) thermal camera information. IR thermal cameras are used as they are capable of generating good quality images under low illumination and can help better object detection at nighttime. Utilizing the two sensors, a two-stream pedestrian detection system is developed using the YOLO (You Only Look Once) architecture and transfer learning technology. The RGB+ IR sensor fused system is evaluated with the available public data sets and compared with the systems developed with a single sensor, an IR camera-only system. The sensor-fused system is successfully deployed on the NVIDIA Jetson Orin. The overall detection results under low light conditions show that the proposed sensor fusion system significantly improves the overall performance in object detection under low lighting conditions.