A Visible and Infrared Fusion Based Visual Odometry for Autonomous Vehicles

An accurate and timely positioning of the vehicle is required at all times for autonomous driving. The global navigation satellite system (GNSS), even when integrated with costly inertial measurement units (IMUs), would often fail to provide high-accuracy positioning due to GNSS-challenged environments such as urban canyons. As a result, visual odometry is proposed as an effective complimentary approach. Although it’s widely recognized that visual odometry should be developed based on both visible and infrared images to address issues such as frequent changes in ambient lightening conditions, the mechanism of visible-infrared fusion is often poorly designed. This study proposes a Generative Adversarial Network (GAN) based model comprises a generator, which aims to produce a fused image combining infrared intensities and visible gradients, and a discriminator whose target is to force the fused image to retain as many details that exist mostly in visible images as possible. Based on the fused image, the Features from Accelerated Segment Test (FAST) algorithm is adopted to extract feature points which are then traced with the Lucas-Kanade (LK) algorithm in subsequent images. Furthermore, to remove mismatched feature points, the Random Sample Consensus (RANSAC) method is employed to detect the outliers iteratively and to compute the essential matrix. Experiments are conducted utilizing the KAIST benchmark dataset. A significant improve in positioning accuracy is observed from experimental results, as compared to visual odometry built upon visible and infrared images only. The proposed visual odometry can provide high-accuracy positioning when the GNSS is challenged.