Visual sensors are widely used in autonomous vehicles (AVs) for object detection due to the advantages of abundant information and low-cost. But the performance of visual sensors is highly affected by low light conditions when AVs driving at nighttime and in the tunnel. The low light conditions decrease the image quality and the performance of object detection, and may cause safety of the intended functionality (SOTIF) problems. Therefore, to analyze the performance limitations of visual sensors in low light conditions, a controlled light experiment on a proving ground is designed. The influences of low light conditions on the two-stage algorithm and the single-stage algorithm are compared and analyzed quantificationally by constructing an evaluation index set from three aspects of missing detection, classification, and positioning accuracy. Five main environmental influencing factors are tested and analyzed in typical nighttime urban driving scenarios: illuminance, the lateral movement of the object, the longitudinal distance of the object, the high beams of the oncoming vehicle, and the low beams of the ego vehicle. The test results show that the performance limitations of recognition algorithms can be triggered by the low illuminance, the lateral movement of the object, the long longitudinal distance of the object, and the high beams of the oncoming vehicle. For different types of recognition algorithms, the performance of Faster R-CNN is better than that of YOLOv5 in most scenarios. As for the environmental factor, the low beams of the ego vehicle, it can improve the performance of recognition algorithms in low light conditions when the longitudinal distance of the object is less than 87.5m. This paper provides a reference for the design and performance evaluation of visual sensors for AVs, as well as improving SOTIF performance.