Tunnel Traffic Anomaly Detection via Multi-Modal LLMs

Tunnels play a crucial role in urban transportation, yet they frequently encounter various incidents during operation. Manual video inspections and sensor-based systems are inefficient and limited in accurately detecting and addressing these issues. The emergence of artificial intelligence has led to the development of object detection models such as YOLO, which have shown promise in real-time anomaly detection. However, these single-modality models achieve suboptimal results when dealing with complex events. Multi-modal large language models (LLMs) offer a potential solution, with their ability to process and understand information from different modalities. This paper develops a novel tunnel traffic anomaly detection method that combines single-modal models and multi-modal LLMs. The proposed system first employs YOLO for an initial detection round and then utilizes a specially designed LLM with an effective prompt and a data filtering strategy tailored for traffic tunnel scenarios. This two-step approach enables the system to detect anomalies such as fires and ponding water, facilitating real-time monitoring of tunnel conditions and maintaining traffic flow. We are the first to introduce a well-designed multi-modal LLM into tunnel traffic anomaly detection, for real-time and accurate detection. We create a tunnel-specific algorithm that covers model design, prompt strategy, and detection logic, effectively handling complex weather and traffic scenarios. The system has demonstrated an accuracy rate of up to 90% in detecting numerous surveillance cameras simultaneously, reducing labor costs and potential economic losses associated with tunnel incidents. Our research thus aims to enhance tunnel safety and efficiency through an innovative and effective anomaly detection system.