Physical Adversarial Infrastructure Attacks on Vision-Language Models for Autonomous Driving

Vision-language models (VLMs) offer a promising end-to-end approach for autonomous driving, but their robustness to physical adversarial attacks remains an unaddressed gap for safety-critical deployment. This paper introduces a systematic framework for evaluating these attacks using the Dolphins VLM and CARLA simulator. We employ Natural Evolution Strategies (NES), a black-box optimization method, to generate adversarial patches without internal model access. Our approach uses semantic similarity loss and Expectation over Transformation (EoT) to ensure physical realizability under varying viewing conditions. Patches are strategically placed on realistic advertising infrastructure (bus shelter panels and highway billboards), reflecting a plausible threat model. We establish two complementary scenarios: crosswalk pedestrian detection suppression and highway steering manipulation. Our evaluation reveals critical vulnerabilities. Adversarial patches achieved a 76.0% overall attack success rate, dramatically exceeding baseline inappropriate action rates (3.8-6.3%). Attacks maintained 75-90% efficacy within the critical 10-25 meter decision range and persisted for 3-4 seconds of sustained failure. We observed catastrophic perceptual degradation, including a 71.1 percentage point drop in pedestrian detection. Scene description analysis revealed holistic corruption with low BLEU-4 (0.18-0.24) and semantic similarity (0.49-0.59) scores, demonstrating that patches fundamentally corrupt the VLM’s scene understanding. These findings expose fundamental security challenges in vision-language integration, questioning the deployment readiness of current VLM architectures.