A Hierarchical Reinforcement Learning Framework with Target-State Guidance for Highway On-Ramp Merging
2026-99-2446
11/20/2026
- Content
- Highway on-ramp merging in high-density mixed traffic is challenging for autonomous vehicles (AVs) due to frequent vehicle interactions, strong lateral–longitudinal coupling, and complex decision-making requirements. To address these challenges, this paper proposes a two-level hierarchical reinforcement learning (HRL) framework to coordinate tactical lane-change decision-making and longitudinal motion control. The framework consists of three functional modules: high-level intent decision, target-state generation, and low-level continuous motion control. Specifically, the high-level policy employs Dueling Double Deep Q-Network (D3QN) to jointly determine lane-change decision and target merging gap. The selected intent is then transformed into target speed and position intervals by a target-state generation module, and a Soft Actor-Critic (SAC)-based low-level controller outputs continuous longitudinal acceleration to track these targets under dynamic traffic interactions. The proposed method is evaluated in a SUMO-based ramp-merging environment against rule-based and existing HRL baselines. Experimental results demonstrate stable convergence and clear advantages in task success rate, traffic efficiency, and safety. In addition, the space–time diagrams indicate that the proposed method induces less disturbance to the mainline traffic flow, which helps reduce the risk of congestion in the merging area.
- Citation
- Li, K., Lei, C., Ding, Y., Wang, S., et al., "A Hierarchical Reinforcement Learning Framework with Target-State Guidance for Highway On-Ramp Merging," 2026 2nd International Conference on Intelligent Transportation and Future Mobility (ITFM2026), Xi'An, China, March 27, 2026, https://doi.org/10.4271/2026-99-2446.