The plug-in hybrid electric vehicles with dual-motor and multi-gear structure can realize multiple operation modes such as series, parallel, hybrid, etc. The traditional rule-based energy management strategy mostly selects some of the modes (such as series and parallel) to construct the energy management strategy. Although this method is simple and reliable, it can’t fully exert the full potential of this structure considering both economy and driving performance. Therefore, it is very important to study the algorithm which can exert the maximum potential of the multi-degree-of-freedom structure. In this paper, a new RL-PMP algorithm is proposed, which does not divide the operation modes, and explores the optimal energy allocation strategy to the maximum extent according to the economic and drivability criteria within the allowable range of the characteristics of the power system components. Moreover, the algorithm can use reinforcement learning to adjust the key parameters adaptively, and it does not need global road information as input, as a result, it has good robustness and real-time. The simulation results show that the algorithm can achieve a better economy under different conditions compared with the rule-based energy management strategy. Besides, it requires fewer computing resources. Compared with the rule-based energy management strategy, there are not so many parameters that need to be calibrated in advance, which has engineering application value.