Optimization of Energy Management Strategy for Range-Extended Electric Vehicle Using Reinforcement Learning and Neural Network

A Range-Extended Electric Vehicle (REEV) uses battery as the primary energy source and engine as the secondary source to extend the total range of the vehicle. Deep Orange 11 program at Clemson University is proposing a REEV for solving the mobility needs in the year 2040. Designing the Energy Management System (EMS) of such a vehicle is a critical aspect of the problem statement of this program to improve the vehicle economy and bring down the cost of operation of the vehicle. This paper proposes a reinforcement learning based algorithm for designing the EMS of such a vehicle. Q-learning is a model-free algorithm which seeks to improve the cumulative reward by finding the best policy over the course of operation. A rule-based strategy is first used to establish a baseline model of engine operation during the operation of vehicle over an EPA drive-cycle (FHDS). The Q-learning strategy is then deployed which learns over the baseline strategy as the vehicle travels over the drive cycle and improves the fuel economy of the vehicle. A high-fidelity vehicle simulator is developed in Simulink to simultaneously train and validate the strategy. Finally, a neural network is used to approximate the relationship between state and actions which can be used for implementation of the strategy in the vehicle, owing to its higher computational efficiency. The target of the research is to cover 120 km most efficiently over FHDS drive cycle. The results of this study showed an improved vehicle economy of 27.78% over the conventional rule-based strategy. The reward function has been designed such that it can be tuned to improve the economy over different driving distances as per requirement. The study also presents a state-based randomization as opposed to a conventional time-based randomization which shows significant improvement in convergence to optimal value.