Reinforcement Learning Based Energy Management of Hybrid Energy Storage Systems in Electric Vehicles

Energy management in electric vehicles plays a significant role in both reducing energy consumption and limiting the rate of battery capacity degradation. It is especially important for systems with multiple energy storage units where optimally arbitrating power demand among the energy storage units is challenging. While many optimal control methods exist for designing a good energy management system, in this work a Reinforcement-Learning (RL) methodology is explored to design an energy management system for an electric vehicle with a Hybrid Energy Storage System (HESS) that included a battery and a supercapacitor. The energy management system is designed to optimally divide the traction power request among a battery and a super-capacitor in real-time; while trying to minimize the overall energy consumption and battery degradation. Along with a vehicle and electric propulsion model, an empirical battery aging model was incorporated in the problem formulation to address effect of energy management on battery capacity degradation. Several optimization objectives including total energy consumption and long-term battery health impact were incorporated to the design to devise and further adapt a power arbitration strategy based on current HESS states and the total power demand to be achieved. The design process is presented starting from initial modeling work and various reinforcement learning formulations as well as training and validation of the RL-based energy management system utilizing developed models and with thousands of kilometers of real-world driving profiles in a high-performance computing (HPC) environment. It was demonstrated that one can successfully incorporate various heuristic design objectives (e.g. minimize loss for EV range, reduce battery throughput for aging) into the optimization problem in a data-driven framework. It was shown through a simulation study that battery aging can be reduced by about 4% for the electric vehicle considered in this work.