Using Deep Reinforcement Learning for Hybrid Electric Vehicle Energy Management under Consideration of Dynamic Emission Models

Abstract

Hybrid electric vehicles (HEV) contribute to reduce emissions from transportation. The energy management controls the powertrain components in HEVs. In addition to minimizing fuel consumption, improving air quality is a major opportunity for hybrid vehicles. Pollutant emissions can only be mapped with sufficient accuracy using dynamic models. The introduced nitrogen oxide model is created using a supervised learning approach based on recorded measurement data. This dynamic model requires input data from previous time steps to ensure sufficient model quality. Classical algorithms such as Dynamic Programming are not able to find solutions for such high-dimensional problems in reasonable computing times. A promising approach to solve the resulting problem is Deep Reinforcement Learning (Deep RL), which has recently been introduced in the field of HEV energy management. Due to the significantly shorter computing times of the Deep RL it is possible to train the energy management with extensive stochastic driving cycles and additionally to consider other relevant system variables such as battery temperature or battery derating. Since the use of the dynamic emission model involves the violation of the Markov condition, the contribution shows an approach to solve the emerging Partially Observable Markov Decision Process (POMDP) using a DDPG agent. The paper thus presents a new multi-criteria optimization algorithm for the design of a diesel hybrid energy management observing fuel consumption and nitrogen oxide emissions. The results show a great potential concerning the reduction of the regarded exhaust gas components in real traffic situations.