Advanced Energy Management Strategies for Plug-In Hybrid Electric Vehicles via Deep Reinforcement Learning
Technical Paper
2022-01-7109
ISSN: 0148-7191, e-ISSN: 2688-3627
Annotation ability available
Sector:
Language:
English
Abstract
Plug-in Hybrid Electric Vehicles (PHEVs) achieve significant fuel economy by
utilizing advanced energy management strategies in controlling the power
distribution decision in real-time. Traditional heuristic approaches bring no
additional benefits, including efficiency and development cost, considering the
increasing complexity in control objectives. This paper extends a previous study
of the same problem (RL) and vehicle topology to develop a Reinforcement
Learning agent by investigating the performance of state-of-the-art algorithms,
such as Rainbow-DQN with its variants, PPO and A3C, against the baseline
rule-based and Dynamic Programming (DP) strategies. The developed RL agent is
optimizing challenging control objectives such as fuel economy, vehicle
drivability and driver comfort. The Rainbow-DQN is studied separately to
optimize the agent compared to all the algorithm variants and after wards, the
best performing variant is compared to tuned PPO and A3C agents. Proper
evaluation criteria is defined and the concerned agents are tested with nine
different scenarios to examine the generalization capabilities and performance
robustness. The results revealed that the A3C agent surplussed both the PPO and
the Rainbow-DQN achieving a maximum performance of 98.43% of the DP with a
robustness of 97.32% ± 0.78 for the other cycles and an average of 177.7 sec for
each engine start compared to 96.3 sec for the rule-based approach. Furthermore,
as a future work, the paper investigated and proposed a cloud-based training
concept for automated scaled-up training, evaluation and deployment of RL
policies for the (P)HEVs of the future.
Authors
Citation
Mousa, A. and Weiss, G., "Advanced Energy Management Strategies for Plug-In Hybrid Electric Vehicles via Deep Reinforcement Learning," SAE Technical Paper 2022-01-7109, 2022.Also In
References
- International Council on Clean Transportation 2016
- Sandra Wappelhorst , G.B. 2021 https://theicct.org/blog/staff/phev-europe-company-cars-apr2021
- Liu , W. Introduction to Hybrid Vehicle System Modeling and Control John Wiley and Sons 2013 10.1002/9781118407400
- O’Keefe , M.P. and Markel , T. Dynamic Programming Applied to Investigate Energy Management Strategies for a Plug-In HEV1 22nd International Battery, Hybrid and Fuel Cell Electric Vehicle Symposium and Exposition, EVS 2006 1035 1046 2006
- Yang , Y. , Hu , X. , Pei , H. , and Peng , Z. Comparison of Power-Split and Parallel Hybrid Powertrain Architectures with a Single Electric Machine: Dynamic Programming Approach Appl. Energy 168 2016 683 690 10.1016/j.apenergy.2016.02.023
- Sundström , O. and Guzzella , L. A Generic Dynamic Programming Matlab Function Proceedings of the IEEE International Conference on Control Applications 2009 1625 1630 10.1109/CCA.2009.5281131
- Wang , R. and Lukic , S.M. Dynamic Programming Technique in Hybrid Electric Vehicle Optimization 2012 IEEE International Electric Vehicle Conference 2012 10.1109/IEVC.2012.6183284
- Li , Y. , He , H. , Peng , J. , and Wang , H. Deep Reinforcement Learning-Based Energy Management for a Series Hybrid Electric Vehicle Enabled by History Cumulative Trip Information IEEE Trans. Veh. Technol. 68 8 2019 7416 7430 10.1109/TVT.2019.2926472
- Liu , T. , Tang , X. , Hu , X. , Tan , W. et al. 2020 10.48550/arxiv.2007.10126
- Rezaei , A. , Burl , J.B. , and Zhou , B. Estimation of the ECMS Equivalent Factor Bounds for Hybrid Electric Vehicles IEEE Trans. Control Syst. Technol. 26 6 2198 2205 2018 10.1109/TCST.2017.2740836
- Borhan , H.A. , Zhang , C. , Vahidi , A. , Phillips , A.M. et al. Nonlinear Model Predictive Control for Power-Split Hybrid Electric Vehicles Proc. IEEE Conf. Decis. Control 2010 4890 4895 10.1109/CDC.2010.5718075
- Taghavipour , A. , Azad , N.L. , and McPhee , J. Real-Time Predictive Control Strategy for a Plug-In Hybrid Electric Powertrain Mechatronics 29 2015 13 27 10.1016/J.MECHATRONICS.2015.04.020
- Wang , Y. , Wang , X. , Sun , Y. , and You , S. Model Predictive Control Strategy for Energy Optimization of Series-Parallel Hybrid Electric Vehicle J. Clean. Prod. 199 2018 348 358 10.1016/J.JCLEPRO.2018.07.191
- Chen , H. Predictive Control Strategies of Plug-in HEVs 2019
- Mnih , V. et al. Human-level control through deep reinforcement learning Nature 518 7540 2015 529 533 10.1038/nature14236
- Silver , D. et al. 2017 10.48550/arxiv.1712.01815
- Sutton , R.S. and Barto , A.G. Reinforcement Learning, Second Edition: An Introduction - Complete Draft MIT Press 2018 1 3
- Zhu , Z. , Liu , Y. , and Canova , M. Energy Management of Hybrid Electric Vehicles via Deep Q-Networks 2020 American Control Conference (ACC) 2020 3077 3082 2020 10.23919/ACC45564.2020.9147479
- Hofstetter , J. , Bauer , H. , Li , W. , and Wachtmeister , G. Energy and Emission Management of Hybrid Electric Vehicles using Reinforcement Learning IFAC-PapersOnLine 52 29 2019 19 24 10.1016/J.IFACOL.2019.12.615
- Liessner , R. , Schroer , C. , Dietermann , A. , and Bäker , B. Deep Reinforcement Learning for Advanced Energy Management of Hybrid Electric Vehicles ICAART 2018 - Proceeding 10th International Conference on Agents and Artificial Intelligence 2 61 72 2018 10.5220/0006573000610072
- Li , Y. , He , H. , Peng , J. , and Zhang , H. Power Management for a Plug-in Hybrid Electric Vehicle Based on Reinforcement Learning with Continuous State and Action Spaces Energy Procedia 142 2017 2270 2275 10.1016/J.EGYPRO.2017.12.629
- Hoang , D.T. , Wang , P. , Niyato , D. , and Hossain , E. Charging and Discharging of Plug-In Electric Vehicles (PEVs) in Vehicle-to-Grid (V2G) Systems: A Cyber Insurance-Based Model IEEE Access 5 2017 732 754 10.1109/ACCESS.2017.2649042
- Sun , C. , Moura , S.J. , Hu , X. , Hedrick , J.K. et al. Dynamic Traffic Feedback Data Enabled Energy Management in Plug-in Hybrid Electric Vehicles IEEE Trans. Control Syst. Technol. 23 3 2015 1075 1086 10.1109/TCST.2014.2361294
- Mnih , V. et al. Asynchronous Methods for Deep Reinforcement Learning 33rd International Conference on Machine Learning (ICML 2016) 4 2850 2869 2016 10.48550/arxiv.1602.01783
- Mousa , A. Extended-Deep Q-Network : A Functional Reinforcement Learning-Based Energy Management Strategy for Plug-in Hybrid Electric Vehicles Eng. Sci. Technol. an Int. J. 2022
- Mousa , A. AI-based Energy Management Strategies for P2 Plug-in Hybrid Electric Vehicles M.S. thesis, AMM, FH OOE 2021 10.5281/ZENODO.7684683 https://doi.org/10.5281/zenodo.7684682
- Liu , J. and Peng , H. Modeling and Control of a Power-Split Hybrid Vehicle IEEE Trans. Control Syst. Technol. 16 6 2008 1242 1251 10.1109/TCST.2008.919447
- Xue , Q. , Zhang , X. , Teng , T. , Zhang , J. et al. 2020
- Sciarretta , A. and Guzzella , L. Optimal Energy-Management Strategies IEEE Control Syst. 27 2 60 70 2007 10.1109/MCS.2007.338280
- Hou , C. , Xu , L. , Wang , H. , Ouyang , M. et al. Energy Management of Plug-In Hybrid Electric Vehicles with Unknown Trip Length J. Franklin Inst. 352 2 2015 500 518 10.1016/j.jfranklin.2014.07.009
- Ambuhl , D. and Guzzella , L. Predictive Reference Signal Generator for Hybrid Electric Vehicles IEEE Trans. Veh. Technol. 58 9 2009 4730 4740 10.1109/TVT.2009.2027709
- Vinyals , O. et al. StarCraft II: A New Challenge for Reinforcement Learning 2017 https://arxiv.org/abs/1708.04782v1
- Liang , E. et al. RLlib: Abstractions for Distributed Reinforcement Learning 37th International Conference on Machine Learning ICML 2018 7 4768 4780 2017 10.48550/arxiv.1712.09381
- Brockman , G. et al. 2016 10.48550/arxiv.1606.01540
- van Hasselt , H. , Guez , A. , and Silver , D. Deep Reinforcement Learning with Double Q-Learning 30th AAAI Conference on Artificial Intelligence AAAI 2016 2094 2100 2015 https://arxiv.org/abs/1509.06461v3
- Wang , Z. , Schaul , T. , Hessel , M. , Van Hasselt , H. et al. Dueling Network Architectures for Deep Reinforcement Learning 33rd International Conference on Machine Learning (ICML 2016) 4 2939 2947 2015 10.48550/arxiv.1511.06581
- Schaul , T. , Quan , J. , Antonoglou , I. , and Silver , D. Prioritized Experience Replay 4th International Conference on Learning Representations ICLR 2016 - Conference Track Proceedings 2015 2021 https://arxiv.org/abs/1511.05952v4
- Hessel , M. et al. Rainbow: Combining Improvements in Deep Reinforcement Learning 32nd AAAI Conference on Artificial Intelligence, AAAI 2018 3215 3222 2017 10.48550/arxiv.1710.02298
- Schulman , J. , Wolski , F. , Dhariwal , P. , Radford , A. et al. 2017 10.48550/arxiv.1707.06347
- Zeng , F. , Wang , C. , and Ge , S.S. A Survey on Visual Navigation for Artificial Agents with Deep Reinforcement Learning IEEE Access 8 2020 135426 135442 10.1109/ACCESS.2020.3011438
- EPA 2021 https://www.epa.gov/vehicle-and-fuel-emissions-testing/dynamometer-drive-schedules