A Decentralized Multi-agent Energy Management Strategy Based on a Look-Ahead Reinforcement Learning Approach

Journal Article
14-11-02-0012
ISSN: 2691-3747, e-ISSN: 2691-3755
Published November 05, 2021 by SAE International in United States
A Decentralized Multi-agent Energy Management Strategy Based on a
                    Look-Ahead Reinforcement Learning Approach
Citation: Khalatbarisoltani, A., Kandidayeni, M., Boulon, L., and Hu, X., "A Decentralized Multi-agent Energy Management Strategy Based on a Look-Ahead Reinforcement Learning Approach," SAE Int. J. Elec. Veh. 11(2):151-164, 2022, https://doi.org/10.4271/14-11-02-0012.
Language: English

References

  1. Cano , Z.P. , Banham , D. , Ye , S. , Hintennach , A. et al. Batteries and Fuel Cells for Emerging Electric Vehicle Markets Nature Energy 3 2018 279 289
  2. Das , H.S. , Tan , C.W. , and Yatim , A.H.M. Fuel Cell Hybrid Electric Vehicles: A Review on Power Conditioning Units and Topologies Renewable and Sustainable Energy Reviews 76 2017 268 291
  3. Hua , T. , Ahluwalia , R. , Eudy , L. , Singer , G. et al. Status of Hydrogen Fuel Cell Electric Buses Worldwide Journal of Power Sources 269 2014 975 993
  4. Eberle , U. , Müller , B. , and Helmolt , R. Fuel Cell Electric Vehicles and Hydrogen Infrastructure: Status 2012 Energy & Environmental Science 5 2012 8790 8798
  5. Marx , N. , Boulon , L. , Gustin , F. , Hissel , D. et al. A Review of Multi-Stack and Modular Fuel Cell Systems: Interests, Application Areas and On-Going Research Activities International Journal of Hydrogen Energy 39 2014 12101 12111
  6. Garcia , J.E. , Herrera , D.F. , Boulon , L. , Sicard , P. et al. Power Sharing for Efficiency Optimisation into a Multi Fuel Cell System 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE) Istanbul, Turkey 2014 218 223
  7. Marx , N. , Hissel , D. , Gustin , F. , Boulon , L. et al. On the Sizing and Energy Management of an Hybrid Multistack Fuel Cell—Battery System for Automotive Applications International Journal of Hydrogen Energy 42 2017 1518 1526
  8. Khalatbarisoltani , A. , Cepeda , J.C.O. , Boulon , L. , St-Pierre , D.L. et al. A New Real-Time Centralized Energy Management Strategy for Modular Electric Vehicles 2018 IEEE Vehicle Power and Propulsion Conference (VPPC) Chicago, IL 2018 1 5
  9. Somaiah , B. and Agarwal , V. Distributed Maximum Power Extraction From Fuel Cell Stack Arrays Using Dedicated Power Converters in Series and Parallel Configuration IEEE Transactions on Energy Conversion 31 2016 1442 1451
  10. Zhang , H. , Li , X. , Liu , X. , and Yan , J. Enhancing Fuel Cell Durability for Fuel Cell Plug-In Hybrid Electric Vehicles through Strategic Power Management Applied Energy 241 2019 483 490
  11. Fernandez , A.O.M. , Kandidayeni , M. , Boulon , L. , and Chaoui , H. An Adaptive State Machine Based Energy Management Strategy for a Multi-Stack Fuel Cell Hybrid Electric Vehicle IEEE Transactions on Vehicular Technology 69 2019 220 234
  12. Wang , T. , Li , Q. , Yang , H. , Yin , L. et al. Adaptive Current Distribution Method for Parallel-Connected PEMFC Generation System Considering Performance Consistency Energy Conversion and Management 196 2019 866 877
  13. Khalatbarisoltani , A. , Kandidayeni , M. , Boulon , L. , and Hu , X. Power Allocation Strategy Based on Decentralized Convex Optimization in Modular Fuel Cell Systems for Vehicular Applications IEEE Transactions on Vehicular Technology 69 2020 14563 14574
  14. US Department of Energy 2016
  15. Sutton , R.S. and Barto , A.G. Reinforcement Learning: An Introduction Cambridge, MA MIT Press 2018
  16. Xu , B. and Li , X. A Q-Learning Based Transient Power Optimization Method for Organic Rankine Cycle Waste Heat Recovery System in Heavy Duty Diesel Engine Applications Applied Energy 286 2021 116532
  17. Dulac-Arnold , G. , Mankowitz , D. , and Hester , T. 2019
  18. Zhang , H. , Peng , J. , Tan , H. , Dong , H. et al. A Deep Reinforcement Learning Based Energy Management Framework with Lagrangian Relaxation for Plug-In Hybrid Electric Vehicle IEEE Transactions on Transportation Electrification 7 2020 1146 1160
  19. Koller , T. , Berkenkamp , F. , Turchetta , M. , and Krause , A. Learning-Based Model Predictive Control for Safe Exploration 2018 IEEE Conference on Decision and Control (CDC) Miami Beach, FL 2018 6059 6066
  20. Zanon , M. and Gros , S. Safe Reinforcement Learning Using Robust MPC IEEE Transactions on Automatic Control 66 2020 3638 3652
  21. Wen , L. , Duan , J. , Li , S.E. , Xu , S. et al. Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) Rhodes, Greece 2020 1 7
  22. Jansen , N. , Könighofer , B. , Junges , S. , Serban , A. et al. Safe Reinforcement Learning Using Probabilistic Shields 31st International Conference on Concurrency Theory Vienna, Austria 2020
  23. Alshiekh , M. , Bloem , R. , Ehlers , R. , Könighofer , B. et al. Safe Reinforcement Learning via Shielding Proceedings of the AAAI Conference on Artificial Intelligence New Orleans, LA 2018
  24. Berkenkamp , F. , Turchetta , M. , Schoellig , A.P. , and Krause , A. 2017
  25. Hu , X. , Liu , T. , Qi , X. , and Barth , M. Reinforcement Learning for Hybrid and Plug-In Hybrid Electric Vehicle Energy Management: Recent Advances and Prospects IEEE Industrial Electronics Magazine 13 2019 16 25
  26. Liu , T. , Zou , Y. , Liu , D. , and Sun , F. Reinforcement Learning of Adaptive Energy Management with Transition Probability for a Hybrid Electric Tracked Vehicle IEEE Transactions on Industrial Electronics 62 2015 7837 7846
  27. Liu , T. , Hu , X. , Li , S.E. , and Cao , D. Reinforcement Learning Optimized Look-Ahead Energy Management of a Parallel Hybrid Electric Vehicle IEEE/ASME Transactions on Mechatronics 22 2017 1497 1507
  28. Zou , Y. , Liu , T. , Liu , D. , and Sun , F. Reinforcement Learning-Based Real-Time Energy Management for a Hybrid Tracked Vehicle Applied Energy 171 2016 372 382
  29. Liu , T. , Hu , X. , Hu , W. , and Zou , Y. A Heuristic Planning Reinforcement Learning-Based Energy Management for Power-Split Plug-in Hybrid Electric Vehicles IEEE Transactions on Industrial Informatics 15 2019 6436 6445
  30. Du , G. , Zou , Y. , Zhang , X. , Kong , Z. et al. Intelligent Energy Management for Hybrid Electric Tracked Vehicles Using Online Reinforcement Learning Applied Energy 251 2019 113388
  31. Zhou , Q. , Li , J. , Shuai , B. , Williams , H. et al. Multistep Reinforcement Learning for Model-Free Predictive Energy Management of an Electrified Off-Highway Vehicle Applied Energy 255 2019 113755
  32. Shuai , B. , Zhou , Q. , Li , J. , He , Y. et al. Heuristic Action Execution for Energy Efficient Charge-Sustaining Control of Connected Hybrid Vehicles with Model-Free Double Q-Learning Applied Energy 267 2020 114900
  33. Xu , B. , Hu , X. , Tang , X. , Lin , X. et al. Ensemble Reinforcement Learning-Based Supervisory Control of Hybrid Electric Vehicle for Fuel Economy Improvement IEEE Transactions on Transportation Electrification 6 2020 717 727
  34. Xu , B. , Tang , X. , Hu , X. , Lin , X. et al. Q-Learning-Based Supervisory Control Adaptability Investigation for Hybrid Electric Vehicles IEEE Transactions on Intelligent Transportation Systems 2021 1 10 https://doi.org/10.1109/TITS.2021.3062179
  35. Hu , Y. , Li , W. , Xu , K. , Zahid , T. et al. Energy Management Strategy for a Hybrid Electric Vehicle Based on Deep Reinforcement Learning Applied Sciences 8 2018 187
  36. Wu , Y. , Tan , H. , Peng , J. , Zhang , H. et al. Deep Reinforcement Learning of Energy Management with Continuous Control Strategy and Traffic Information for a Series-Parallel Plug-in Hybrid Electric Bus Applied Energy 247 2019 454 466
  37. Han , X. , He , H. , Wu , J. , Peng , J. et al. Energy Management Based on Reinforcement Learning with Double Deep Q-Learning for a Hybrid Electric Tracked Vehicle Applied Energy 254 2019 113708
  38. Guo , X. , Liu , T. , Tang , B. , Tang , X. et al. Transfer Deep Reinforcement Learning-Enabled Energy Management Strategy for Hybrid Tracked Vehicle IEEE Access 8 2020 165837 165848
  39. Lian , R. , Tan , H. , Peng , J. , Li , Q. et al. Cross-Type Transfer for Deep Reinforcement Learning Based Hybrid Electric Vehicle Energy Management IEEE Transactions on Vehicular Technology 69 2020 8367 8380
  40. Zhou , J. , Xue , S. , Xue , Y. , Liao , Y. et al. A Novel Energy Management Strategy of Hybrid Electric Vehicle via an Improved TD3 Deep Reinforcement Learning Energy 224 2021 120118
  41. Liu , T. , Tang , X. , Hu , X. , Tan , W. et al. 2020
  42. Lin , W.-S. and Zheng , C.-H. Energy Management of a Fuel Cell/Ultracapacitor Hybrid Power System Using an Adaptive Optimal-Control Method Journal of Power Sources 196 2011 3280 3289
  43. Hsu , R.C. , Chen , S. , Chen , W. , and Liu , C. A Reinforcement Learning Based Dynamic Power Management for Fuel Cell Hybrid Electric Vehicle 2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS) Sapporo, Japan 2016 460 464
  44. Yuan , J. , Yang , L. , and Chen , Q. Intelligent Energy Management Strategy Based on Hierarchical Approximate Global Optimization for Plug-in Fuel Cell Hybrid Electric Vehicles International Journal of Hydrogen Energy 43 2018 8063 8078
  45. Reddy , N.P. , Pasdeloup , D. , Zadeh , M.K. , and Skjetne , R. An Intelligent Power and Energy Management System for Fuel Cell/Battery Hybrid Electric Vehicle Using Reinforcement Learning 2019 IEEE Transportation Electrification Conference and Expo (ITEC) Novi, MI 2019 1 6
  46. Lin , X. , Zhou , B. , and Xia , Y. Online Recursive Power Management Strategy based on the Reinforcement Learning Algorithm with Cosine Similarity and a Forgetting Factor IEEE Transactions on Industrial Electronics 68 2020 5013 5023
  47. Sun , H. , Fu , Z. , Tao , F. , Zhu , L. et al. Data-Driven Reinforcement-Learning-Based Hierarchical Energy Management Strategy for Fuel Cell/Battery/Ultracapacitor Hybrid Electric Vehicles Journal of Power Sources 455 2020 227964
  48. Soltani , A.K. , Kandidayeni , M. , Boulon , L. , and St-Pierre , D.L. Modular Energy Systems in Vehicular Applications Energy Procedia 162 2019 14 23
  49. Logenthiran , T. , Srinivasan , D. , and Khambadkone , A.M. Multiagent System for Energy Resource Scheduling of Integrated Microgrids in a Distributed System Electric Power Systems Research 81 2011 138 148
  50. Busoniu , L. , Babuska , R. , and De Schutter , B. A Comprehensive Survey of Multiagent Reinforcement Learning IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38 2008 156 172
  51. Zhang , K. , Yang , Z. , and Başar , T. 2019
  52. Solano , J. , Jemei , S. , Boulon , L. , Silva , L. et al. IEEE VTS Motor Vehicles Challenge 2020—Energy Management of a Fuel Cell/Ultracapacitor/Lead-Acid Battery Hybrid Electric Vehicle 2019 IEEE Vehicle Power and Propulsion Conference (VPPC) Hanoi, Vietnam 2019 1 6
  53. Martinez , J.S. , Hissel , D. , Péra , M.-C. , and Amiet , M. Practical Control Structure and Energy Management of a Testbed Hybrid Electric Vehicle IEEE Transactions on Vehicular Technology 60 2011 4139 4152
  54. Kandidayeni , M. , Fernandez , A.M. , Boulon , L. , and Kelouwani , S. Efficiency Upgrade of Hybrid Fuel Cell Vehicles’ Energy Management Strategies by Online Systemic Management of Fuel Cell IEEE Transactions on Industrial Electronics 68 2020 4941 4953
  55. Watkins , C.J. and Dayan , P. Q-Learning Machine Learning 8 1992 279 292
  56. Chen , H. , Pei , P. , and Song , M. Lifetime Prediction and the Economic Lifetime of Proton Exchange Membrane Fuel Cells Applied Energy 142 2015 154 163
  57. Herr , N. , Nicod , J.-M. , Varnier , C. , Jardin , L. et al. Decision Process to Manage Useful Life of Multi-Stacks Fuel Cell Systems under Service Constraint Renewable Energy 105 2017 590 600
  58. Satyapal , S. U.S. Department of Energy Hydrogen and Fuel Cell Technology Overview 14th International Hydrogen and Fuel Cell Expo (FC EXPO 2018) Tokyo 2018
  59. USDRIVE 2017
  60. Mongird , K. , Viswanathan , V.V. , Balducci , P.J. , Alam , M.J.E. et al. 2019
  61. Liessner , R. , Schmitt , J. , Dietermann , A. , and Bäker , B. Hyperparameter Optimization for Deep Reinforcement Learning in Vehicle Energy Management ICAART Prague, Czech Republic 2019 2 134 144
  62. Fernandez , F.C. and Caarls , W. Parameters Tuning and Optimization for Reinforcement Learning Algorithms Using Evolutionary Computing 2018 International Conference on Information Systems and Computer Science (INCISCOS) Quito, Ecuador 2018 301 305
  63. Kandidayeni , M. , Fernandez , A.O.M. , Khalatbarisoltani , A. , Boulon , L. et al. An Online Energy Management Strategy for a Fuel Cell/Battery Vehicle Considering the Driving Pattern and Performance Drift Impacts IEEE Transactions on Vehicular Technology 68 2019 11427 11438

Cited By