OPTIMIZATION AND SCHEDULING OF ELECTROTHERMAL HYDROGEN COUPLED COMPREHENSIVE ENERGY SYSTEM BASED ON PPO ALGORITHM

Liang Tao; Zhang Xiaochan; Tan Jianxin; Jing Yanwei; Lyu Liangnian

doi:10.19912/j.0254-0096.tynxb.2023-1107

PDF(4721 KB)

Acta Energiae Solaris Sinica ›› 2024, Vol. 45 ›› Issue (11) : 73-83. DOI: 10.19912/j.0254-0096.tynxb.2023-1107

OPTIMIZATION AND SCHEDULING OF ELECTROTHERMAL HYDROGEN COUPLED COMPREHENSIVE ENERGY SYSTEM BASED ON PPO ALGORITHM

Liang Tao¹, Zhang Xiaochan¹, Tan Jianxin², Jing Yanwei², Lyu Liangnian³

Author information +

History +

Abstract

In order to change the traditional operation mode of “source with load” and increase energy storage, and to realize the coordination and interaction of power grid, load, energy storage and other links, this paper establishes an electric-thermal-hydrogen coupled integrated energy system （ETHC-IES） for optimal scheduling. The use of hydrogen energy storage to realize the safe and stable operation of a new type of integrated energy system “source network storage” has become a current research hotspot. The aim of this paper is to reduce the operation cost of IES and the waste of wind and light. The ETHC-IES optimal scheduling problem is transformed into a Markov decision process （MDP）, and an optimal scheduling method based on continuous action proximal policy optimization （PPO） is proposed for the integrated energy system. Firstly, a mathematical model of each part of the electric hydrogen storage system is established. Then, the model is solved using a deep learning proximal policy optimization algorithm with the optimization objectives of economy and reduction of wind and light waste. The action space, state space, and reward function of the deep reinforcement learning model are set up. Intelligent agents are trained and learned to achieve dynamic scheduling optimization decisions for ETHC-IES. Finally, the effectiveness and applicability of the proposed model and method are verified by simulation.

Key words

reinforcement learning / energy storage / renewable energy / proximal policy optimization / ETHC-IES

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

Liang Tao, Zhang Xiaochan, Tan Jianxin, Jing Yanwei, Lyu Liangnian. OPTIMIZATION AND SCHEDULING OF ELECTROTHERMAL HYDROGEN COUPLED COMPREHENSIVE ENERGY SYSTEM BASED ON PPO ALGORITHM[J]. Acta Energiae Solaris Sinica. 2024, 45(11): 73-83 https://doi.org/10.19912/j.0254-0096.tynxb.2023-1107

References

[1] SUN T, ZHAO J Y, QIU Y K, et al.Research on multi-objective optimal scheduling strategy of new energy hydrogen production system[C]//2022 Power System and Green Energy Conference (PSGEC). Shanghai, China, 2022: 289-295.
[2] 李亮荣, 彭建, 付兵, 等. 碳中和愿景下绿色制氢技术发展趋势及应用前景分析[J]. 太阳能学报, 2022, 43(6): 508-520.
LI L R, PENG J, FU B, et al.Development trend and application prospect of green hydrogen production technologies under carbon neutrality vision[J]. Acta energiae solaris sinica, 2022, 43(6): 508-520.
[3] 王永真, 康利改, 张靖, 等. 综合能源系统的发展历程、典型形态及未来趋势[J]. 太阳能学报, 2021, 42(8): 84-95.
WANG Y Z, KANG L G, ZHANG J, et al.Development history, typical form and future trend of integrated energy system[J]. Acta energiae solaris sinica, 2021, 42(8): 84-95.
[4] LI H, HOU K, XU X D, et al.Probabilistic energy flow calculation for regional integrated energy system considering cross-system failures[J]. Applied energy, 2022, 308: 118326.
[5] 汪露露, 吴红斌, 周亦尧. 基于供能可靠性的综合能源系统优化配置[J]. 太阳能学报, 2021, 42(12): 395-400.
WANG L L, WU H B, ZHOU Y Y.Optimal configuration of integrated energy system based on energy supply reliability[J]. Acta energiae solaris sinica, 2021, 42(12): 395-400.
[6] LYU G Y, CAO B, JUN L, et al.Optimal scheduling of integrated energy system under the background of carbon neutrality[J]. Energy reports, 2022, 8: 1236-1248.
[7] MA W X, DENG W, PEI W, et al.Operation optimization of electric power-hot water-steam integrated energy system[J]. Energy reports, 2022, 8: 475-482.
[8] HOU H, CHEN Y, LIU P, et al.Multisource energy storage system optimal dispatch among electricity hydrogen and heat networks from the energy storage operator prospect[J]. IEEE transactions on industry applications, 2022, 58(2): 2825-2835.
[9] SHI M G, WANG H, LYU C, et al.A hybrid model of energy scheduling for integrated multi-energy microgrid with hydrogen and heat storage system[J]. Energy reports, 2021, 7: 357-368.
[10] 马志侠, 张林鍹, 郑兴, 等. 基于PEMFC-P2G与风光不确定的综合能源系统优化调度[J]. 太阳能学报, 2022, 43(6): 441-447.
MA Z X, ZHANG L X, ZHENG X, et al.Optimal scheduling of integrated energy system based on PEMFC-P2G and inpact of wind power and photovoltaic uncertainty[J]. Acta energiae solaris sinica, 2022, 43(6): 441-447.
[11] 帅挽澜, 朱自伟, 李雪萌, 等. 考虑风电消纳的综合能源系统“源-网-荷-储” 协同优化运行[J]. 电力系统保护与控制, 2021, 49(19): 18-26.
SHUAI W L, ZHU Z W, LI X M, et al.“Source-network-load-storage” coordinated optimization operation for an integrated energy system considering wind power consumption[J]. Power system protection and control, 2021, 49(19): 18-26.
[12] 侯慧, 刘鹏, 黄亮, 等. 考虑不确定性的电-热-氢综合能源系统规划[J]. 电工技术学报, 2021, 36(增刊1): 133-144.
HOU H, LIU P, HUANG L, et al.Planning of electricity-heat-hydrogen integrated energy system considering uncertainties[J]. Transactions of China Electrotechnical Society, 2021, 36(S1): 133-144.
[13] WU M, XU J Z, ZENG L J, et al.Two-stage robust optimization model for park integrated energy system based on dynamic programming[J]. Applied energy, 2022, 308: 118249.
[14] ZHU Z S, WENG Z M, ZHENG H L.Optimal operation of a microgrid with hydrogen storage based on deep reinforcement learning[J]. Electronics, 2022, 11(2): 196.
[15] 乔骥, 王新迎, 张擎, 等. 基于柔性行动器-评判器深度强化学习的电-气综合能源系统优化调度[J]. 中国电机工程学报, 2021, 41(3): 819-833.
QIAO J, WANG X Y, ZHANG Q, et al.Optimal dispatch of integrated electricity-gas system with soft actor-critic deep reinforcement learning[J]. Proceedings of the CSEE, 2021, 41(3): 819-833.
[16] 蔺伟山, 王小君, 孙庆凯, 等. 不确定性环境下基于深度强化学习的综合能源系统动态调度[J]. 电力系统保护与控制, 2022, 50(18): 50-60.
LIN W S, WANG X J, SUN Q K, et al.Dynamic dispatch of an integrated energy system based on deep reinforcement learning in an uncertain environment[J]. Power system protection and control, 2022, 50(18): 50-60.
[17] 石文喆, 李冰洁, 尤培培, 等. 基于深度强化学习的建筑能源系统优化策略[J]. 中国电力, 2023, 56(6): 114-122.
SHI W Z, LI B J, YOU P P, et al.Optimization strategy of building energy system based on deep reinforcement learning[J]. Electric power, 2023, 56(6): 114-122.
[18] 杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
YANG T, ZHAO L Y, LIU Y C, et al.Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of electric power systems, 2021, 45(5): 39-47.
[19] JIANG W Y, LIU Y Q, FANG G H, et al.Research on short-term optimal scheduling of hydro-wind-solar multi-energy power system based on deep reinforcement learning[J]. Journal of cleaner production, 2023, 385: 135704.
[20] ZHANG B, HU W H, CAO D, et al.Deep reinforcement learning-based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy[J]. Energy conversion and management, 2019, 202: 112199.
[21] 梁涛, 刘伟, 曹欣, 等.基于深度确定性策略梯度算法的可再生能源大规模制氢系统能量调度[J/OL].电网技术:1-16[2023-07-02].https://doi.org/10.13335/j.1000-3673.pst.2023.0211.
LIANG T, LIU W, CAO X, et al.Research on energy scheduling of renewable energy large-scale hydrogen production system based on deep deterministic strategy gradient algorithm[J/OL]. Power system technology. 1-16[2023-07-02].https://doi.org/10.13335/j.1000-3673.pst.2023.0211.
[22] 祝荣, 任永峰, 孟庆天, 等. 基于合作博弈的综合能源系统电-热-气协同优化运行策略[J]. 太阳能学报, 2022, 43(4): 20-29.
ZHU R, REN Y F, MENG Q T, et al.Electricity-heat-gas cooperative optimal operation strategy of integrated energy system based on cooperative game[J]. Acta energiae solaris sinica, 2022, 43(4): 20-29.
[23] HEMMATI R, MEHRJERDI H, BORNAPOUR M.Hybrid hydrogen-battery storage to smooth solar energy volatility and energy arbitrage considering uncertain electrical-thermal loads[J]. Renewable energy, 2020, 154: 1180-1187.
[24] PANG Y, PAN L, ZHANG J M, et al.Integrated sizing and scheduling of an off-grid integrated energy system for an isolated renewable energy hydrogen refueling station[J]. Applied energy, 2022, 323: 119573.
[25] 王磊, 姜涛, 宋丹, 等. 基于灵活热电比的区域综合能源系统多目标优化调度[J]. 电力系统保护与控制, 2021, 49(8): 151-159.
WANG L, JIANG T, SONG D, et al.Multi-objective optimal dispatch of a regional integrated energy system based on a flexible heat-to-electric ratio[J]. Power system protection and control, 2021, 49(8): 151-159.
[26] 罗文健, 张靖, 何宇, 等. 基于优势柔性策略-评价算法和迁移学习的区域综合能源系统优化调度[J]. 电网技术, 2023, 47(4): 1601-1615.
LUO W J, ZHANG J, HE Y, et al.Optimal scheduling of regional integrated energy system based on advantage learning soft actor-critic algorithm and transfer learning[J]. Power system technology, 2023, 47(4): 1601-1615.
[27] GONG X, DONG F F, MOHAMED M A, et al.A secured energy management architecture for smart hybrid microgrids considering PEM-fuel cell and electric vehicles[J]. IEEE access, 2020, 8: 47807-47823.