基于PPO算法的电热氢耦合综合能源系统优化调度

梁涛; 张晓婵; 谭建鑫; 井延伟; 吕梁年

doi:10.19912/j.0254-0096.tynxb.2023-1107

PDF(4721 KB)

太阳能学报 ›› 2024, Vol. 45 ›› Issue (11) : 73-83. DOI: 10.19912/j.0254-0096.tynxb.2023-1107

基于PPO算法的电热氢耦合综合能源系统优化调度

梁涛¹, 张晓婵¹, 谭建鑫², 井延伟², 吕梁年³

作者信息 +

OPTIMIZATION AND SCHEDULING OF ELECTROTHERMAL HYDROGEN COUPLED COMPREHENSIVE ENERGY SYSTEM BASED ON PPO ALGORITHM

Liang Tao¹, Zhang Xiaochan¹, Tan Jianxin², Jing Yanwei², Lyu Liangnian³

Author information +

文章历史 +

摘要

为改变“源随荷动”的传统运作模式并增加储能,实现能源网、负荷、储能等各环节协调互动,建立电热氢耦合综合能源系统（ETHC-IES）优化调度,其中应用氢储能实现安全稳定运行的“源-网-荷-储”的新型综合能源系统成为目前的研究热点。以降低综合能源系统运行成本并减少弃风弃光为目标,将ETHC-IES优化调度问题转换为马尔可夫决策过程（MDP）,提出应用基于连续动作的近端策略优化算法（PPO）的综合能源系统优化调度方法。首先建立电热氢储能各部分的数学模型,综合考虑功率平衡,安全状态等约束条件,然后采用PPO算法对模型进行求解,以提高经济性和减少弃风弃光为优化目标,重新设计深度强化学习模型的动作空间、状态空间、奖励函数等,智能体通过训练学习实现ETHC-IES的动态调度优化决策。最后,通过仿真验证所提出模型和优化方法的有效性和优越性。

Abstract

In order to change the traditional operation mode of “source with load” and increase energy storage, and to realize the coordination and interaction of power grid, load, energy storage and other links, this paper establishes an electric-thermal-hydrogen coupled integrated energy system （ETHC-IES） for optimal scheduling. The use of hydrogen energy storage to realize the safe and stable operation of a new type of integrated energy system “source network storage” has become a current research hotspot. The aim of this paper is to reduce the operation cost of IES and the waste of wind and light. The ETHC-IES optimal scheduling problem is transformed into a Markov decision process （MDP）, and an optimal scheduling method based on continuous action proximal policy optimization （PPO） is proposed for the integrated energy system. Firstly, a mathematical model of each part of the electric hydrogen storage system is established. Then, the model is solved using a deep learning proximal policy optimization algorithm with the optimization objectives of economy and reduction of wind and light waste. The action space, state space, and reward function of the deep reinforcement learning model are set up. Intelligent agents are trained and learned to achieve dynamic scheduling optimization decisions for ETHC-IES. Finally, the effectiveness and applicability of the proposed model and method are verified by simulation.

导出引用

梁涛, 张晓婵, 谭建鑫, 井延伟, 吕梁年. 基于PPO算法的电热氢耦合综合能源系统优化调度[J]. 太阳能学报. 2024, 45(11): 73-83 https://doi.org/10.19912/j.0254-0096.tynxb.2023-1107

Liang Tao, Zhang Xiaochan, Tan Jianxin, Jing Yanwei, Lyu Liangnian. OPTIMIZATION AND SCHEDULING OF ELECTROTHERMAL HYDROGEN COUPLED COMPREHENSIVE ENERGY SYSTEM BASED ON PPO ALGORITHM[J]. Acta Energiae Solaris Sinica. 2024, 45(11): 73-83 https://doi.org/10.19912/j.0254-0096.tynxb.2023-1107

中图分类号： TK519

参考文献

[1] SUN T, ZHAO J Y, QIU Y K, et al.Research on multi-objective optimal scheduling strategy of new energy hydrogen production system[C]//2022 Power System and Green Energy Conference (PSGEC). Shanghai, China, 2022: 289-295.
[2] 李亮荣, 彭建, 付兵, 等. 碳中和愿景下绿色制氢技术发展趋势及应用前景分析[J]. 太阳能学报, 2022, 43(6): 508-520.
LI L R, PENG J, FU B, et al.Development trend and application prospect of green hydrogen production technologies under carbon neutrality vision[J]. Acta energiae solaris sinica, 2022, 43(6): 508-520.
[3] 王永真, 康利改, 张靖, 等. 综合能源系统的发展历程、典型形态及未来趋势[J]. 太阳能学报, 2021, 42(8): 84-95.
WANG Y Z, KANG L G, ZHANG J, et al.Development history, typical form and future trend of integrated energy system[J]. Acta energiae solaris sinica, 2021, 42(8): 84-95.
[4] LI H, HOU K, XU X D, et al.Probabilistic energy flow calculation for regional integrated energy system considering cross-system failures[J]. Applied energy, 2022, 308: 118326.
[5] 汪露露, 吴红斌, 周亦尧. 基于供能可靠性的综合能源系统优化配置[J]. 太阳能学报, 2021, 42(12): 395-400.
WANG L L, WU H B, ZHOU Y Y.Optimal configuration of integrated energy system based on energy supply reliability[J]. Acta energiae solaris sinica, 2021, 42(12): 395-400.
[6] LYU G Y, CAO B, JUN L, et al.Optimal scheduling of integrated energy system under the background of carbon neutrality[J]. Energy reports, 2022, 8: 1236-1248.
[7] MA W X, DENG W, PEI W, et al.Operation optimization of electric power-hot water-steam integrated energy system[J]. Energy reports, 2022, 8: 475-482.
[8] HOU H, CHEN Y, LIU P, et al.Multisource energy storage system optimal dispatch among electricity hydrogen and heat networks from the energy storage operator prospect[J]. IEEE transactions on industry applications, 2022, 58(2): 2825-2835.
[9] SHI M G, WANG H, LYU C, et al.A hybrid model of energy scheduling for integrated multi-energy microgrid with hydrogen and heat storage system[J]. Energy reports, 2021, 7: 357-368.
[10] 马志侠, 张林鍹, 郑兴, 等. 基于PEMFC-P2G与风光不确定的综合能源系统优化调度[J]. 太阳能学报, 2022, 43(6): 441-447.
MA Z X, ZHANG L X, ZHENG X, et al.Optimal scheduling of integrated energy system based on PEMFC-P2G and inpact of wind power and photovoltaic uncertainty[J]. Acta energiae solaris sinica, 2022, 43(6): 441-447.
[11] 帅挽澜, 朱自伟, 李雪萌, 等. 考虑风电消纳的综合能源系统“源-网-荷-储” 协同优化运行[J]. 电力系统保护与控制, 2021, 49(19): 18-26.
SHUAI W L, ZHU Z W, LI X M, et al.“Source-network-load-storage” coordinated optimization operation for an integrated energy system considering wind power consumption[J]. Power system protection and control, 2021, 49(19): 18-26.
[12] 侯慧, 刘鹏, 黄亮, 等. 考虑不确定性的电-热-氢综合能源系统规划[J]. 电工技术学报, 2021, 36(增刊1): 133-144.
HOU H, LIU P, HUANG L, et al.Planning of electricity-heat-hydrogen integrated energy system considering uncertainties[J]. Transactions of China Electrotechnical Society, 2021, 36(S1): 133-144.
[13] WU M, XU J Z, ZENG L J, et al.Two-stage robust optimization model for park integrated energy system based on dynamic programming[J]. Applied energy, 2022, 308: 118249.
[14] ZHU Z S, WENG Z M, ZHENG H L.Optimal operation of a microgrid with hydrogen storage based on deep reinforcement learning[J]. Electronics, 2022, 11(2): 196.
[15] 乔骥, 王新迎, 张擎, 等. 基于柔性行动器-评判器深度强化学习的电-气综合能源系统优化调度[J]. 中国电机工程学报, 2021, 41(3): 819-833.
QIAO J, WANG X Y, ZHANG Q, et al.Optimal dispatch of integrated electricity-gas system with soft actor-critic deep reinforcement learning[J]. Proceedings of the CSEE, 2021, 41(3): 819-833.
[16] 蔺伟山, 王小君, 孙庆凯, 等. 不确定性环境下基于深度强化学习的综合能源系统动态调度[J]. 电力系统保护与控制, 2022, 50(18): 50-60.
LIN W S, WANG X J, SUN Q K, et al.Dynamic dispatch of an integrated energy system based on deep reinforcement learning in an uncertain environment[J]. Power system protection and control, 2022, 50(18): 50-60.
[17] 石文喆, 李冰洁, 尤培培, 等. 基于深度强化学习的建筑能源系统优化策略[J]. 中国电力, 2023, 56(6): 114-122.
SHI W Z, LI B J, YOU P P, et al.Optimization strategy of building energy system based on deep reinforcement learning[J]. Electric power, 2023, 56(6): 114-122.
[18] 杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
YANG T, ZHAO L Y, LIU Y C, et al.Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of electric power systems, 2021, 45(5): 39-47.
[19] JIANG W Y, LIU Y Q, FANG G H, et al.Research on short-term optimal scheduling of hydro-wind-solar multi-energy power system based on deep reinforcement learning[J]. Journal of cleaner production, 2023, 385: 135704.
[20] ZHANG B, HU W H, CAO D, et al.Deep reinforcement learning-based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy[J]. Energy conversion and management, 2019, 202: 112199.
[21] 梁涛, 刘伟, 曹欣, 等.基于深度确定性策略梯度算法的可再生能源大规模制氢系统能量调度[J/OL].电网技术:1-16[2023-07-02].https://doi.org/10.13335/j.1000-3673.pst.2023.0211.
LIANG T, LIU W, CAO X, et al.Research on energy scheduling of renewable energy large-scale hydrogen production system based on deep deterministic strategy gradient algorithm[J/OL]. Power system technology. 1-16[2023-07-02].https://doi.org/10.13335/j.1000-3673.pst.2023.0211.
[22] 祝荣, 任永峰, 孟庆天, 等. 基于合作博弈的综合能源系统电-热-气协同优化运行策略[J]. 太阳能学报, 2022, 43(4): 20-29.
ZHU R, REN Y F, MENG Q T, et al.Electricity-heat-gas cooperative optimal operation strategy of integrated energy system based on cooperative game[J]. Acta energiae solaris sinica, 2022, 43(4): 20-29.
[23] HEMMATI R, MEHRJERDI H, BORNAPOUR M.Hybrid hydrogen-battery storage to smooth solar energy volatility and energy arbitrage considering uncertain electrical-thermal loads[J]. Renewable energy, 2020, 154: 1180-1187.
[24] PANG Y, PAN L, ZHANG J M, et al.Integrated sizing and scheduling of an off-grid integrated energy system for an isolated renewable energy hydrogen refueling station[J]. Applied energy, 2022, 323: 119573.
[25] 王磊, 姜涛, 宋丹, 等. 基于灵活热电比的区域综合能源系统多目标优化调度[J]. 电力系统保护与控制, 2021, 49(8): 151-159.
WANG L, JIANG T, SONG D, et al.Multi-objective optimal dispatch of a regional integrated energy system based on a flexible heat-to-electric ratio[J]. Power system protection and control, 2021, 49(8): 151-159.
[26] 罗文健, 张靖, 何宇, 等. 基于优势柔性策略-评价算法和迁移学习的区域综合能源系统优化调度[J]. 电网技术, 2023, 47(4): 1601-1615.
LUO W J, ZHANG J, HE Y, et al.Optimal scheduling of regional integrated energy system based on advantage learning soft actor-critic algorithm and transfer learning[J]. Power system technology, 2023, 47(4): 1601-1615.
[27] GONG X, DONG F F, MOHAMED M A, et al.A secured energy management architecture for smart hybrid microgrids considering PEM-fuel cell and electric vehicles[J]. IEEE access, 2020, 8: 47807-47823.