NASH-Q REINFORCEMENT LEARNING SCHEDULING STRATEGY FOR VIRTUAL POWER PLANTS CONTAINING VIRTUAL INERTIA

Liu Dongqi; Qian Yiheng; Zhang Xi

doi:10.19912/j.0254-0096.tynxb.2023-1042

PDF(2012 KB)

Acta Energiae Solaris Sinica ›› 2024, Vol. 45 ›› Issue (11) : 15-24. DOI: 10.19912/j.0254-0096.tynxb.2023-1042

NASH-Q REINFORCEMENT LEARNING SCHEDULING STRATEGY FOR VIRTUAL POWER PLANTS CONTAINING VIRTUAL INERTIA

Liu Dongqi, Qian Yiheng, Zhang Xi

Author information +

History +

Abstract

Currently, there is a lack of research on the problem of efficiently utilising distributed power sources that are widely connected to the grid. In this paper, by combining game theory and reinforcement learning, a Nash-Q reinforcement learning scheduling strategy with virtual inertia for virtual power plants is proposed to solve the problem of coordinated optimal scheduling of distributed power sources with high stochasticity and uncertainty. Firstly, a hybrid objective scheduling model of virtual power plant with integrated consideration of carbon emission, economy, output and inertia constraints is established, and a Nash equilibrium model is further constructed. Then, multi-intelligentsia are defined for the output of gas turbine unit, wind turbine unit, storage battery unit and carbon trading volume, and the state space, action space and reward function of multi-intelligence are constructed, and the state space, action space and reward function of the multi-intelligentsia are continuously learnt to update through Markov decision-making value function. Finally, the optimal scheduling policy for multiple time scales is output through online derivation. The results show that the Nash-Q reinforcement learning scheduling strategy improves the revenue by 9.7% and reduces the carbon emission by 13.6% compared with the traditional scheduling method, and effectively improves the inertia support capacity of the virtual power plant by using the virtual inertia of the storage battery packs, which achieves the low-carbon, high-efficiency, safe and economic and operation of the virtual power plant.

Key words

distributed generation / virtual power plants / reinforcement learning / game theory / virtual inertia

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

Liu Dongqi, Qian Yiheng, Zhang Xi. NASH-Q REINFORCEMENT LEARNING SCHEDULING STRATEGY FOR VIRTUAL POWER PLANTS CONTAINING VIRTUAL INERTIA[J]. Acta Energiae Solaris Sinica. 2024, 45(11): 15-24 https://doi.org/10.19912/j.0254-0096.tynxb.2023-1042

References

[1] 周任军, 孙洪, 唐夏菲, 等. 双碳量约束下风电-碳捕集虚拟电厂低碳经济调度[J]. 中国电机工程学报, 2018, 38(6): 1675-1683.
ZHOU R J, SUN H, TANG X F, et al.Low-carbon economic dispatch based on virtual power plant made up of carbon capture unit and wind power under double carbon constraint[J]. Proceedings of the CSEE, 2018, 38(6): 1675-1683.
[2] 张新民, 郭铭海, 林亚培, 等. 考虑灵活性的含分布式光伏配电网双层优化调度方法[J]. 电力科学与技术学报, 2021, 36(3): 56-66.
ZHANG X M, GUO M H, LIN Y P, et al.A bi-layer optimal dispatch approach for distribution networks with distributed photovoltaic considering the flexibility[J]. Journal of electric power science and technology, 2021, 36(3): 56-66.
[3] 汪莞乔, 苏剑, 潘娟, 等. 虚拟电厂通信网络架构及关键技术研究展望[J]. 电力系统自动化, 2022, 46(18): 15-25.
WANG G Q, SU J, PAN J, et al.Prospect of research on communication network architecture and key technologies for virtual power plant[J]. Automation of electric power systems, 2022, 46(18): 15-25.
[4] 陈会来, 张海波, 王兆霖. 不同类型虚拟电厂市场及调度特性参数聚合算法研究综述[J]. 中国电机工程学报, 2023, 43(1): 15-27.
CHEN H L, ZHANG H B, WANG Z L.A review of market and scheduling characteristic parameter aggregation algorithm of different types of virtual power plants[J]. Proceedings of the CSEE, 2023, 43(1): 15-27.
[5] 师阳, 李宏伟, 陈继开, 等. 计及激励型需求响应的热电互联虚拟电厂优化调度[J]. 太阳能学报, 2023, 44(4): 349-358.
SHI Y, LI H W, CHEN J K, et al.Optimal scheduling of thermoelectric interconnection virtual power plant considering incentive demand response[J]. Acta energiae solaris sinica, 2023, 44(4): 349-358.
[6] 袁桂丽, 贾新潮, 陈少梁, 等. 虚拟电厂源-荷协调多目标优化调度[J]. 太阳能学报, 2021, 42(5): 105-112.
YUAN G L, JIA X C, CHEN S L, et al.Multiobjective optimal dispatch considering source-load coordination for virtual power plant[J]. Acta energiae solaris sinica, 2021, 42(5): 105-112.
[7] 孙晶琪, 王愿, 郭晓慧, 等. 考虑环境外部性和风光出力不确定性的虚拟电厂运行优化[J]. 电力系统自动化, 2022, 46(8): 50-59.
SUN J Q, WANG Y, GUO X H, et al.Optimal operation of virtual power plant considering environmental externality and output uncertainty of wind and photovoltaic power[J]. Automation of electric power systems, 2022, 46(8): 50-59.
[8] ZHONG W L, TZOUNAS G, LIU M Y, et al.On-line inertia estimation of Virtual Power Plants[J]. Electric power systems research, 2022, 212: 108336.
[9] SINGH K, ZAHEERUDDIN. Enhancement of frequency regulation in tidal turbine power plant using virtual inertia from capacitive energy storage system[J]. Journal of energy storage, 2021, 35: 102332.
[10] LIN L, GUAN X, PENG Y, et al.Deep reinforcement learning for economic dispatch of virtual power plant in internet of energy[J]. IEEE internet of things journal, 2020, 7(7): 6288-6301.
[11] WANG Y, AI X, TAN Z F, et al.Interactive dispatch modes and bidding strategy of multiple virtual power plants based on demand response and game theory[J]. IEEE transactions on smart grid, 2016, 7(1): 510-519.
[12] DE CARVALHO BENTO G, BOUZA ALLENDE G, PEREIRA Y R L. A newton-like method for variable order vector optimization problems[J]. Journal of optimization theory and applications, 2018, 177(1): 201-221.
[13] ZHANG L R, XU J J, LIU Y, et al.Particle swarm optimization algorithm with multi-strategies for delay scheduling[J]. Neural processing letters, 2022, 54(5): 4563-4592.
[14] 刘俊峰, 王晓生, 卢俊菠, 等. 基于多主体博弈和强化学习的多微网系统协同优化研究[J]. 电网技术, 2022, 46(7): 2722-2732.
LIU J F, WANG X S, LU J B, et al.Collaborative optimization of multi-microgrid system based on multi-agent game and reinforcement learning[J]. Power system technology, 2022, 46(7): 2722-2732.
[15] ZHANG B, HU W H, LI J H, et al.Dynamic energy conversion and management strategy for an integrated electricity and natural gas system with renewable energy: deep reinforcement learning approach[J]. Energy conversion and management, 2020, 220: 113063.
[16] 鲁刚, 元博, 赵琮皓, 等. 计及系统惯量需求的发电容量市场定价方法[J]. 电力系统自动化, 2023, 47(14): 12-20.
LU G, YUAN B, ZHAO C H, et al.Pricing method for generation capacity market considering system inertia demand[J]. Automation of electric power systems,2023, 47(14): 12-20.
[17] CHEN P W, QI C C, CHEN X.Virtual inertia estimation method of DFIG-based wind farm with additional frequency control[J]. Journal of modern power systems and clean energy, 2021, 9(5): 1076-1087.
[18] 刘中建, 周明, 李昭辉, 等. 高比例新能源电力系统的惯量控制技术与惯量需求评估综述[J]. 电力自动化设备, 2021, 41(12): 1-11, 53.
LIU Z J, ZHOU M, LI Z H, et al.Review of inertia control technology and requirement evaluation in renewable-dominant power system[J]. Electric power automation equipment, 2021, 41(12): 1-11, 53.
[19] 彭春华, 陈思畏, 徐佳璐, 等. 综合能源系统混合时间尺度多目标强化学习低碳经济调度[J]. 电网技术, 2022, 46(12): 4914-4923.
PENG C H, CHEN S W, XU J L, et al.Low carbon economic scheduling for integrated energy systems with mixed timescale & multi-objective reinforcement learning[J]. Power system technology, 2022, 46(12): 4914-4923.
[20] 胡丹尔, 彭勇刚, 韦巍, 等. 多时间尺度的配电网深度强化学习无功优化策略[J]. 中国电机工程学报, 2022, 42(14): 5034-5044.
HU D E, PENG Y G, WEI W, et al.Multi-timescale deep reinforcement learning for reactive power optimization of distribution network[J]. Proceedings of the CSEE, 2022, 42(14): 5034-5044.
[21] 王芸芸, 马志程, 周强, 等. 兼顾公平性的多能源合作博弈优化调度[J]. 太阳能学报, 2022, 43(10): 482-492.
WANG Y Y, MA Z C, ZHOU Q, et al.Multi energy cooperative game optimal scheduling considering fairness[J]. Acta energiae solaris sinica, 2022, 43(10): 482-492.
[22] 孙庆凯, 王小君, 王怡, 等. 基于多智能体Nash-Q强化学习的综合能源市场交易优化决策[J]. 电力系统自动化, 2021, 45(16): 124-133.
SUN Q K, WANG X J, WANG Y, et al.Optimal trading decision-making for integrated energy market based on multi-agent Nash-Q reinforcement learning[J]. Automation of electric power systems, 2021, 45(16): 124-133.