微网储能侧DC-DC变换器的强化学习自抗扰控制策略

马幼捷; 刘熠铭; 周雪松; 王博; 陶珑; 问虎龙

doi:10.19912/j.0254-0096.tynxb.2023-1955

PDF(3313 KB)

太阳能学报 ›› 2025, Vol. 46 ›› Issue (3) : 63-72. DOI: 10.19912/j.0254-0096.tynxb.2023-1955

微网储能侧DC-DC变换器的强化学习自抗扰控制策略

马幼捷¹, 刘熠铭¹, 周雪松¹, 王博¹, 陶珑¹, 问虎龙²

作者信息 +

REINFORCEMENT LEARNING ACTIVE DISTURBANCE REJECTION CONTROL STRATEGY FOR MICROGRID ENERGY STORAGE SIDE DC-DC CONVERTER

Ma Youjie¹, Liu Yiming¹, Zhou Xuesong¹, Wang Bo¹, Tao Long¹, Wen Hulong²

Author information +

文章历史 +

摘要

直流微电网电压稳定性研究是新型电力系统面临的关键问题。针对微电网系统中直流母线电压波动大和抗干扰能力弱等缺陷,该文提出一种由 Q-learning算法赋能的DC-DC变换器自抗扰控制策略。通过引入线性扩张状态观测器,实现对模型内部摄动与外部扰动量的精确估计与补偿,利用Q-learning 算法实现控制策略参数自适应优化,从而更高效地维持输出电压稳定。基于理论分析,推导范数意义下的Q-learning算法收敛性,并运用Lyapunov理论判据证明线性自抗扰的稳定性。最后,通过仿真对比该文提出的控制策略、线性自抗扰控制与双闭环PI控制在不同工况下的结果,充分验证该策略在提升 DC-DC 变换器抗扰能力和鲁棒水平的高效性与优越性。

Abstract

The study of voltage stability in DC microgrids is a key issue faced by new power systems. This paper proposes a active disturbance rejection control strategy for DC-DC converters empowered by Q-learning algorithm to address the shortcomings of large DC bus voltage fluctuations and weak anti-interference ability in microgrid systems. By introducing a linear expansion state observer, precise estimation and compensation of internal and external disturbances in the model are achieved. Q-learning algorithm is used to achieve adaptive optimization of control strategy parameters, thereby maintaining output voltage stability more efficiently. Based on theoretical analysis, the convergence of Q-learning algorithm in the norm sense was derived, and the stability of linear active disturbance rejection was proved using Lyapunov theory criterion. Finally, by comparing the results of the proposed control strategy, linear active disturbance rejection control, and dual closed-loop PI control under different operating conditions through simulation, the efficiency and superiority of this strategy in improving the disturbance rejection ability and robustness level of DC-DC converters are fully verified.

导出引用

马幼捷, 刘熠铭, 周雪松, 王博, 陶珑, 问虎龙. 微网储能侧DC-DC变换器的强化学习自抗扰控制策略[J]. 太阳能学报. 2025, 46(3): 63-72 https://doi.org/10.19912/j.0254-0096.tynxb.2023-1955

Ma Youjie, Liu Yiming, Zhou Xuesong, Wang Bo, Tao Long, Wen Hulong. REINFORCEMENT LEARNING ACTIVE DISTURBANCE REJECTION CONTROL STRATEGY FOR MICROGRID ENERGY STORAGE SIDE DC-DC CONVERTER[J]. Acta Energiae Solaris Sinica. 2025, 46(3): 63-72 https://doi.org/10.19912/j.0254-0096.tynxb.2023-1955

中图分类号： TM615

参考文献

[1] 朱晓荣, 孟欣欣. 直流微电网的稳定性分析及有源阻尼控制研究[J]. 高电压技术, 2020, 46(5): 1670-1681.
ZHU X R, MENG X X.Stability analysis and research of active damping control method for dc microgrids[J]. High voltage engineering, 2020, 46(5): 1670-1681.
[2] 张纯江, 董杰, 刘君, 等. 蓄电池与超级电容混合储能系统的控制策略[J]. 电工技术学报, 2014,29(4): 334-340.
ZHANG C J, DONG J, LIU J, et al.A control strategy for battery-ultracapacitor hybrid energy storage system[J]. Transactions of China Electrotechnical Society, 2014, 29(4): 334-340.
[3] 马幼捷, 袁业沧, 周雪松, 等. 模型信息联合校正型自抗扰控制策略[J]. 太阳能学报, 2024, 45(3): 389-398.
MA Y J, YUAN Y C, ZHOU X S, et al.Model information combined correction active disturbance rejection voltage stabilizing control strategy[J]. Acta energiae solaris sinica, 2024, 45(3): 389-398.
[4] 李霞林, 王成山, 郭力, 等. 直流微电网稳定控制关键技术研究综述[J]. 供用电, 2015, 32(10): 1-14.
LI X L, WANG C S, GUO L, et al.A review on the key stability control technologies of DC microgrid[J]. Distribution & utilization, 2015, 32(10): 1-14.
[5] 杨惠, 骆姗, 孙向东, 等. 光伏储能双向DC-DC变换器的自抗扰控制方法研究[J]. 太阳能学报, 2018, 39(5): 1342-1350.
YANG H, LUO S, SUN X D, et al.Research on ADRC method for bidirectional DC-DC converter of solar energy storage system[J]. Acta energiae solaris sinica, 2018, 39(5): 1342-1350.
[6] 苏泳, 陈艳峰. 基于非线性扰动观测的储能双向DC-DC变换器非奇异终端滑模控制策略[J]. 储能科学与技术, 2024, 13(5): 1523-1531.
SU Y, CHEN Y F.Nonsingular terminal sliding mode control strategy of bidirectional DC-DC converter in energy storage system based on the nonlinear disturbance observer[J]. Energy storage science and technology, 2024, 13(5): 1523-1531.
[7] 张世欣, 皇金锋, 杨艺. 基于平坦理论的直流微电网双向DC-DC变换器改进滑模自抗扰控制[J]. 电力系统保护与控制, 2023, 51(5): 107-116.
ZHANG S X, HUANG J F, YANG Y.Improved sliding mode and active disturbance rejection control based on flatness theory for a bi-directional DC-DC converter in a DC microgrid[J]. Power system protection and control, 2023, 51(5): 107-116.
[8] 许力, 曹青松, 易星. 电动汽车双向DC-DC变换器分数阶PI^λ控制[J]. 微电机, 2021, 54(12): 77-81.
XU L, CAO Q S, YI X.Fractional order PI^λ control of bidirectional DC-DC converter for electric vehicle[J]. Micromotors, 2021, 54(12): 77-81.
[9] 韩京清. 自抗扰控制器及其应用[J]. 控制与决策, 1998, 13(1): 19-23.
HAN J Q.Auto-disturbances-rejection controller and its applications[J]. Control and decision, 1998, 13(1): 19-23.
[10] GAO Z Q.Scaling and bandwidth-parameterization based controller tuning[C]//Proceedings of the 2003 American Control Conference. Denver, CO, USA, 2003: 4989-4996.
[11] 金辉宇, 张瑞青, 王雷, 等. 线性自抗扰控制参数整定鲁棒性的根轨迹分析[J]. 控制理论与应用, 2018, 35(11): 1648-1653.
JIN H Y, ZHANG R Q, WANG L, et al.Root locus analysis on parameter tuning robustness of linear active disturbance rejection control[J]. Control theory & applications, 2018, 35(11): 1648-1653.
[12] 周雪松, 王博, 马幼捷, 等. 含二阶扰动补偿的交错并联变换器自抗扰控制[J]. 电机与控制学报, 2023, 27(12): 159-170.
ZHOU X S, WANG B, MA Y J, et al.Active disturbance rejection control of interleaving parallel converter with second-order disturbance compensation[J]. Electric machines and control, 2023, 27(12): 159-170.
[13] CHEN Z Q, QIN B B, SUN M W, et al.Q-Learning-based parameters adaptive algorithm for active disturbance rejection control and its application to ship course control[J]. Neurocomputing, 2020, 408: 51-63.
[14] 龚春阳, 林嘉伟, 黄冬梅, 等. 储能系统双向Buck-Boost变换器控制策略研究[J]. 太阳能学报, 2023, 44(2): 229-238.
GONG C Y, LIN J W, HUANG D M, et al.Research on control strategy of bidirectional Buck-Boost converter in energy storage system[J]. Acta energiae solaris sinica, 2023, 44(2): 229-238.
[15] 张晨. 基于改进自抗扰和遗传算法的永磁同步电机速度控制[J]. 科技视界, 2020(5): 82-84.
ZHANG C.Speed control of permanent magnet synchronous motor based on improved ADRC and genetic algorithm[J]. Science & technology vision, 2020(5): 82-84.
[16] 王宏志, 王婷婷, 胡黄水, 等. 基于Q学习优化BP神经网络的BLDCM转速PID控制[J]. 吉林大学学报(工学版), 2021, 51(6): 2280-2286.
WANG H Z, WANG T T, HU H S, et al.PID control based on BP neural network optimized by Q-learning for speed control of BLDCM[J]. Journal of Jilin University (engineering and technology edition), 2021, 51(6): 2280-2286.
[17] MELO F S.Convergence of Q-learning: a simple proof[R]. Institute of Systems and Robotics, tech. 2001:1-4.
[18] 杨翔宇, 肖先勇, 马俊鹏, 等. 基于电感电流反馈的双向DC-DC变换器下垂控制[J]. 中国电机工程学报, 2020, 40(8): 2638-2647.
YANG X Y, XIAO X Y, MA J P, et al.Droop control of bi-directional DC-DC converters based on inductive current feedback[J]. Proceedings of the CSEE, 2020, 40(8): 2638-2647.
[19] 周雪松, 张心茹, 赵浛宇, 等. 基于DDPG算法的微网负载端接口变换器自抗扰控制[J]. 电力系统保护与控制, 2023, 51(21): 66-75.
ZHOU X S, ZHANG X R, ZHAO H Y, et al.Active disturbance rejection control of a microgrid load-side interface converter based on a DDPG algorithm[J]. Power system protection and control, 2023, 51(21): 66-75.
[20] 孟建辉, 吴小龙, 张自力, 等. 三相隔离型AC-DC-DC电源自适应线性自抗扰控制方法及纹波抑制补偿策略[J]. 电工技术学报, 2023, 38(14): 3898-3908.
MENG J H, WU X L, ZHANG Z L, et al.Adaptive linear active disturbance rejection control method and ripple suppression compensation strategy for three-phase isolated AC-DC-DC power supply[J]. Transactions of China Electrotechnical Society, 2023, 38(14): 3898-3908.
[21] 朱士加, 王颖, 贺春光, 等. 一阶LADRC的风电并网控制及其稳定性分析[J]. 电力系统及其自动化学报, 2020, 32(7): 33-38.
ZHU S J, WANG Y, HE C G, et al.Control of wind power grid-connection based on first-order LADRC and its stability analysis[J]. Proceedings of the CSU-EPSA, 2020, 32(7): 33-38.
[22] TAO L, WANG P, WANG Y F, et al.Variable structure ADRC-based control for load-side buck interface converter: formation, analysis, and verification[J]. IEEE transactions on industrial electronics, 2022, 69(6): 6236-6246.