REINFORCEMENT LEARNING ACTIVE DISTURBANCE REJECTION CONTROL STRATEGY FOR MICROGRID ENERGY STORAGE SIDE DC-DC CONVERTER

Ma Youjie; Liu Yiming; Zhou Xuesong; Wang Bo; Tao Long; Wen Hulong

doi:10.19912/j.0254-0096.tynxb.2023-1955

PDF(3313 KB)

Acta Energiae Solaris Sinica ›› 2025, Vol. 46 ›› Issue (3) : 63-72. DOI: 10.19912/j.0254-0096.tynxb.2023-1955

REINFORCEMENT LEARNING ACTIVE DISTURBANCE REJECTION CONTROL STRATEGY FOR MICROGRID ENERGY STORAGE SIDE DC-DC CONVERTER

Ma Youjie¹, Liu Yiming¹, Zhou Xuesong¹, Wang Bo¹, Tao Long¹, Wen Hulong²

Author information +

History +

Abstract

The study of voltage stability in DC microgrids is a key issue faced by new power systems. This paper proposes a active disturbance rejection control strategy for DC-DC converters empowered by Q-learning algorithm to address the shortcomings of large DC bus voltage fluctuations and weak anti-interference ability in microgrid systems. By introducing a linear expansion state observer, precise estimation and compensation of internal and external disturbances in the model are achieved. Q-learning algorithm is used to achieve adaptive optimization of control strategy parameters, thereby maintaining output voltage stability more efficiently. Based on theoretical analysis, the convergence of Q-learning algorithm in the norm sense was derived, and the stability of linear active disturbance rejection was proved using Lyapunov theory criterion. Finally, by comparing the results of the proposed control strategy, linear active disturbance rejection control, and dual closed-loop PI control under different operating conditions through simulation, the efficiency and superiority of this strategy in improving the disturbance rejection ability and robustness level of DC-DC converters are fully verified.

Key words

microgrids / DC-DC converter / ADRC / Q-learning algorithm / Lyapunov criterion

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

Ma Youjie, Liu Yiming, Zhou Xuesong, Wang Bo, Tao Long, Wen Hulong. REINFORCEMENT LEARNING ACTIVE DISTURBANCE REJECTION CONTROL STRATEGY FOR MICROGRID ENERGY STORAGE SIDE DC-DC CONVERTER[J]. Acta Energiae Solaris Sinica. 2025, 46(3): 63-72 https://doi.org/10.19912/j.0254-0096.tynxb.2023-1955

References

[1] 朱晓荣, 孟欣欣. 直流微电网的稳定性分析及有源阻尼控制研究[J]. 高电压技术, 2020, 46(5): 1670-1681.
ZHU X R, MENG X X.Stability analysis and research of active damping control method for dc microgrids[J]. High voltage engineering, 2020, 46(5): 1670-1681.
[2] 张纯江, 董杰, 刘君, 等. 蓄电池与超级电容混合储能系统的控制策略[J]. 电工技术学报, 2014,29(4): 334-340.
ZHANG C J, DONG J, LIU J, et al.A control strategy for battery-ultracapacitor hybrid energy storage system[J]. Transactions of China Electrotechnical Society, 2014, 29(4): 334-340.
[3] 马幼捷, 袁业沧, 周雪松, 等. 模型信息联合校正型自抗扰控制策略[J]. 太阳能学报, 2024, 45(3): 389-398.
MA Y J, YUAN Y C, ZHOU X S, et al.Model information combined correction active disturbance rejection voltage stabilizing control strategy[J]. Acta energiae solaris sinica, 2024, 45(3): 389-398.
[4] 李霞林, 王成山, 郭力, 等. 直流微电网稳定控制关键技术研究综述[J]. 供用电, 2015, 32(10): 1-14.
LI X L, WANG C S, GUO L, et al.A review on the key stability control technologies of DC microgrid[J]. Distribution & utilization, 2015, 32(10): 1-14.
[5] 杨惠, 骆姗, 孙向东, 等. 光伏储能双向DC-DC变换器的自抗扰控制方法研究[J]. 太阳能学报, 2018, 39(5): 1342-1350.
YANG H, LUO S, SUN X D, et al.Research on ADRC method for bidirectional DC-DC converter of solar energy storage system[J]. Acta energiae solaris sinica, 2018, 39(5): 1342-1350.
[6] 苏泳, 陈艳峰. 基于非线性扰动观测的储能双向DC-DC变换器非奇异终端滑模控制策略[J]. 储能科学与技术, 2024, 13(5): 1523-1531.
SU Y, CHEN Y F.Nonsingular terminal sliding mode control strategy of bidirectional DC-DC converter in energy storage system based on the nonlinear disturbance observer[J]. Energy storage science and technology, 2024, 13(5): 1523-1531.
[7] 张世欣, 皇金锋, 杨艺. 基于平坦理论的直流微电网双向DC-DC变换器改进滑模自抗扰控制[J]. 电力系统保护与控制, 2023, 51(5): 107-116.
ZHANG S X, HUANG J F, YANG Y.Improved sliding mode and active disturbance rejection control based on flatness theory for a bi-directional DC-DC converter in a DC microgrid[J]. Power system protection and control, 2023, 51(5): 107-116.
[8] 许力, 曹青松, 易星. 电动汽车双向DC-DC变换器分数阶PI^λ控制[J]. 微电机, 2021, 54(12): 77-81.
XU L, CAO Q S, YI X.Fractional order PI^λ control of bidirectional DC-DC converter for electric vehicle[J]. Micromotors, 2021, 54(12): 77-81.
[9] 韩京清. 自抗扰控制器及其应用[J]. 控制与决策, 1998, 13(1): 19-23.
HAN J Q.Auto-disturbances-rejection controller and its applications[J]. Control and decision, 1998, 13(1): 19-23.
[10] GAO Z Q.Scaling and bandwidth-parameterization based controller tuning[C]//Proceedings of the 2003 American Control Conference. Denver, CO, USA, 2003: 4989-4996.
[11] 金辉宇, 张瑞青, 王雷, 等. 线性自抗扰控制参数整定鲁棒性的根轨迹分析[J]. 控制理论与应用, 2018, 35(11): 1648-1653.
JIN H Y, ZHANG R Q, WANG L, et al.Root locus analysis on parameter tuning robustness of linear active disturbance rejection control[J]. Control theory & applications, 2018, 35(11): 1648-1653.
[12] 周雪松, 王博, 马幼捷, 等. 含二阶扰动补偿的交错并联变换器自抗扰控制[J]. 电机与控制学报, 2023, 27(12): 159-170.
ZHOU X S, WANG B, MA Y J, et al.Active disturbance rejection control of interleaving parallel converter with second-order disturbance compensation[J]. Electric machines and control, 2023, 27(12): 159-170.
[13] CHEN Z Q, QIN B B, SUN M W, et al.Q-Learning-based parameters adaptive algorithm for active disturbance rejection control and its application to ship course control[J]. Neurocomputing, 2020, 408: 51-63.
[14] 龚春阳, 林嘉伟, 黄冬梅, 等. 储能系统双向Buck-Boost变换器控制策略研究[J]. 太阳能学报, 2023, 44(2): 229-238.
GONG C Y, LIN J W, HUANG D M, et al.Research on control strategy of bidirectional Buck-Boost converter in energy storage system[J]. Acta energiae solaris sinica, 2023, 44(2): 229-238.
[15] 张晨. 基于改进自抗扰和遗传算法的永磁同步电机速度控制[J]. 科技视界, 2020(5): 82-84.
ZHANG C.Speed control of permanent magnet synchronous motor based on improved ADRC and genetic algorithm[J]. Science & technology vision, 2020(5): 82-84.
[16] 王宏志, 王婷婷, 胡黄水, 等. 基于Q学习优化BP神经网络的BLDCM转速PID控制[J]. 吉林大学学报(工学版), 2021, 51(6): 2280-2286.
WANG H Z, WANG T T, HU H S, et al.PID control based on BP neural network optimized by Q-learning for speed control of BLDCM[J]. Journal of Jilin University (engineering and technology edition), 2021, 51(6): 2280-2286.
[17] MELO F S.Convergence of Q-learning: a simple proof[R]. Institute of Systems and Robotics, tech. 2001:1-4.
[18] 杨翔宇, 肖先勇, 马俊鹏, 等. 基于电感电流反馈的双向DC-DC变换器下垂控制[J]. 中国电机工程学报, 2020, 40(8): 2638-2647.
YANG X Y, XIAO X Y, MA J P, et al.Droop control of bi-directional DC-DC converters based on inductive current feedback[J]. Proceedings of the CSEE, 2020, 40(8): 2638-2647.
[19] 周雪松, 张心茹, 赵浛宇, 等. 基于DDPG算法的微网负载端接口变换器自抗扰控制[J]. 电力系统保护与控制, 2023, 51(21): 66-75.
ZHOU X S, ZHANG X R, ZHAO H Y, et al.Active disturbance rejection control of a microgrid load-side interface converter based on a DDPG algorithm[J]. Power system protection and control, 2023, 51(21): 66-75.
[20] 孟建辉, 吴小龙, 张自力, 等. 三相隔离型AC-DC-DC电源自适应线性自抗扰控制方法及纹波抑制补偿策略[J]. 电工技术学报, 2023, 38(14): 3898-3908.
MENG J H, WU X L, ZHANG Z L, et al.Adaptive linear active disturbance rejection control method and ripple suppression compensation strategy for three-phase isolated AC-DC-DC power supply[J]. Transactions of China Electrotechnical Society, 2023, 38(14): 3898-3908.
[21] 朱士加, 王颖, 贺春光, 等. 一阶LADRC的风电并网控制及其稳定性分析[J]. 电力系统及其自动化学报, 2020, 32(7): 33-38.
ZHU S J, WANG Y, HE C G, et al.Control of wind power grid-connection based on first-order LADRC and its stability analysis[J]. Proceedings of the CSU-EPSA, 2020, 32(7): 33-38.
[22] TAO L, WANG P, WANG Y F, et al.Variable structure ADRC-based control for load-side buck interface converter: formation, analysis, and verification[J]. IEEE transactions on industrial electronics, 2022, 69(6): 6236-6246.