RESEARCH ON ACTIVE DISTURBANCE REJECTION CONTROL OF BIDIRECTIONAL DC-DC CONVERTER IN PHOTOVOLTAIC AND ENERGY STORAGE MICROGRID BASED ON TD3 REINFORCEMENT LEARNING

Ma Youjie; Hu Yu; Zhou Xuesong; Yan Fengxiang; Bai Xin; Tao Long

doi:10.19912/j.0254-0096.tynxb.2024-0793

PDF(3373 KB)

Acta Energiae Solaris Sinica ›› 2026, Vol. 47 ›› Issue (1) : 202-213. DOI: 10.19912/j.0254-0096.tynxb.2024-0793

RESEARCH ON ACTIVE DISTURBANCE REJECTION CONTROL OF BIDIRECTIONAL DC-DC CONVERTER IN PHOTOVOLTAIC AND ENERGY STORAGE MICROGRID BASED ON TD3 REINFORCEMENT LEARNING

Ma Youjie, Hu Yu, Zhou Xuesong, Yan Fengxiang, Bai Xin, Tao Long

Author information +

History +

Abstract

Considering the uncertainty caused by the high proportion of renewable energy integration, leading to significant fluctuations in the DC bus voltage of microgrids,this paper proposes an active disturbance rejection control strategy for bidirectional DC-DC converter based on twin delayed deep deterministic policy gradient （TD3） reinforcement learning. Firstly, a linear extended state observer was used for system reconstruction to estimate and compensate for total disturbances, and frequency-domain analysis were conducted on the tracking and anti-interference ability of the control strategy. Subsequently, a large amount of simulation self-learning was used to obtain observer parameters to intelligently adjust the weight update method of the neural network, optimize the form of the reward function, and utilize the online network for real-time parameter scheduling, enabling sufficient training to achieve an approximate optimal control law. Finally, digital simulation platform and low-power experiment were used to verify that the proposed control strategy has superior dynamic and steady-state performance, such as smaller voltage deviation and faster response speed under multiple operating conditions, compared to PI control and traditional linear active disturbance rejection control , effectively improving the anti-interference ability of DC bus voltage.

Key words

bidirectional DC-DC converter / photovoltaic and energy-storage microgrid / active disturbance rejection control / TD3 deep reinforcement learning algorithm

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

Ma Youjie, Hu Yu, Zhou Xuesong, Yan Fengxiang, Bai Xin, Tao Long. RESEARCH ON ACTIVE DISTURBANCE REJECTION CONTROL OF BIDIRECTIONAL DC-DC CONVERTER IN PHOTOVOLTAIC AND ENERGY STORAGE MICROGRID BASED ON TD3 REINFORCEMENT LEARNING[J]. Acta Energiae Solaris Sinica. 2026, 47(1): 202-213 https://doi.org/10.19912/j.0254-0096.tynxb.2024-0793

References

[1] 李建林, 袁晓冬, 郁正纲, 等. 利用储能系统提升电网电能质量研究综述[J]. 电力系统自动化, 2019, 43(8): 15-24.
LI J L, YUAN X D, YU Z G, et al.Comments on power quality enhancement research for power grid by energy storage system[J]. Automation of electric power systems, 2019, 43(8): 15-24.
[2] 王成山, 李微, 王议锋, 等. 直流微电网母线电压波动分类及抑制方法综述[J]. 中国电机工程学报, 2017, 37(1): 84-98.WANG C S, LI W, WANG Y F, et al. DC bus voltage fluctuation classification and restraint methods review for DC microgrid[J]. Proceedings of the CSEE, 2017, 37(1): 84-98.
[3] 王雨祺, 赵兴勇, 赖建中, 等. 基于自抗扰控制的光伏微电网储能控制策略[J]. 电气工程学报, 2024, 19(3): 423-431.WANG Y Q,ZHAO X Y, LAI J Z,et al. Energy storage control strategy for photovoltaic microgrid based on self disturbance rejection control[J]. Journal of electrical engineering, 2024, 19(3): 423-431.
[4] 尚彤, 崔学深, 徐明荣, 等. 蓄电池-超级电容混合储能系统放电控制策略[J]. 电源技术, 2017, 41(4): 595-597, 637.SHANG T, CUI X S, XU M R, et al. Discharge control strategies of supercapacitor-battery hybrid energy storage system[J]. Chinese journal of power sources, 2017, 41(4): 595-597, 637.
[5] 黄家才, 马鹏, 高芳征. 基于分数阶自抗扰控制的位置伺服系统研究[J]. 电气传动, 2020, 50(12): 59-63, 110.HUANG J C, MA P, GAO F Z. Research on position servo system based on fractional-order active disturbance rejection control[J]. Electric drive, 2020, 50(12): 59-63, 110.
[6] 侯利民, 任一夫, 王怀震, 等. 永磁同步电机调速系统的滑模自抗扰控制[J]. 控制工程, 2019, 26(8): 1460-1465.HOU L M, REN Y F, WANG H Z, et al. Sliding mode based on active disturbance rejection control of speed governing system for PMSM[J]. Control engineering of China, 2019, 26(8): 1460-1465.
[7] 郑昊, 杜贵平, 雷雁雄, 等. 混合储能双向DC/DC变换器线性自抗扰控制研究[J]. 电力电子技术, 2022, 56(9): 80-83.ZHENG H, DU G P, LEI Y X, et al. Research on linear active disturbance rejection control of hybrid energy storage bidirectional DC/DC converter[J]. Power electronics, 2022, 56(9): 80-83.
[8] 孙雨萌, 张旭秀. 改进遗传算法整定自抗扰控制器参数及应用[J]. 自动化与仪表, 2020, 35(3): 13-17, 45.SUN Y M, ZHANG X X. Parameters optimization and application of auto-disturbance-rejection-controller based on improved genetic algorithm[J]. Automation & instrumentation, 2020, 35(3): 13-17, 45.
[9] 魏立新, 赵默林, 范锐, 等. 基于改进鲨鱼优化算法的自抗扰控制参数整定[J]. 控制与决策, 2019, 34(4): 816-820.WEI L X, ZHAO M L, FAN R, et al. Parameter tuning of active disturbance rejection control based on ameliorated shark smell optimization algorithm[J]. Control and decision, 2019, 34(4): 816-820.
[10] 王晨, 周雪松, 马幼捷, 等. 基于混合策略麻雀搜索算法优化的DC-DC变换器自抗扰稳压策略[J]. 国外电子测量技术, 2024, 43(7): 46-56.WANG C, ZHOU X S, MA Y J, et al. Active disturbance rejection voltage stablization strategy for DC-DC convertor based on hybrid sparrow search algorithm[J]. Foreign electronic measurement technology, 2024, 43(7): 46-56.
[11] 周雪松, 张心茹, 赵浛宇, 等. 基于DDPG算法的微网负载端接口变换器自抗扰控制[J]. 电力系统保护与控制, 2023, 51(21): 66-75.ZHOU X S, ZHANG X R, ZHAO H Y, et al. Active disturbance rejection control of a microgrid load-side interface converter based on a DDPG algorithm[J]. Power system protection and control, 2023, 51(21): 66-75.
[12] 高志强. 自抗扰控制思想探究[J]. 控制理论与应用, 2013, 30(12): 1498-1510.GAO Z Q. On the foundation of active disturbance rejection control[J]. Control theory & applications, 2013, 30(12): 1498-1510.
[13] 郭宪, 方勇纯. 仿生机器人运动步态控制: 强化学习方法综述[J]. 智能系统学报, 2020, 15(1): 152-159.GUO X, FANG Y C. Locomotion gait control for bionic robots: a review of reinforcement learning methods[J]. CAAI transactions on intelligent systems, 2020, 15(1): 152-159.
[14] 刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1-27.LIU Q, ZHAI J W, ZHANG Z Z, et al. A survey on deep reinforcement learning[J]. Chinese journal of computers, 2018, 41(1): 1-27.
[15] CAO D, HU W H, ZHAO J B, et al.Reinforcement learning and its applications in modern power and energy systems: a review[J]. Journal of modern power systems and clean energy, 2020, 8(6): 1029-1042.
[16] 张延宇, 饶新朋, 周书奎, 等. 基于深度强化学习的电动汽车充电调度算法研究进展[J]. 电力系统保护与控制, 2022, 50(16): 179-187.ZHANG Y Y, RAO X P, ZHOU S K, et al. Research progress of electric vehicle charging scheduling algorithms based on deep reinforcement learning[J]. Power system protection and control, 2022, 50(16): 179-187.
[17] 尚立, 蔡硕, 崔俊彬, 等. 基于软件定义网络的电网边缘计算资源分配[J]. 电力系统保护与控制, 2021, 49(20): 136-143.SHANG L, CAI S, CUI J B, et al. SDN-based MEC resource allocation of a power grid[J]. Power system protection and control, 2021, 49(20): 136-143.
[18] 陈希亮, 曹雷, 李晨溪, 等. 基于重抽样优选缓存经验回放机制的深度强化学习方法[J]. 控制与决策, 2018, 33(4): 600-606.CHEN X L, CAO L, LI C X, et al. Deep reinforcement learning via good choice resampling experience replay memory[J]. Control and decision, 2018, 33(4): 600-606.
[19] 皇金锋, 杨振宇, 李帅杰. 基于混合储能的光储直流微网改进型滑模自抗扰控制[J]. 电力系统保护与控制, 2023, 51(23): 151-159.HUANG J F, YANG Z Y, LI S J. Improved sliding mode self-disturbance resistance control of an optical storage DC micro-grid based on hybrid energy storage[J]. Power system protection and control, 2023, 51(23): 151-159.
[20] 袁东, 马晓军, 曾庆含, 等. 二阶系统线性自抗扰控制器频带特性与参数配置研究[J]. 控制理论与应用, 2013, 30(12): 1630-1640.YUAN D, MA X J, ZENG Q H, et al. Research on frequency-band characteristics and parameters configuration of linear active disturbance rejection control for second-order systems[J]. Control theory & applications, 2013, 30(12): 1630-1640.
[21] 马幼捷, 杨清, 周雪松, 等. 基于模糊神经网络的储能侧双向DC-DC变换器自抗扰控制策略[J]. 太阳能学报, 2023, 44(10): 488-495.MA Y J, YANG Q, ZHOU X S, et al. Active disturbance rejection control strategy of bidirectional DC-DC converter on energy storage side based on fuzzy neural network[J]. Acta energiae solaris sinica, 2023, 44(10): 488-495.
[22] 农宝翔, 杨超, 杜刃刃, 等. 基于模糊自抗扰的光伏储能控制策略研究[J]. 电力科学与工程, 2023, 39(3): 17-24.NONG B X, YANG C, DU R R, et al. Research on photovoltaic energy storage control strategy based on fuzzy auto disturbance rejection[J]. Electric power science and engineering, 2023, 39(3): 17-24.
[23] 龚春阳, 林嘉伟, 黄冬梅, 等. 储能系统双向Buck-Boost变换器控制策略研究[J]. 太阳能学报, 2023, 44(2): 229-238.GONG C Y, LIN J W, HUANG D M, et al. Research on control strategy of bidirectional Buck-Boost converter in energy storage system[J]. Acta energiae solaris sinica, 2023, 44(2): 229-238.
[24] 周雪松, 王博, 马幼捷, 等. 微电网负载侧变换器的观测偏差优化自抗扰策略[J]. 中国电机工程学报, 2024, 44(23): 9354-9367.ZHOU X S, WANG B, MA Y J, et al. Model observation deviation correction ADRC-based voltage stabilization strategy for microgrid load-side converters[J]. Proceedings of the CSEE, 2024, 44(23): 9354-9367.