基于TD3强化学习的光储微网双向DC-DC变换器自抗扰控制研究

马幼捷; 胡钰; 周雪松; 闫凤祥; 白鑫; 陶珑

doi:10.19912/j.0254-0096.tynxb.2024-0793

PDF(3373 KB)

太阳能学报 ›› 2026, Vol. 47 ›› Issue (1) : 202-213. DOI: 10.19912/j.0254-0096.tynxb.2024-0793

基于TD3强化学习的光储微网双向DC-DC变换器自抗扰控制研究

马幼捷, 胡钰, 周雪松, 闫凤祥, 白鑫, 陶珑

作者信息 +

RESEARCH ON ACTIVE DISTURBANCE REJECTION CONTROL OF BIDIRECTIONAL DC-DC CONVERTER IN PHOTOVOLTAIC AND ENERGY STORAGE MICROGRID BASED ON TD3 REINFORCEMENT LEARNING

Ma Youjie, Hu Yu, Zhou Xuesong, Yan Fengxiang, Bai Xin, Tao Long

Author information +

文章历史 +

摘要

考虑到高比例新能源接入带来的不确定性问题会导致微电网直流母线电压的大幅波动难以平抑,该文提出一种基于双延迟深度确定性策略梯度算法（TD3）强化学习的双向 DC-DC 变换器的自抗扰控制策略。首先,利用线性扩张状态观测器进行系统重构来实现对总扰动的估计补偿,并就控制策略的跟踪性和抗扰性进行频域分析。接着,通过大量的仿真交互自学习获得观测器参数来智能调节神经网络的权值更新方式,优化奖励函数形式,并在线利用网络进行参数实时调度,使其充分训练以实现近似最优控制律。最后,利用数字仿真平台和小功率实验验证了在多工况下所提控制策略较双闭环 PI控制和传统线性自抗扰控制具有更小的电压偏差及更快的响应速度等优越的动稳态性能,有效提升了直流母线电压的抗扰能力。

Abstract

Considering the uncertainty caused by the high proportion of renewable energy integration, leading to significant fluctuations in the DC bus voltage of microgrids,this paper proposes an active disturbance rejection control strategy for bidirectional DC-DC converter based on twin delayed deep deterministic policy gradient （TD3） reinforcement learning. Firstly, a linear extended state observer was used for system reconstruction to estimate and compensate for total disturbances, and frequency-domain analysis were conducted on the tracking and anti-interference ability of the control strategy. Subsequently, a large amount of simulation self-learning was used to obtain observer parameters to intelligently adjust the weight update method of the neural network, optimize the form of the reward function, and utilize the online network for real-time parameter scheduling, enabling sufficient training to achieve an approximate optimal control law. Finally, digital simulation platform and low-power experiment were used to verify that the proposed control strategy has superior dynamic and steady-state performance, such as smaller voltage deviation and faster response speed under multiple operating conditions, compared to PI control and traditional linear active disturbance rejection control , effectively improving the anti-interference ability of DC bus voltage.

导出引用

马幼捷, 胡钰, 周雪松, 闫凤祥, 白鑫, 陶珑. 基于TD3强化学习的光储微网双向DC-DC变换器自抗扰控制研究[J]. 太阳能学报. 2026, 47(1): 202-213 https://doi.org/10.19912/j.0254-0096.tynxb.2024-0793

Ma Youjie, Hu Yu, Zhou Xuesong, Yan Fengxiang, Bai Xin, Tao Long. RESEARCH ON ACTIVE DISTURBANCE REJECTION CONTROL OF BIDIRECTIONAL DC-DC CONVERTER IN PHOTOVOLTAIC AND ENERGY STORAGE MICROGRID BASED ON TD3 REINFORCEMENT LEARNING[J]. Acta Energiae Solaris Sinica. 2026, 47(1): 202-213 https://doi.org/10.19912/j.0254-0096.tynxb.2024-0793

中图分类号： TM46

参考文献

[1] 李建林, 袁晓冬, 郁正纲, 等. 利用储能系统提升电网电能质量研究综述[J]. 电力系统自动化, 2019, 43(8): 15-24.
LI J L, YUAN X D, YU Z G, et al.Comments on power quality enhancement research for power grid by energy storage system[J]. Automation of electric power systems, 2019, 43(8): 15-24.
[2] 王成山, 李微, 王议锋, 等. 直流微电网母线电压波动分类及抑制方法综述[J]. 中国电机工程学报, 2017, 37(1): 84-98.WANG C S, LI W, WANG Y F, et al. DC bus voltage fluctuation classification and restraint methods review for DC microgrid[J]. Proceedings of the CSEE, 2017, 37(1): 84-98.
[3] 王雨祺, 赵兴勇, 赖建中, 等. 基于自抗扰控制的光伏微电网储能控制策略[J]. 电气工程学报, 2024, 19(3): 423-431.WANG Y Q,ZHAO X Y, LAI J Z,et al. Energy storage control strategy for photovoltaic microgrid based on self disturbance rejection control[J]. Journal of electrical engineering, 2024, 19(3): 423-431.
[4] 尚彤, 崔学深, 徐明荣, 等. 蓄电池-超级电容混合储能系统放电控制策略[J]. 电源技术, 2017, 41(4): 595-597, 637.SHANG T, CUI X S, XU M R, et al. Discharge control strategies of supercapacitor-battery hybrid energy storage system[J]. Chinese journal of power sources, 2017, 41(4): 595-597, 637.
[5] 黄家才, 马鹏, 高芳征. 基于分数阶自抗扰控制的位置伺服系统研究[J]. 电气传动, 2020, 50(12): 59-63, 110.HUANG J C, MA P, GAO F Z. Research on position servo system based on fractional-order active disturbance rejection control[J]. Electric drive, 2020, 50(12): 59-63, 110.
[6] 侯利民, 任一夫, 王怀震, 等. 永磁同步电机调速系统的滑模自抗扰控制[J]. 控制工程, 2019, 26(8): 1460-1465.HOU L M, REN Y F, WANG H Z, et al. Sliding mode based on active disturbance rejection control of speed governing system for PMSM[J]. Control engineering of China, 2019, 26(8): 1460-1465.
[7] 郑昊, 杜贵平, 雷雁雄, 等. 混合储能双向DC/DC变换器线性自抗扰控制研究[J]. 电力电子技术, 2022, 56(9): 80-83.ZHENG H, DU G P, LEI Y X, et al. Research on linear active disturbance rejection control of hybrid energy storage bidirectional DC/DC converter[J]. Power electronics, 2022, 56(9): 80-83.
[8] 孙雨萌, 张旭秀. 改进遗传算法整定自抗扰控制器参数及应用[J]. 自动化与仪表, 2020, 35(3): 13-17, 45.SUN Y M, ZHANG X X. Parameters optimization and application of auto-disturbance-rejection-controller based on improved genetic algorithm[J]. Automation & instrumentation, 2020, 35(3): 13-17, 45.
[9] 魏立新, 赵默林, 范锐, 等. 基于改进鲨鱼优化算法的自抗扰控制参数整定[J]. 控制与决策, 2019, 34(4): 816-820.WEI L X, ZHAO M L, FAN R, et al. Parameter tuning of active disturbance rejection control based on ameliorated shark smell optimization algorithm[J]. Control and decision, 2019, 34(4): 816-820.
[10] 王晨, 周雪松, 马幼捷, 等. 基于混合策略麻雀搜索算法优化的DC-DC变换器自抗扰稳压策略[J]. 国外电子测量技术, 2024, 43(7): 46-56.WANG C, ZHOU X S, MA Y J, et al. Active disturbance rejection voltage stablization strategy for DC-DC convertor based on hybrid sparrow search algorithm[J]. Foreign electronic measurement technology, 2024, 43(7): 46-56.
[11] 周雪松, 张心茹, 赵浛宇, 等. 基于DDPG算法的微网负载端接口变换器自抗扰控制[J]. 电力系统保护与控制, 2023, 51(21): 66-75.ZHOU X S, ZHANG X R, ZHAO H Y, et al. Active disturbance rejection control of a microgrid load-side interface converter based on a DDPG algorithm[J]. Power system protection and control, 2023, 51(21): 66-75.
[12] 高志强. 自抗扰控制思想探究[J]. 控制理论与应用, 2013, 30(12): 1498-1510.GAO Z Q. On the foundation of active disturbance rejection control[J]. Control theory & applications, 2013, 30(12): 1498-1510.
[13] 郭宪, 方勇纯. 仿生机器人运动步态控制: 强化学习方法综述[J]. 智能系统学报, 2020, 15(1): 152-159.GUO X, FANG Y C. Locomotion gait control for bionic robots: a review of reinforcement learning methods[J]. CAAI transactions on intelligent systems, 2020, 15(1): 152-159.
[14] 刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1-27.LIU Q, ZHAI J W, ZHANG Z Z, et al. A survey on deep reinforcement learning[J]. Chinese journal of computers, 2018, 41(1): 1-27.
[15] CAO D, HU W H, ZHAO J B, et al.Reinforcement learning and its applications in modern power and energy systems: a review[J]. Journal of modern power systems and clean energy, 2020, 8(6): 1029-1042.
[16] 张延宇, 饶新朋, 周书奎, 等. 基于深度强化学习的电动汽车充电调度算法研究进展[J]. 电力系统保护与控制, 2022, 50(16): 179-187.ZHANG Y Y, RAO X P, ZHOU S K, et al. Research progress of electric vehicle charging scheduling algorithms based on deep reinforcement learning[J]. Power system protection and control, 2022, 50(16): 179-187.
[17] 尚立, 蔡硕, 崔俊彬, 等. 基于软件定义网络的电网边缘计算资源分配[J]. 电力系统保护与控制, 2021, 49(20): 136-143.SHANG L, CAI S, CUI J B, et al. SDN-based MEC resource allocation of a power grid[J]. Power system protection and control, 2021, 49(20): 136-143.
[18] 陈希亮, 曹雷, 李晨溪, 等. 基于重抽样优选缓存经验回放机制的深度强化学习方法[J]. 控制与决策, 2018, 33(4): 600-606.CHEN X L, CAO L, LI C X, et al. Deep reinforcement learning via good choice resampling experience replay memory[J]. Control and decision, 2018, 33(4): 600-606.
[19] 皇金锋, 杨振宇, 李帅杰. 基于混合储能的光储直流微网改进型滑模自抗扰控制[J]. 电力系统保护与控制, 2023, 51(23): 151-159.HUANG J F, YANG Z Y, LI S J. Improved sliding mode self-disturbance resistance control of an optical storage DC micro-grid based on hybrid energy storage[J]. Power system protection and control, 2023, 51(23): 151-159.
[20] 袁东, 马晓军, 曾庆含, 等. 二阶系统线性自抗扰控制器频带特性与参数配置研究[J]. 控制理论与应用, 2013, 30(12): 1630-1640.YUAN D, MA X J, ZENG Q H, et al. Research on frequency-band characteristics and parameters configuration of linear active disturbance rejection control for second-order systems[J]. Control theory & applications, 2013, 30(12): 1630-1640.
[21] 马幼捷, 杨清, 周雪松, 等. 基于模糊神经网络的储能侧双向DC-DC变换器自抗扰控制策略[J]. 太阳能学报, 2023, 44(10): 488-495.MA Y J, YANG Q, ZHOU X S, et al. Active disturbance rejection control strategy of bidirectional DC-DC converter on energy storage side based on fuzzy neural network[J]. Acta energiae solaris sinica, 2023, 44(10): 488-495.
[22] 农宝翔, 杨超, 杜刃刃, 等. 基于模糊自抗扰的光伏储能控制策略研究[J]. 电力科学与工程, 2023, 39(3): 17-24.NONG B X, YANG C, DU R R, et al. Research on photovoltaic energy storage control strategy based on fuzzy auto disturbance rejection[J]. Electric power science and engineering, 2023, 39(3): 17-24.
[23] 龚春阳, 林嘉伟, 黄冬梅, 等. 储能系统双向Buck-Boost变换器控制策略研究[J]. 太阳能学报, 2023, 44(2): 229-238.GONG C Y, LIN J W, HUANG D M, et al. Research on control strategy of bidirectional Buck-Boost converter in energy storage system[J]. Acta energiae solaris sinica, 2023, 44(2): 229-238.
[24] 周雪松, 王博, 马幼捷, 等. 微电网负载侧变换器的观测偏差优化自抗扰策略[J]. 中国电机工程学报, 2024, 44(23): 9354-9367.ZHOU X S, WANG B, MA Y J, et al. Model observation deviation correction ADRC-based voltage stabilization strategy for microgrid load-side converters[J]. Proceedings of the CSEE, 2024, 44(23): 9354-9367.