Abstract: Currently, decarbonization has become an emerging trend in the power system arena. However, the increasing number of photovoltaic units distributed into a distribution network may result in voltage issues, providing challenges for voltage regulation across a large-scale power grid network. Reinforcement learning based intelligent control of smart inverters and other smart building energy management (EM) systems can be leveraged to alleviate these issues. To achieve the best EM strategy for building microgrids in a power system, this paper presents two large-scale multi-agent strategy evaluation methods to preserve building occupants’ comfort while pursuing system-level objectives. The EM problem is formulated as a general-sum game to optimize the benefits at both the system and building levels. The α-rank algorithm can solve the general-sum game and guarantee the ranking theoretically, but it is limited by the interaction complexity and hardly applies to the practical power system. A new evaluation algorithm (TcEval) is proposed by practically scaling the α-rank algorithm through a tensor complement to reduce the interaction complexity. Then, considering the noise prevalent in practice, a noise processing model with domain knowledge is built to calculate the strategy payoffs, and thus the TcEval-AS algorithm is proposed when noise exists. Both evaluation algorithms developed in this paper greatly reduce the interaction complexity compared with existing approaches, including ResponseGraphUCB (RG-UCB) and α InformationGain (α-IG). Finally, the effectiveness of the proposed algorithms is verified in the EM case with realistic data.

基于拓展α-rank的多智能体策略评估方法在能源管理中的应用

[1]Brookes DH, Listgarten J, 2018. Design by adaptive sampling. https://arxiv.org/pdf/1810.03714v4

[2]Brookes DH, Park H, Listgarten J, 2019. Conditioning by adaptive sampling for robust design. Proc 36^th Int Conf on Machine Learning, p.773-782.

[3]Cai WQ, Kordabad AB, Gros S, 2023. Energy management in residential microgrid using model predictive control-based reinforcement learning and Shapley value. Eng Appl Artif Intell, 119:105793.

[4]Claessens BJ, Vrancx P, Ruelens F, 2018. Convolutional neural networks for automatic state-time feature extraction in reinforcement learning applied to residential load control. IEEE Trans Smart Grid, 9(4):3259-3269.

[5]Czarnecki WM, Gidel G, Tracey B, et al., 2020. Real world games look like spinning tops. Proc 34^th Int Conf on Neural Information Processing Systems, Article 1463.

[6]Dong Q, Wu ZY, Lu J, et al., 2022. Existence and practice of gaming: thoughts on the development of multi-agent system gaming. Front Inform Technol Electron Eng, 23(7):995-1001.

[7]Du YL, Yan X, Chen X, et al., 2021. Estimating α-rank from a few entries with low rank matrix completion. Proc 38^th Int Conf on Machine Learning, p.2870-2879.

[8]Lowe R, Wu Y, Tamar A, et al., 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Proc 31^st Int Conf on Neural Information Processing Systems, p.6382-6393.

[9]Muller P, Omidshafiei S, Rowland M, et al., 2020. A generalized training approach for multiagent learning. Proc 8^th Int Conf on Learning Representations.

[10]Omidshafiei S, Papadimitriou C, Piliouras G, et al., 2019. α-rank: multi-agent evaluation by evolution. Sci Rep, 9(1):9937.

[11]Pigott A, Crozier C, Baker K, et al., 2022. GridLearn: multiagent reinforcement learning for grid-aware building energy management. Electr Power Syst Res, 213:108521.

[12]Rashid T, Zhang C, Ciosek K, 2021. Estimating α-rank by maximizing information gain. Proc AAAI Conf on Artificial Intelligence, p.5673-5681.

[13]Rowland M, Omidshafiei S, Tuyls K, et al., 2019. Multiagent evaluation under incomplete information. Proc 33^rd Int Conf on Neural Information Processing Systems, Article 1101.

[14]Shalev-Shwartz S, Ben-David S, 2014. Understanding Machine Learning: from Theory to Algorithms. Cambridge University Press, Cambridge, UK.

[15]Signorino CS, Ritter JM, 1999. Tau-b or not tau-b: measuring the similarity of foreign policy positions. Int Stud Q, 43(1):115-144.

[16]Silver D, Huang A, Maddison CJ, et al., 2016. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489.

[17]Su WC, Wang JH, 2012. Energy management systems in microgrid operations. Electr J, 25(8):45-60.

[18]Tong Z, Li N, Zhang HM, et al., 2023. Dynamic user-centric multi-dimensional resource allocation for a wide-area coverage signaling cell based on DQN. Front Inform Technol Electron Eng, 24(1):154-163.

[19]Tuyls K, Perolat J, Lanctot M, et al., 2018. A generalised method for empirical game theoretic analysis. Proc 17^th Int Conf on Autonomous Agents and Multiagent Systems, p.77-85.

[20]Vincent R, Ait-Ahmed M, Houari A, et al., 2020. Residential microgrid energy management considering flexibility services opportunities and forecast uncertainties. Int J Electr Power Energy Syst, 120:105981.

[21]Williams CKI, Rasmussen CE, 1995. Gaussian processes for regression. Proc 8^th Int Conf on Neural Information Processing Systems, p.514-520.

[22]Xia D, Yuan M, Zhang CH, 2021. Statistically optimal and computationally efficient low rank tensor completion from noisy entries. Ann Stat, 49(1):76-99.

[23]Xu HC, Domínguez-García AD, Sauer PW, 2020. Optimal tap setting of voltage regulation Transformers using batch reinforcement learning. IEEE Trans Power Syst, 35(3):1990-2001.

[24]Zhang YY, Rao XP, Liu CY, et al., 2023. A cooperative EV charging scheduling strategy based on double deep Q-network and prioritized experience replay. Eng Appl Artif Intell, 118:105642.

[25]Zhao LY, Yang T, Li W, et al., 2022. Deep reinforcement learning-based joint load scheduling for household multi-energy system. Appl Energy, 324:119346.