JZUS - Journal of Zhejiang University SCIENCE

ENGINEERING Information Technology & Electronic Engineering

Accepted manuscript available online (unedited version)

A multi-agent collaboration scheme for energy-efficient task scheduling in a 3D UAV-MEC space

Author(s): Yang LI, Ziling WEI, Jinshu SU, Baokang ZHAO
Affiliation(s): College of Computer, National University of Defense Technology, Changsha 410073, China; more
Corresponding email(s): liyang20@nudt.edu.cn, weiziling@nudt.edu.cn, sjs@nudt.edu.cn, bkzhao@nudt.edu.cn
Key Words: Multi-access edge computing; Multi-agent reinforcement learning; Unmanned aerial vehicles; Task scheduling

Share this article to： More <<< Previous Paper \|Next Paper >>>

Yang LI, Ziling WEI, Jinshu SU, Baokang ZHAO. A multi-agent collaboration scheme for energy-efficient task scheduling in a 3D UAV-MEC space[J]. Frontiers of Information Technology & Electronic Engineering,in press.https://doi.org/10.1631/FITEE.2300393

@article{title="A multi-agent collaboration scheme for energy-efficient task scheduling in a 3D UAV-MEC space",
author="Yang LI, Ziling WEI, Jinshu SU, Baokang ZHAO",
journal="Frontiers of Information Technology & Electronic Engineering",
year="in press",
publisher="Zhejiang University Press & Springer",
doi="https://doi.org/10.1631/FITEE.2300393"
}

%0 Journal Article
%T A multi-agent collaboration scheme for energy-efficient task scheduling in a 3D UAV-MEC space
%A Yang LI
%A Ziling WEI
%A Jinshu SU
%A Baokang ZHAO
%J Frontiers of Information Technology & Electronic Engineering
%P 824-838
%@ 2095-9184
%D in press
%I Zhejiang University Press & Springer
doi="https://doi.org/10.1631/FITEE.2300393"

TY - JOUR
T1 - A multi-agent collaboration scheme for energy-efficient task scheduling in a 3D UAV-MEC space
A1 - Yang LI
A1 - Ziling WEI
A1 - Jinshu SU
A1 - Baokang ZHAO
J0 - Frontiers of Information Technology & Electronic Engineering
SP - 824
EP - 838
%@ 2095-9184
Y1 - in press
PB - Zhejiang University Press & Springer
ER -
doi="https://doi.org/10.1631/FITEE.2300393"

Abstract
Chinese Summary
Academic Network
Reviewer Comment

Abstract: Multi-access edge computing (MEC) presents computing services at the edge of networks to address the enormous processing requirements of intelligent applications. Due to the maneuverability of unmanned aerial vehicles (UAVs), they can be used as temporal aerial edge nodes for providing edge services to ground users in MEC. However, MEC environment is usually dynamic and complicated. It is a challenge for multiple UAVs to select appropriate service strategies. Besides, most of existing works study UAV-MEC with the assumption that the flight heights of UAVs are fixed; i.e., the flying is considered to occur with reference to a two-dimensional plane, which neglects the importance of the height. In this paper, with consideration of the co-channel interference, an optimization problem of energy efficiency is investigated to maximize the number of fulfilled tasks, where multiple UAVs in a three-dimensional space collaboratively fulfill the task computation of ground users. In the formulated problem, we try to obtain the optimal flight and sub-channel selection strategies for UAVs and schedule strategies for tasks. Based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm, we propose a curiosity-driven and twin-networks-structured MADDPG (CTMADDPG) algorithm to solve the formulated problem. It uses the inner reward to facilitate the state exploration of agents, avoiding convergence at the sub-optimal strategy. Furthermore, we adopt the twin critic networks for update stabilization to reduce the probability of Q value overestimation. The simulation results show that CTMADDPG is outstanding in maximizing the energy efficiency of the whole system and outperforms the other benchmarks.

三维无人机-多接入边缘计算场景下的多智能体协作任务调度能效优化方案

李阳¹，魏子令¹，苏金树^1,2，赵宝康¹
¹国防科技大学计算机学院，中国长沙市，410073
²军事科学院，中国北京市，100091
摘要：针对智能应用算力处理需求，多接入边缘计算（multi-access edge computing，MEC）在网络边缘为其提供计算服务。无人机（unmanned aerial vehicle，UAV）具有良好机动性，可在MEC中作为临时空中边缘节点为地面用户提供边缘服务。然而，MEC环境复杂且动态可变，如何为多台无人机制定合适的服务策略具有一定挑战。此外，现有很多UAV-MEC相关工作均假定无人机飞行高度固定，即飞行在二维平面内，忽略了飞行高度的重要性。在同信道干扰存在的前提下，本文通过优化能效实现任务完成量的最大化，多台无人机在三维空间中共同协作为地面用户提供任务计算服务。为实现能效优化目标，最大化任务完成量并最小化飞行能耗，须制定最优飞行策略、子信道选择策略以及任务调度策略。基于多智能体深度确定性策略梯度算法（multi-agent deep deterministic policy gradient，MADDPG），本文提出好奇心驱动和双网络结构的多智能体深度确定性策略梯度算法（curiosity-driven and twin-networks-structured MADDPG，CTMADDPG）解决上述优化问题，通过内部奖励促进智能体的状态探索，避免收敛于次优策略。同时，利用双批评家网络降低Q值高估概率，实现稳定更新。仿真结果表明CTMADDPG算法在最大化整个系统能效方面表现突出，优于其他基准测试算法。

关键词组：多接入边缘计算；多智能体强化学习；无人机；任务调度

Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article

Reference

[1]Al-Hourani A, Kandeepan S, Lardner S, 2014. Optimal LAP altitude for maximum coverage. IEEE Wirel Commun Lett, 3(6):569-572.

[2]Ashraf Ateya AA, Muthanna A, Kirichek R, et al., 2019. Energy- and latency-aware hybrid offloading algorithm for UAVs. IEEE Access, 7:37587-37600.

[3]Badia AP, Sprechmann P, Vitvitskyi A, et al., 2020. Never give up: learning directed exploration strategies. Proc 8^th Int Conf on Learning Representations.

[4]Chakrabarty A, Langelaan J, 2009. Energy maps for long-range path planning for small- and micro-UAVs. AIAA Guidance, Navigation, and Control Conf, Article 6113.

[5]Dai C, Zhu K, Hossain E, 2022. Multi-agent deep reinforcement learning for joint decoupled user association and trajectory design in full-duplex multi-UAV networks. IEEE Trans Mob Comput, 22(10):6056-6070.

[6]Dai ZJ, Zhang Y, Zhang WC, et al., 2022. A multi-agent collaborative environment learning method for UAV deployment and resource allocation. IEEE Trans Signal Inform Process Netw, 8:120-130.

[7]Ding CF, Wang JB, Cheng M, et al., 2023. Dynamic transmission and computation resource optimization for dense LEO satellite assisted mobile-edge computing. IEEE Trans Commun, 71(5):3087-3102.

[8]Fujimoto S, van Hoof H, Meger D, 2018. Addressing function approximation error in actor-critic methods. Proc 35^th Int Conf on Machine Learning, p.1587-1596.

[9]Gu XH, Zhang GA, Wang MX, et al., 2021. UAV-aided energy-efficient edge computing networks: security offloading optimization. IEEE Int Things J, 9(6):4245-4258.

[10]Ji JQ, Zhu K, Yi CY, et al., 2021. Energy consumption minimization in UAV-assisted mobile-edge computing systems: joint resource allocation and trajectory design. IEEE Int Things J, 8(10):8570-8584.

[11]Jiang FB, Wang KZ, Dong L, et al., 2020. Deep-learning-based joint resource scheduling algorithms for hybrid MEC networks. IEEE Int Things J, 7(7):6252-6265.

[12]Joo S, Kang HG, Kang J, 2021. CoSMoS: cooperative sky-ground mobile edge computing system. IEEE Trans Veh Technol, 70(8):8373-8377.

[13]Lakew DS, Tran AT, Dao NN, et al., 2023. Intelligent offloading and resource allocation in heterogeneous aerial access IoT networks. IEEE Int Things J, 10(7):5704-5718.

[14]Liao ZF, Ma YB, Huang JW, et al., 2021. HOTSPOT: a UAV-assisted dynamic mobility-aware offloading for mobile-edge computing in 3-D space. IEEE Int Things J, 8(13):10940-10952.

[15]Liu JF, Li LX, Yang FC, et al., 2019. Minimization of offloading delay for two-tier UAV with mobile edge computing. Proc 15^th Int Wireless Communications & Mobile Computing Conf, p.1534-1538.

[16]Liu Q, Shi L, Sun LL, et al., 2020. Path planning for UAV-mounted mobile edge computing with deep reinforcement learning. IEEE Trans Veh Technol, 69(5):5723-5728.

[17]Liu XY, Xu C, Yu HB, et al., 2022. Multi-agent deep reinforcement learning for end–edge orchestrated resource allocation in industrial wireless networks. Front Inform Technol Electron Eng, 23(1):47-60.

[18]Mei HB, Yang K, Liu Q, et al., 2020. Joint trajectory-resource optimization in UAV-enabled edge-cloud system with virtualized mobile clone. IEEE Int Things J, 7(7):5906-5921.

[19]Tun YK, Park YM, Tran NH, et al., 2021. Energy-efficient resource management in UAV-assisted mobile edge computing. IEEE Commun Lett, 25(1):249-253.

[20]Wang JR, Liu KY, Pan JP, 2020. Online UAV-mounted edge server dispatching for mobile-to-mobile edge computing. IEEE Int Things J, 7(2):1375-1386.

[21]Wang JZ, Zhang XL, He XS, et al., 2023. Bandwidth allocation and trajectory control in UAV-assisted IoV edge computing using multiagent reinforcement learning. IEEE Trans Reliab, 72(2):599-608.

[22]Wang L, Wang KZ, Pan CH, et al., 2021. Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing. IEEE Trans Cogn Commun Netw, 7(1):73-84.

[23]Wang LY, Zhang HX, Guo SS, et al., 2022. Deployment and association of multiple UAVs in UAV-assisted cellular networks with the knowledge of statistical user position. IEEE Trans Wirel Commun, 21(8):6553-6567.

[24]Wang ZQ, Rong HG, Jiang HB, et al., 2022. A load-balanced and energy-efficient navigation scheme for UAV-mounted mobile edge computing. IEEE Trans Netw Sci Eng, 9(5):3659-3674.

[25]Wu SL, Xu WJ, Wang FY, et al., 2022. Distributed federated deep reinforcement learning based trajectory optimization for air-ground cooperative emergency networks. IEEE Trans Veh Technol, 71(8):9107-9112.

[26]Xia WC, Zhu YX, De Simone L, et al., 2022. Multiagent collaborative learning for UAV enabled wireless networks. IEEE J Sel Areas Commun, 40(9):2630-2642.

[27]Xu S, Zhang XY, Li CG, et al., 2022. Deep reinforcement learning approach for joint trajectory design in multi-UAV IoT networks. IEEE Trans Veh Technol, 71(3):3389-3394.

[28]Xu Y, Zhang TK, Loo J, et al., 2021. Completion time minimization for UAV-assisted mobile-edge computing systems. IEEE Trans Veh Technol, 70(11):12253-12259.

[29]Xue NS, 2014. Design and Optimization of Lithium-Ion Batteries for Electric-Vehicle Applications. PhD Thesis, The University of Michigan, Ann Arbor, United States.

[30]Yang L, Yao HP, Wang JJ, et al., 2020. Multi-UAV-enabled load-balance mobile-edge computing for IoT networks. IEEE Int Things J, 7(8):6898-6908.

[31]Yin ZY, Lin Y, Zhang YJ, et al., 2022. Collaborative multi-agent reinforcement learning aided resource allocation for UAV anti-jamming communication. IEEE Int Things J, 9(23):23995-24008.

[32]Yu Y, Bu XY, Yang K, et al., 2021. UAV-aided low latency multi-access edge computing. IEEE Trans Veh Technol, 70(5):4955-4967.

[33]Yu Z, Gong YM, Gong SM, et al., 2020. Joint task offloading and resource allocation in UAV-enabled mobile edge computing. IEEE Int Things J, 7(4):3147-3159.

[34]Zhang L, Ansari N, 2020. Latency-aware IoT service provisioning in UAV-aided mobile-edge computing networks. IEEE Int Things J, 7(10):10573-10580.

[35]Zhao N, Ye ZY, Pei YY, et al., 2022. Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing. IEEE Trans Wirel Commun, 21(9):6949-6960.

[36]Zheng LL, Chen JR, Wang JH, et al., 2021. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Proc 34^th Int Conf on Neural Information Processing Systems, p.3757-3769.

[37]Zhong RK, Liu X, Liu YW, et al., 2022. Multi-agent reinforcement learning in NOMA-aided UAV networks for cellular offloading. IEEE Trans Wirel Commun, 21(3):1498-1512.

[38]Zhou FH, Wu YP, Hu RQ, et al., 2018. Computation rate maximization in UAV-enabled wireless-powered mobile-edge computing systems. IEEE J Sel Areas Commun, 36(9):1927-1941.

Open peer comments: Debate/Discuss/Question/Opinion

<1>

- Go to

三维无人机-多接入边缘计算场景下的多智能体协作任务调度能效优化方案

Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article

Reference