Actor-critic deep learning for efficient user association and bandwidth allocation in dense mobile networks with green base stations

Wireless Networks - In this paper, we introduce an efficient user-association and bandwidth-allocation scheme based on an actor-critic deep learning framework for downlink data transmission in...     

Deep reinforcement learning based resource allocation algorithm in cellular networks

In order to solve multi-objective optimization problem,a resource allocation algorithm based on deep reinforcement learning in cellular networks was proposed.Firstly,deep neural network (DNN) was built to optimize the transmission rate of cellular system and to complete the forward transmission process of the algorithm.Then,the Q-learning mechanism was utilized to construct the error function,which used energy efficiency as the rewards.The gradient descent method was used to train the weights of DNN,and the reverse training process of the algorithm was completed.The simulation results show that the proposed algorithm can determine optimization extent of optimal resource allocation scheme with rapid convergence ability,it is obviously superior to the other algorithms in terms of transmission rate and system energy consumption optimization.     

车联网场景下移动边缘计算协作式资源分配策略

为了有效利用边缘云的计算资源,尽可能降低任务卸载时的平均等待时延,提出了一种满足边缘计算服务器容限阈值和任务卸载成功率约束条件下的多个边缘计算服务器相互协作的资源分配方案,通过单位时间总代价指标优化边缘计算服务器个数.将此方案建模为一个整数优化问题,之后设计了一种最小代价算法求解此优化问题,得到约束条件下的单位时间总代...     

基于深度强化学习的反向散射网络资源分配机制

为了提升反向散射网络中物联网设备的平均吞吐量,提出了一种资源分配机制,构建了用户配对和时隙分配联合优化资源分配模型。由于该模型直接利用深度强化学习(Deep Reinforcement Learning,DRL )算法求解导致动作空间维度较高且神经网络复杂,故将其分解为两层子问题以降低动作空间维度：首先,基于深度强化学习算法,利用历史信道信息推断当前的信道信息以进行最优的用户配对;然后,在用户固定配对的情况下,基于凸优化算法,以最大化物联网设备总吞吐量为目标进行最优的时隙分配。仿真结果表明,与其他资源分配方法相比,所提资源分配方法能有效提升系统吞吐量,且有较好的信道适应性和收敛性。     

Service migration in mobile edge computing: A deep reinforcement learning approach

In mobile edge computing, service migration can not only reduce the access latency but also reduce the network costs for users. However, due to bandwidth bottleneck, migration costs should also be considered during service migration. In this way, the trade-off between benefits of service migration and total service costs is very important for the cloud service providers. In this paper, we propose an efficient dynamic service migration algorithm named SMDQN, which is based on reinforcement learning. We consider each mobile application service can be hosted on one or more edge nodes and each edge node has limited resources. SMDQN takes total delay and migration costs into consideration. And to reduce the size of Markov decision process space, we devise the deep reinforcement learning algorithm to make a fast decision. We implement the algorithm and test the performance and stability of it. The simulation result shows that it can minimize the service costs and adapt well to different mobile access patterns.     

Distributed resource management in wireless sensor networks using reinforcement learning

In wireless sensor networks (WSNs), resource-constrained nodes are expected to operate in highly dynamic and often unattended environments. Hence, support for intelligent, autonomous, adaptive and distributed resource management is an essential ingredient of a middleware solution for developing scalable and dynamic WSN applications. In this article, we present a resource management framework based on a two-tier reinforcement learning scheme to enable autonomous self-learning and adaptive applications with inherent support for efficient resource management. Our design goal is to build a system with a bottom-up approach where each sensor node is responsible for its resource allocation and task selection. The first learning tier (micro-learning) allows individual sensor nodes to self-schedule their tasks by using only local information, thus enabling a timely adaptation. The second learning tier (macro-learning) governs the micro-learners by tuning their operating parameters so as to guide the system towards a global application-specific optimization goal (e.g., maximizing the network lifetime). The effectiveness of our framework is exemplified by means of a target tracking application built on top of it. Finally, the performance of our scheme is compared against other existing approaches by simulation. We show that our two-tier reinforcement learning scheme is significantly more efficient than traditional approaches to resource management while fulfilling the application requirements.     

基于深度强化学习的工业物联网多用户频谱分配

随着5G时代的来临,工业物联网将迎来蓬勃发展.然而,联网设备数量的不断增加,加剧了有限的频谱资源与大量的通信需求之间的矛盾.针对以上问题,提出了一种基于聚类分组和深度强化学习的合作式动态频谱分配算法,使用户可以获得较低的信息传输中断概率以及较少的多跳转发次数,快速找到信息传输的最优路径.在动态频谱分配中,该算法可以有效...     

Dynamic resource allocation in mobile heterogeneous cellular networks

User mobility is a challenging issue in macro and femto cellular networks for the fifth-generation and newer mobile communications due to the time-varying interference and topology experienced. In this paper, we consider an OFDMA-based two-tier network with one macro cell and several femto cells, wherein each macro user and/or femto user can leave or enter its serving cell frequently, referred to as user mobility. A resource allocation problem with different rate requirements of mobile users is then formulated. Assuming well knowledge of the user locations and the channel state information, we propose a dynamic algorithm with static and dynamic parts for a better trade-of between computational complexity and system throughput. The static algorithm, named interference weighted cluster algorithm in this paper, is based on the graph theory to cluster the femtocells by minimizing the interference between clusters, while the dynamic algorithm is to deal with the user mobility by sharing the resource blocks under the constraints of rate requirements. Numerical results are demonstrated to show the effectiveness of the proposed dynamic resource allocation algorithm in terms of capacity, computational time, and outage probability.

     

Secured data offloading using reinforcement learning and Markov decision process in mobile edge computing

Mobile Internet services are developing rapidly for several applications based on computational ability such as augmented/virtual reality, vehicular networks, etc. The mobile terminals are enabled using mobile edge computing (MEC) for offloading the task at the edge of the cellular networks, but offloading is still a challenging issue due to the dynamism, and uncertainty of upcoming IoT requests and wireless channel state. Moreover, securing the offloading data enhanced the challenges of computational complexities and required a secure and efficient offloading technique. To tackle the mentioned issues, a reinforcement learning-based Markov decision process offloading model is proposed that optimized energy efficiency, and mobile users' time by considering the constrained computation of IoT devices, moreover guarantees efficient resource sharing among multiple users. An advanced encryption standard is employed in this work to fulfil the requirements of data security. The simulation outputs reveal that the proposed approach surpasses the existing baseline models for offloading overhead and service cost QoS parameters ensuring secure data offloading.     

Cross-layer radio resource allocation in packet CDMA wireless mobile networks

An efficient radio resource allocation scheme is crucial for guaranteeing the quality of service (QoS) requirements and fully utilizing the scarce radio resources in wireless mobile networks. Most of previous studies of radio resource allocation in traditional wireless networks concentrates on network layer connection blocking probability QoS. In this paper, we show that physical layer techniques and QoS have significant impacts on network layer QoS. We use a concept of cross-layer effective bandwidth to measure the unified radio resource usage taking into account both physical layer linear minimum-mean square error (LMMSE) receivers and varying statistical characteristics of the packet traffic in code devision multiple access (CDMA) networks. We demonstrate the similarity between traditional circuit-switched networks and packet CDMA networks, which enables rich theories developed in traditional wireless mobile networks to be used in packet CDMA networks. Moreover, since both physical layer signal-to-interference ratio (SIR) QoS and network layer connection blocking probability QoS are considered simultaneously, we can explore the tradeoff between physical layer QoS and network layer QoS in packet CDMA networks. This work is supported by Natural Science and Engineering Research Council of Canada. Please address all correspondence to Professor Vikram Krishnamurthy at the above address. Fei Yu received the Ph.D. degree in electrical engineering from the University of British Columbia in 2003. From 2002 to 2004, he was with Ericsson (in Lund, Sweden), where he worked on the research and development of dual mode UMTS/GPRS handsets. From 2005, he has been working in Silicon Valley at a start-up, where he conducts research and development in the areas of advanced wireless communication technologies and new standards. After completing the PhD, he has been a research associate in the Department of Electrical and Computer Engineering at the University of British Columbia. His research interests include cross-layer optimization, QoS provisioning and security in wireless networks. Vikram Krishnamurthy (S’90-M’91-SM’99-F’05) was born in 1966. He received his bachelor’s degree from the University of Auckland, New Zealand in 1988, and Ph.D. from the Australian National University, Canberra, in 1992. Since 2002, he has been a professor and Canada Research Chair at the Department of Electrical Engineering, University of British Columbia, Vancouver, Canada. Prior to this he was a chaired professor at the Department of Electrical and Electronic Engineering, University of Melbourne, Australia. His research interests span several areas including ion channels and nanobiology, stochastic scheduling and control, statistical signal processing and wireless telecommunications. Dr. Krishnamurthy has served as associate editor for IEEE Transactions on Signal Processing, IEEE Transactions Aerospace and Electronic Systems, IEEE Transactions Nanobioscience, IEEE Transactions Circuits and Systems II, Systems and Control Letters and European Journal of Applied Signal Processing. He was guest editor of a special issue of IEEE Transactions on NanoBioScience, March 2005 on bio-nanotubes.     

Multi-agent reinforcement learning for edge information sharing in vehicular networks

To guarantee the heterogeneous delay requirements of the diverse vehicular services, it is necessary to design a full cooperative policy for both Vehicle to Infrastructure (V2I) and Vehicle to Vehicle (V2V) links. This paper investigates the reduction of the delay in edge information sharing for V2V links while satisfying the delay requirements of the V2I links. Specifically, a mean delay minimization problem and a maximum individual delay minimization problem are formulated to improve the global network performance and ensure the fairness of a single user, respectively. A multi-agent reinforcement learning framework is designed to solve these two problems, where a new reward function is proposed to evaluate the utilities of the two optimization objectives in a unified framework. Thereafter, a proximal policy optimization approach is proposed to enable each V2V user to learn its policy using the shared global network reward. The effectiveness of the proposed approach is finally validated by comparing the obtained results with those of the other baseline approaches through extensive simulation experiments.     

Real-time routing algorithm for mobile ad hoc networks using reinforcement learning and heuristic algorithms

Mobile ad hoc networks (MANETs) consist of a set of nodes which can move freely and communicate with each other wirelessly. Due to the movement of nodes and unlike wired networks, the available routes used among the nodes for transmitting data packets are not stable. Hence, proposing real-time routing protocols for MANETs is regarded as one of the major challenges in this research domain. Algorithms compatible with the changes created in the network due to the nodes’ movements are of high significance. For reducing data packet transmission time among nodes, not only should route shortness be considered but also route stability should be taken into consideration. Since available factors in different environments have specific behavior patterns especially in human environments, the parameters of link stability and route shortness were taken into consideration and the reinforcement learning was used to propose a method so as to make the best choice among the neighbors at any moment to transmit a packet to the destination. That is, the proposed method was aimed at predicting the behavior pattern of the nodes in relation to the target node through using reinforcement learning. The proposed method used Q-learning algorithm which has more homogeneity to estimate the value of actions. Simulation results in OPNET demonstrate the superiority of the proposed scheme over conventional MANET routing methods.     

Flexible resource allocation strategies for class-based QoS provisioning in mobile networks

Flexible resource-allocation (FRA) strategies have been proposed in the literature to mitigate the high blocking rate caused in high-speed mobile communication networks when resource aggregation is used to increase the data rate. In this paper, new FRA strategies that cope with scenarios with multiple service types and multiple priorities are proposed. These are called the FRA strategy with differentiated priorities and quality of service (FRAQoS) and the FRA strategy with prioritized levels (FRASPL). The main distinguishing feature of these strategies is their capacity to prioritize some service types over others. FRAQoS prioritizes the quality of service (QoS) of particular service types over others by introducing the concepts of prioritized call degradation and compensation. However, FRAQoS provides a limited ability to prioritize particular service types over the rest because any incoming call, irrespective of its service type and priority, may trigger resource reallocations to service it. The FRASPL overcomes this limitation by introducing a call-admission mechanism that, when necessary, rejects calls originated by low-priority service types. This enhances the protection to high-priority service types. By prioritizing some services over the others, FRASPL can trade off capacity against QoS. FRAQoS represents particular limiting cases of FRASPL. A mathematical model is developed to investigate the performance of FRAQoS and FRASPL. Then, they are compared with other FRA strategies in a scenario with multiple differentiated service types. Results show that the proposed strategies effectively prioritize service types, providing them with better QoS. This makes FRAQoS and FRASPL specially suitable for class-based QoS provisioning in mobile networks.     

可移动基站网络频谱资源块动态分配方法研究

首先从应用需求,引出了移动式基站的概念,然后设计了一种针对该网络资源分配的优先级策略并从提 高资源利用率和降低频谱重分配次数2个角度出发,提出了2种基于图论的资源分配策略,最后完整地提出了一 种低复杂度的分布式频谱资源块动态分配算法。仿真结果表明,所提算法有良好的性能。     

Joint optimization of power control and time slot allocation for wireless body area networks via deep reinforcement learning

Wireless Networks - E-healthcare system based on wireless body area network (WBAN) promises to produce potential benefits in health-care industry. A major issue of such an on-body networked system...     

Deep reinforcement learning-based joint task offloading and bandwidth allocation for multi-user mobile edge computing

The rapid growth of mobile internet services has yielded a variety of computation-intensive applications such as virtual/augmented reality. Mobile Edge Computing (MEC), which enables mobile terminals to offload computation tasks to servers located at the edge of the cellular networks, has been considered as an efficient approach to relieve the heavy computational burdens and realize an efficient computation offloading. Driven by the consequent requirement for proper resource allocations for computation offloading via MEC, in this paper, we propose a Deep-Q Network (DQN) based task offloading and resource allocation algorithm for the MEC. Specifically, we consider a MEC system in which every mobile terminal has multiple tasks offloaded to the edge server and design a joint task offloading decision and bandwidth allocation optimization to minimize the overall offloading cost in terms of energy cost, computation cost, and delay cost. Although the proposed optimization problem is a mixed integer nonlinear programming in nature, we exploit an emerging DQN technique to solve it. Extensive numerical results show that our proposed DQN-based approach can achieve the near-optimal performance.     

深度强化学习在物联网边缘计算中的应用研究

针对传统的物联网边缘计算方法存在计算成本过高,计算时间过长等问题,文中引入了深度强化学习技术,对物联网边缘计算方法进行优化.通过物联网拓扑结构设定物联网边缘计算周期,获取数据上传速度.设计边缘计算执行过程,提升边缘计算资源分配效率.引入深度强化学习技术中的CNN模型实现卷积计算,完成物联网边缘计算的资源分配.至此,实现...     

基于深度学习的异构资源分配算法研究

资源分配是目前云计算领域中一个重要的研究方向。在异构云计算体系结构下的复杂应用问题研究中,为了满足异构资源分配的需求,提升资源利用效率,文中提出了一种基于深度学习的面向应用的资源分配算法。该算法将数据特征进行量化,更加精确地刻画了不同服务器资源之间的性能差异,在分配算法中加入了一个工作负载预测模型,使给出的资源分配方案与需求更加匹配,同时提高了资源利用率。     

Radio resource allocation for interference management in mobile broadband OFDMA based networks

This paper focuses on the inter‐cell interference (ICI) management problem in the downlink channel for mobile broadband wireless OFDMA‐based systems. This subject is addressed from the standpoint of different interrelated resource allocation mechanisms operating in multi‐cell scenarios in order to exploit frequency and multi‐user diversity: ICI coordination/avoidance and adaptive subcarrier and power allocation. Even though these methods can be applied in a stand‐alone way, a significant performance improvement is achieved if they are jointly designed and operate in a combined basis. Several alternatives for mixed frequency and power ICI coordination schemes are proposed in this paper. Connected with a proper power mask‐based design, the potential gain of a flexible frequency sectorization solution, halfway between fractional/soft frequency reuse and pure frequency sectorization, is explored. The main objective is to outperform fractional/soft frequency reuse offering an attractive trade‐off between cell‐edge user data rates and average cell throughput. Proposals concerning ICI coordination/avoidance have been evaluated in combination with several heuristic adaptive subcarrier and power allocation algorithms. Copyright © 2009 John Wiley & Sons, Ltd.     

Radio resource allocation using genetic algorithm in heterogeneous networks

Radio resource management to enable user association and resource block (RB) allocation is crucial for enhancing the performance of heterogeneous networks (HetNets), which are required for fifth generation (5G) mobile networks. This paper proposed a resource allocation technique based on a genetic algorithm (GA) for use in HetNets. We aimed to optimize user association and RB allocation simultaneously to fulfill multiple objectives, such as throughput and fairness measure. In addition to the four primary phases used in GA process, namely initialization, crossover, mutation, and selection, a further operator was provided for managing illegal offspring generated during a GA process. We performed a simulation to compare the proposed GA‐based approach with best channel quality indicator (CQI) algorithm and integer linear programming (ILP) approach in terms of total throughput and fairness measure. The simulation results revealed that the total throughput obtained using the proposed approach is 32.7% and 37.6% better than that obtained using the ILP and best CQI approaches, respectively. Moreover, the fairness measure obtained using the proposed GA‐based approach was 31.8% and 33.2% higher than that obtained using ILP and best CQI approaches, respectively.