超密集网络中基于改进DQN的接入选择算法

在超密集网络环境中，各个接入点密集部署在热点区域，构成了复杂的异构网络，用户需要选择接入合适的网络以获得最好的性能。如何为用户选择最优的网络，使用户自身或网络性能达到最佳，称为网络接入选择问题。为了解决超密集网络中用户的接入选择问题，综合考虑网络状态、用户偏好以及业务类型，结合负载均衡策略，提出了一种基于改进深度Q网络(deep Q network, DQN)的超密集网络接入选择算法。首先，通过分析网络属性和用户业务的偏好对网络选择的影响，选择合适的网络参数作为接入选择算法的参数；其次，将网络接入选择问题利用马尔可夫决策过程建模，分别对模型中的状态、动作和奖励函数进行设计；最后，利用DQN求解选网模型，得到最优选网策略。此外，为了避免DQN过高估计Q值，对传统DQN的目标函数进行优化，并且在训练神经网络时，引入了优先经验回放机制以提升学习效率。仿真结果表明，所提算法能够解决传统DQN的高估问题，加快神经网络的收敛，有效减少用户的阻塞，并改善网络的吞吐能力。

Access selection algorithm based on improved DQN for ultra-dense networks

In the ultra-dense network environment, each access point is deployed in the hotspot area, which forms a complex heterogeneous network. Users need to choose the appropriate network to access, so as to achieve the best performance. Network selection problem is to choose the optimal network for the user, so that the user or network performance reaches the best. In order to solve the access selection problem of users in ultra-dense networks, we proposed an ultra-dense network access selection algorithm based on the improved deep Q network (DQN), considering network states, user preferences, and service types, and combining with load balancing strategies. First, by analyzing the influence of network attributes and user preferences on network selection, the appropriate network parameters were selected as the parameters of the access selection algorithm. Then, the problem of network access selection was modeled by Markov decision-making process, and the states, actions, and reward functions of the model were designed. Finally, the optimal network strategy was obtained by using DQN to solve the network selection model. In addition, the target function of traditional DQN was optimized to avoid overestimation of Q value by DQN, and a priority experience replay mechanism was introduced to improve learning efficiency. Simulation results show that the method could well solve the problem of overestimation of traditional DQN, accelerate the convergence of neural network, effectively reduce user congestion, and improve network throughput performance.