CN104023384A

Movatterモバイル変換

Info

Publication number: CN104023384A
Application number: CN201410231267.1A
Authority: CN
Inventors: 李正富; 路兆铭; 温向明; 景文鹏; 张志才; 扶奉超; 何盛华; 陈昆
Original assignee: Beijing University of Posts and Telecommunications
Current assignee: Beijing University of Posts and Telecommunications
Priority date: 2014-05-28
Filing date: 2014-05-28
Publication date: 2014-09-03

Abstract

本发明实施例涉及一种家庭基站密集部署在宏蜂窝网络中上行链路的一种低复杂度的无线资源分配方法。该方法在最大化家庭基站上行链路的吞吐量的同时也考虑了延迟QoS指标，提出了子信道分配与功率控制算法。宏用户优先选择子信道和功率分配策略并广播给家庭基站，从而保证大量引入家庭基站并不会对宏用户产生较大的影响。用户的时延QoS指标是通过用户的有效容量的延迟因子来实现的。本发明通过强化学习中的Q_learning机制选择最优的子信道和功率分配策略，与非合作博弈方式比较，可以较快的达到均衡，而且该发明解决了瞬时吞吐量无法找到关于发射功率的闭式解的问题，实现了低复杂度的分配方案。

The embodiment of the present invention relates to a low-complexity wireless resource allocation method for densely deployed home base stations in uplink in a macro cellular network. In this method, the delay QoS index is also considered while maximizing the uplink throughput of the Femtocell, and a subchannel allocation and power control algorithm are proposed. The macro user preferentially selects the subchannel and power allocation strategy and broadcasts it to the home base station, so as to ensure that the introduction of a large number of home base stations will not have a great impact on the macro user. The user's delay QoS index is realized through the delay factor of the user's effective capacity. The present invention selects the optimal sub-channel and power allocation strategy through the Q_learning mechanism in reinforcement learning. Compared with the non-cooperative game mode, the equilibrium can be reached quickly, and the present invention solves the problem that the instantaneous throughput cannot find a closed formula for transmitting power. solve the problem and implement a low-complexity allocation scheme.

Description

Translated fromChinese

一种家庭基站密集部署场景中考虑时延限制的双层网络资源分配方法A two-layer network resource allocation method considering delay constraints in the dense deployment of femtocells

技术领域technical field

本发明涉及第四代移动通信技术领域，具体涉及在家庭基站密集部署的双层网络中上行链路的无线资源分配方法。此方案考虑了时延性能指标的同时，也保证了用户可达的最大吞吐量。 The present invention relates to the technical field of the fourth generation of mobile communication, in particular to an uplink wireless resource allocation method in a double-layer network in which home base stations are densely deployed. While this solution takes into account the delay performance index, it also guarantees the maximum throughput that can be achieved by users. the

背景技术Background technique

随着人们生活水平的提高，移动通信技术的飞速发展，各种移动多媒体业务逐渐流行，这造成了移动数据流量的指数式增长。统计显示，超过70％的移动数据业务发生在室内，因此，为了解决室内覆盖不足以及为室内用户提供高速无线数据接入的问题，家庭基站技术应运而生。家庭基站是一种用户部署的、即插即用型低功率小型基站，主要应用于家庭、公司、商场等场所，解决室内热点地区的覆盖问题。在家庭基站和宏基站异构部署的双层网络环境中，由于频谱资源的稀缺性，家庭小区与宏小区一般采用共享频谱的策略，这样就会造成家庭小区与宏小区间的跨层干扰。同时，在办公楼、大型商场和居民区等家庭基站密集部署的区域，相邻家庭基站之间也存在着同层干扰的问题。严重的同层干扰和跨层干扰不仅会大大降低家庭基站小区的网络容量，而且还会影响宏基站用户的QoS，是家庭基站部署时必须要解决的问题之一。 With the improvement of people's living standards and the rapid development of mobile communication technology, various mobile multimedia services are gradually becoming popular, which results in an exponential growth of mobile data traffic. Statistics show that more than 70% of mobile data services occur indoors. Therefore, in order to solve the problem of insufficient indoor coverage and provide high-speed wireless data access for indoor users, femtocell technology has emerged as the times require. The femtocell is a plug-and-play low-power small base station deployed by users, which is mainly used in homes, companies, shopping malls and other places to solve the coverage problem of indoor hot spots. In a two-layer network environment where home base stations and macro base stations are deployed heterogeneously, due to the scarcity of spectrum resources, home cells and macro cells generally adopt a strategy of sharing spectrum, which will cause cross-layer interference between home cells and macro cells. At the same time, in areas where femtocells are densely deployed, such as office buildings, large shopping malls, and residential areas, there is also the problem of same-layer interference between adjacent femtocells. Severe same-layer interference and cross-layer interference will not only greatly reduce the network capacity of the femtocell, but also affect the QoS of the macro base station users, which is one of the problems that must be solved when the femtocell is deployed. the

另外，随着数据业务的迅速增加，网络的延迟相对变大，然而人们对数据吞吐量的要求越来越高，因此提出保证用户时延QoS指标并使用户吞吐量最大化的策略分配方案极为重要。 In addition, with the rapid increase of data services, the delay of the network is relatively large, but people's requirements for data throughput are getting higher and higher, so it is extremely important to propose a strategy allocation scheme that guarantees user delay QoS indicators and maximizes user throughput. important. the

当前学术界和工业界关于家庭基站的研究主要关注于最大化家庭基站的小区容量，减小同层干扰或者跨层干扰，而关于家庭基站密集部署场景下的资源管理方案的能效提升一直未得到足够重视。虽然也有学者提出了一些低复杂度的子信道分配方法，但是并没有考虑相邻家庭基站的干扰问题。所以本发明就是针对家庭基站网络中的子信道分配与功率分配联合进行时的复杂性，提出了一种分步进行子信道分配与功率控制的分布式无线资源管理方案。 Current research on femtocells in academia and industry mainly focuses on maximizing the cell capacity of femtocells and reducing same-layer or cross-layer interference. However, the energy efficiency improvement of resource management schemes in densely deployed femtocells has not been achieved. enough attention. Although some scholars have proposed some low-complexity sub-channel allocation methods, they have not considered the interference problem of adjacent femtocells. Therefore, the present invention proposes a distributed wireless resource management scheme for step-by-step sub-channel allocation and power control, aiming at the complexity of joint sub-channel allocation and power allocation in the home base station network. the

发明内容Contents of the invention

本发明旨在针对密集部署的OFDMA家庭基站网络，解决如何在最大化用户吞吐量的同时保证延迟QoS指标。针对子信道与功率联合分配算法的复杂性，提出了分步进行子信道分配与功率分配的分布式方案。 The invention aims at solving the problem of how to guarantee the delay QoS index while maximizing the user throughput for the densely deployed OFDMA home base station network. Aiming at the complexity of the sub-channel and power joint allocation algorithm, a distributed scheme for step-by-step sub-channel allocation and power allocation is proposed. the

为了实现上述目的，解决相应的技术问题，本发明具体通过如下步骤来实现： In order to achieve the above object and solve the corresponding technical problems, the present invention is specifically realized through the following steps:

步骤1：确定子信道的数量，根据经验对家庭用户和宏用户的发射功率离散化，并初始化其发射功率。 Step 1: Determine the number of sub-channels, discretize the transmit power of home users and macro users according to experience, and initialize their transmit power. the

步骤2：T＝0时刻初始化宏用户和家庭用户的Q_值。 Step 2: Initialize the Q_values of macro users and home users at time T=0. the

步骤3：T+1时刻到来时，宏用户和家庭用户根据(1)式更新策略。 Step 3: When the time T+1 arrives, the macro user and the home user update the policy according to formula (1). the

${v v}_{i i,, {a a}_{i i}}^{n no,, t t + + 11} = = \frac{exp exp (({Q Q}_{i i}^{t t,, n no} (({p p}_{i i})) / / τ τ))}{\underset{{p p}_{i i} &Element; &Element; {P P}_{i i}}{Σ Σ} exp exp (({Q Q}_{i i}^{t t,, n no} (({p p}_{i i})) / / τ τ))} - - - - - - ((11))$

其中，当进行子信道分配时，是随着子信道n变化的；当进行功率控制时是随着功率p_i变化的。 Among them, when performing sub-channel allocation, It varies with subchannel n; when performing power control It varies with the power p_i .

步骤4：宏用户(MU)根据T时刻的策略选择使Q_值最大的动作(传输子信道或发射功率)，并广播给家庭用户(FU) Step 4: The macro user (MU) selects the action (transmission sub-channel or transmission power) that maximizes the Q_value according to the strategy at time T, and broadcasts it to the home user (FU)

步骤5：家庭用户(FU)根据宏用户广播的策略和T时刻MU的策略选择最优动作，并将其策略信息发送给MU。 Step 5: The home user (FU) chooses the optimal action according to the strategy broadcast by the macro user and the strategy of the MU at time T, and sends its strategy information to the MU. the

步骤6：MU根据(2)式更新其立即回报，FU根据(3)式更新立即回报。 Step 6: MU updates its immediate reward according to formula (2), and FU updates its immediate reward according to formula (3). the

$\begin{matrix} {R R}_{00} = = {C C}_{00} (({p p}_{00}^{n no} (({a a}_{00})),, {\overset{&RightArrow; &Right Arrow;}{v v}}_{- - 00}^{n no,, t t + + 11})) \\ = = \underset{{p p}_{- - 00} &Element; &Element; {ρ ρ}_{- - 00}}{Σ Σ} {C C}_{00} (({p p}_{00}^{n no} (({a a}_{00})),, {p p}_{- - 00}^{n no} (({a a}_{- - 00})))) \underset{\underset{j j &NotEqual; &NotEqual; 00}{j j &Element; &Element; I I}}{Π Π} {v v}_{j j,, {a a}_{j j}}^{n no,, t t + + 11} \end{matrix} - - - - - - ((22))$

${R R}_{i i} = = {\overset{~ ~}{C C}}_{i i}^{t t + + 11} (({p p}_{i i}^{n no} (({a a}_{i i})),, {\overset{&RightArrow; &Right Arrow;}{v v}}_{- - i i}^{n no,, t t + + 11})) = = \underset{{p p}_{00}^{n no} &Element; &Element; {ρ ρ}_{00}}{Σ Σ} {v v}_{00}^{n no,, t t + + 11} {\overset{~ ~}{C C}}_{i i}^{t t + + 11} (({p p}_{i i}^{n no} (({a a}_{i i})),, {p p}_{00}^{n no})) - - - - - - ((33))$

其中，为宏用户的有效容量，为通过宏用户选择的策略对家庭用户有效容量的估计， in, is the effective capacity of the macro user, is an estimate of the effective capacity of a household user via a macro user-selected policy,

${\overset{~ ~}{C C}}_{i i}^{t t + + 11} (({p p}_{i i}^{n no} (({a a}_{i i})),, {p p}_{00}^{n no} (({a a}_{00})))) = = \{\begin{matrix} \frac{{C C}_{i i}^{n no} (({p p}_{i i}^{n no} (({a a}_{i i})),, {p p}_{- - i i}^{n no})) - - {\overset{~ ~}{C C}}_{i i}^{t t} (({p p}_{i i}^{n no} (({a a}_{i i})),, {p p}_{00}^{n no,, t t + + 11} (({a a}_{00}))))}{{n no}^{t t} (({p p}_{i i}^{n no} (({a a}_{i i})),, {p p}_{00}^{n no} (({a a}_{00})))) + + 11} + + {\overset{~ ~}{C C}}_{i i}^{t t} (({p p}_{i i}^{n no} (({a a}_{i i})),, {p p}_{00}^{n no,, t t} (({a a}_{00})))),, {p p}_{00}^{n no,, t t + + 11} (({a a}_{00})) = = {p p}_{00}^{n no,, t t} (({a a}_{00})) \\ {\overset{~ ~}{C C}}_{i i}^{t t} (({p p}_{i i}^{n no} (({a a}_{i i})),, {p p}_{00}^{n no,, t t} (({a a}_{00})))),, otherwise otherwise \end{matrix}$

其中是宏用户选择策略家庭用户选择策略的次数。 in is the macro user selection policy Home User Selection Strategies times.

步骤7：宏用户和家庭用户通过(4)式更新Q_值。 Step 7: The macro user and the home user update the Q_ value through formula (4). the

${Q Q}_{i i}^{t t + + 11} (({p p}_{i i}^{n no} (({a a}_{i i})),, {\overset{&RightArrow; &Right Arrow;}{v v}}_{- - i i}^{n no,, t t + + 11})) = = {Q Q}_{i i}^{t t} (({p p}_{i i}^{n no} (({a a}_{i i})),, {\overset{&RightArrow; &Right Arrow;}{v v}}_{- - i i}^{n no,, t t})) + + α α (({R R}_{i i} - - {Q Q}_{i i}^{t t} (({p p}_{i i}^{n no} (({a a}_{i i})),, {\overset{&RightArrow; &Right Arrow;}{v v}}_{- - i i}^{n no,, t t})))) - - - - - - ((44))$

其中，α为学习率。 Among them, α is the learning rate. the

步骤8：判断子信道或者功率分配是否达到均衡，即Q_值是否收敛、稳定，或者达到最大迭代次数。如果上述条件都未满足，则返回步骤3，继续进行子信道分配或者功率分配；若满足上述的任意一个条件，则退出迭代，进入步骤9。 Step 8: Determine whether the sub-channel or power allocation is balanced, that is, whether the Q_ value is converged, stable, or reaches the maximum number of iterations. If none of the above conditions is met, return to step 3 and continue subchannel allocation or power allocation; if any of the above conditions is met, exit the iteration and enter step 9. the

步骤9：取Q值收敛时分配的子信道和宏用户和家庭用户选择的功率作为发射功率。 Step 9: Take the subchannel allocated when the Q value converges and the power selected by the macro user and the home user as the transmission power. the

说明： illustrate:

在步骤1中，子信道数量在子信道分配中是作为可选择的策略种类。根据经验将用户发射功率离散化，离散化的发射功率将在后续步骤4，5，6中用到，并作为选择的功率分配策略。 In step 1, the number of subchannels is used as an optional policy category in subchannel allocation. Discretize user transmit power based on experience, and the discretized transmit power will be used in subsequent steps 4, 5, and 6, and used as the selected power allocation strategy. the

在步骤2中，为初始化强化学习中Q_学习机制中的Q_值。 In step 2, to initialize the Q_value in the Q_learning mechanism in reinforcement learning. the

在步骤3中，是利用玻尔兹曼分布更新子信道分配和功率控制策略。 In step 3, the subchannel allocation and power control strategies are updated using the Boltzmann distribution. the

在步骤4中，宏用户根据更新的策略优先选择动作，并将其策略空间广播给家庭用户，使家庭用户依据宏用户的策略空间选择策略，保证家庭用户不会对宏用户产生较大的干扰。 In step 4, the macro user preferentially selects an action according to the updated policy, and broadcasts its policy space to the home users, so that the home user selects a policy according to the policy space of the macro user, ensuring that the home user will not cause great interference to the macro user . the

在步骤5中，家庭用户的策略空间信息传送给宏用户，使得宏用户可以参考家庭用户执行的策略选择最优动作。 In step 5, the policy space information of the home user is transmitted to the macro user, so that the macro user can refer to the policy executed by the home user to select the optimal action. the

在步骤6中，立即回报是以用户的有效容量作为评价指标。 In step 6, the immediate return is based on the user's effective capacity as an evaluation index. the

有效容量： Effective capacity:

$C C (({p p}_{i i}^{n no},, {p p}_{- - i i}^{n no})) = = - - \frac{11}{θ θ} ln ln ((E E. [[exp exp ((- - θ θ \underset{n no &Element; &Element; N N}{Σ Σ} {R R}_{i i}^{n no} (({p p}_{i i}^{n no},, {p p}_{- - i i}^{n no}))))]])) - - - - - - ((55))$

${R R}_{i i}^{n no} (({p p}_{i i}^{n no},, {p p}_{- - i i}^{n no})) = = w w {log log}_{22} ((11 + + \frac{{p p}_{i i}^{n no} {h h}_{ii i}^{n no}}{\underset{j j &NotEqual; &NotEqual; i i}{Σ Σ} {p p}_{j j}^{n no} {h h}_{ji the ji}^{n no} + + Σ Σ {p p}_{s the s}^{n no} {g g}_{si the si}^{n no} + + {σ σ}^{22}})) - - - - - - ((66))$

其中θ为延迟QoS指数，为用户i在子信道n上的发射功率为的香农容量，为用户i到其所在基站i在子信道n上的增益，为占用子信道n的宏用户s到基站i的增益。 where θ is the delay QoS index, The transmit power of user i on subchannel n is The Shannon capacity of is the gain from user i to base station i on subchannel n, is the gain from macro user s occupying subchannel n to base station i.

宏用户可以知道所有家庭用户的策略，所以可以用(2)式计算执行策略的立即回报，而家庭用户并不知道其他家庭用户的策略信息，仅仅接收到宏用户传来的策略信息，另外家庭用户可以感知到其他用户对他产生的总干扰，因此我们采用(3)式估计家庭用户的立即回报。 The macro user can know the policies of all home users, so the execution policy can be calculated by formula (2) , and the home user does not know the policy information of other home users, but only receives the policy information from the macro user. In addition, the home user can perceive the total interference from other users, so we use formula (3) Estimate immediate returns for home users.

在步骤8中，最大迭代次数通过仿真得到经验值。 In step 8, the maximum number of iterations is empirically obtained through simulation. the

从以上技术方案可以看出，本发明的技术方案是通过将家庭基站密集部署场景下的子信道分配与功率分配分解成两步来处理：首先通过经验初始化用户的发射功率，以子信道为策略空间通过Q_学习找到最优的子信道分配方案，然后根据分配好的子信道，以离散化的功率为策略空间通过Q_学习机制进行功率控制。 From the above technical solutions, it can be seen that the technical solution of the present invention is to decompose the sub-channel allocation and power allocation in the scenario of densely deployed home base stations into two steps: first, initialize the transmit power of the user through experience, and use the sub-channel as the strategy The space uses Q_learning to find the optimal subchannel allocation scheme, and then according to the allocated subchannels, the discretized power is used as the strategy space to perform power control through the Q_learning mechanism. the

下面通过附图和具体实施例对本发明的技术方案进行进一步的阐述。 The technical solutions of the present invention will be further described below with reference to the drawings and specific embodiments. the

附图说明Description of drawings

图1为本发明专利具体适用的场景图； Figure 1 is a scene diagram of the specific application of the patent of the present invention;

图2为本发明中家庭基站进行子信道分配的流程图； Fig. 2 is the flow chart that home base station carries out sub-channel allocation in the present invention;

图3为本发明中家庭基站进行功率控制的流程图； Fig. 3 is the flowchart of the power control of the home base station in the present invention;

具体实施方式Detailed ways

为使本发明优势描述更加清楚，下面结合附图对本发明的具体实施方式进一步详细阐述，显然所描述的实施例只是本发明的部分实施例，而不是全部的实施例。根据本发明的实施例，本领域的普通技术人员在不经创造性劳动的基础上实现的本发明的所有其他实施例，都属于本发明的保护范围。下面的描述中，对与本发明无关的技术只做简要的技术说明或者直接略过。 In order to make the description of the advantages of the present invention clearer, the specific implementation manners of the present invention will be further elaborated below in conjunction with the accompanying drawings. Apparently, the described embodiments are only some of the embodiments of the present invention, not all of them. According to the embodiments of the present invention, all other embodiments of the present invention realized by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. In the following description, the technologies irrelevant to the present invention are only briefly explained or directly skipped. the

本发明的主要思想是在家庭基站密集部署的OFDMA双层网络场景中，针对之前的基于非合作博弈联合进行功率控制和子信道分配所用穷举算法的复杂性所带来的时间损耗与能量消耗的状况，提出一种利用Q_学习算法分步进行子信道分配与功率分配的分布式算法。 The main idea of the present invention is to address the time loss and energy consumption caused by the complexity of the exhaustive algorithm used in the previous non-cooperative game-based joint power control and sub-channel allocation in the OFDMA double-layer network scenario where home base stations are densely deployed. situation, a distributed algorithm using Q_learning algorithm for step-by-step sub-channel allocation and power allocation is proposed. the

图1所示为本发明中具体OFDMA家庭基站网络的具体场景图； Fig. 1 shows the specific scenario diagram of the specific OFDMA home base station network in the present invention;

图2所示为本发明中家庭基站密集部署的双层网络考虑时延QoS指标进行子信道分配流程图。具体包括： FIG. 2 is a flow chart of sub-channel allocation in consideration of delay QoS index in a two-tier network with dense deployment of femtocells in the present invention. Specifically include:

步骤201：确定子信道的数量，根据经验初始化家庭用户和宏用户的发射功率。 Step 201: Determine the number of sub-channels, and initialize the transmission power of home users and macro users according to experience. the

步骤202：T＝0时刻初始化宏用户和家庭用户的Q_值。 Step 202: Initialize the Q_values of macro users and home users at time T=0. the

步骤203：T+1时刻到来时，宏用户和家庭用户基于T时刻的Q_值利用玻尔兹曼分布更新策略。 Step 203: When time T+1 arrives, macro users and home users use the Boltzmann distribution update strategy based on the Q_value at time T. the

步骤204：宏用户(MU)根据其策略空间选择使Q_值最大的传输子信道，并将该策略空间广播给家庭用户(FU)。 Step 204: The macro user (MU) selects the transmission sub-channel with the largest Q_ value according to its policy space, and broadcasts the policy space to the home user (FU). the

步骤205：家庭用户(FU)根据宏用户广播的策略和FU更新的策略选择最优传输子信道，并将其策略信息发送给MU。 Step 205: The home user (FU) selects the optimal transmission sub-channel according to the strategy broadcast by the macro user and the strategy updated by the FU, and sends its strategy information to the MU. the

步骤206：MU更新其立即回报，FU更新立即回报。 Step 206: MU updates its immediate report, and FU updates its immediate report. the

步骤207：宏用户和家庭用户通过更新Q_值。 Step 207: The Q_value is updated by the macro user and the home user. the

步骤208：判断子信道分配是否达到均衡，即Q_值是否收敛、稳定，或者达到最大迭代次数。如果上述条件都未满足，则返回步骤203，继续进行子信道分配；若满足上述的任意一个条件，则退出迭代，进入步骤209。 Step 208: Determine whether the sub-channel allocation has reached equilibrium, that is, whether the Q_value has converged and stabilized, or reached the maximum number of iterations. If none of the above conditions are met, return to step 203 and continue subchannel allocation; if any one of the above conditions is met, exit the iteration and enter step 209 . the

步骤209：取Q值收敛时所分配的子信道，并以此信道作为发送信息的信道。 Step 209: Take the sub-channel allocated when the Q value converges, and use this channel as the channel for sending information. the

图3所示为本发明中家庭基站密集部署的双层网络考虑时延QoS指标进行功率控制的流程图。具体包括： Fig. 3 is a flow chart showing power control in consideration of delay QoS index in a two-tier network with dense deployment of femtocells in the present invention. Specifically include:

步骤301：根据经验离散化家庭用户和宏用户的发射功率。 Step 301: Discretize transmit powers of home users and macro users based on experience. the

步骤302：T＝0时刻初始化宏用户和家庭用户的Q_值。 Step 302: Initialize the Q_values of macro users and home users at time T=0. the

步骤303：T+1时刻到来时，宏用户和家庭用户基于T时刻的Q_值利用玻尔兹曼分布更新策略。 Step 303: When time T+1 arrives, macro users and home users use the Boltzmann distribution update strategy based on the Q_value at time T. the

步骤304：宏用户(MU)根据其策略空间选择使Q_值最大的发射功率，并将该策略空间广播给家庭用户(FU)。 Step 304: The macro user (MU) selects the transmit power that maximizes the Q_value according to its policy space, and broadcasts the policy space to the home user (FU). the

步骤305：家庭用户(FU)根据宏用户广播的策略和FU更新的策略选择最优功率(Q_值最大)，并将其策略信息发送给MU。 Step 305: The home user (FU) selects the optimal power (with the largest Q_value) according to the policy broadcast by the macro user and the policy updated by the FU, and sends its policy information to the MU. the

步骤306：MU更新其立即回报，FU更新立即回报。 Step 306: MU updates its immediate report, and FU updates its immediate report. the

步骤307：宏用户和家庭用户更新Q_值。 Step 307: The macro user and the home user update the Q_value. the

步骤308：判断功率分配是否达到均衡，即Q_值是否收敛、稳定，或者是否达到最大迭代次数。如果上述条件都未满足，则返回步骤303，继续进行功率分配；若满足上述的任意一个条件，则退出迭代，进入步骤309。 Step 308: Determine whether the power distribution is balanced, that is, whether the Q_value is converged and stable, or whether the maximum number of iterations is reached. If none of the above conditions is met, return to step 303 and continue power allocation; if any one of the above conditions is met, exit the iteration and enter step 309 . the

步骤309：取Q值收敛时所选择的发射功率，并以此功率发送信息。 Step 309: Get the transmit power selected when the Q value converges, and send information at this power. the