CN114143212B - A Social Learning Approach for Smart Cities - Google Patents

Abstract

The invention discloses a social learning method for a smart city, which comprises the following steps: constructing a layered social learning system; establishing a task evaluation model based on deep reinforcement learning, and optimizing the task evaluation model by using the task states and channel states of all Internet of things equipment to obtain a basic decision; the edge server utilizes the task evaluation model received by the federal learning edge aggregation and optimizes the task evaluation model on the edge server according to the basic decision to obtain a high-level decision; the edge server guides a model in the Internet of things equipment by using transfer learning; the cloud server utilizes the task evaluation model received by the federal learning cloud aggregation to make a city-level decision according to the high-level decision and the task evaluation model on the cloud server, and utilizes the migration learning to guide the task evaluation model on the edge server. According to the method, the cooperation among the intelligent agents in the layer is improved by using federal learning, and the guidance of the upper layer to the lower layer is realized by using transfer learning among the layers, so that the performance of the model is improved.

Description

Translated fromChinese

一种面向智慧城市的社会化学习方法A Social Learning Approach for Smart Cities

技术领域technical field

本发明属于大数据技术领域，具体涉及一种面向智慧城市的社会化学习方法。The invention belongs to the technical field of big data, and in particular relates to a socialized learning method oriented to a smart city.

背景技术Background technique

近期城市化进程中出现的人口爆炸、资源失衡、交通拥堵等恶化趋势，对市民的高品质生活提出了日益高涨的要求。随着5G、物联网(InternetofThings,IoT)和人工智能(Artificial Intelligence,AI)的空前繁荣，智慧城市成为城市发展的新趋势指日可待。随着智慧城市的普及，2021年来自物联网设备的数据量将急剧增加，将达到850Zettabytes。数十亿的物联网设备与智慧城市相关联，为整个城市构建各种智能小区域。这些物联网设备通常部署有中等计算能力，例如智能路灯、智能交通灯、智能监控摄像头和智能手机。此外，通信功能弥合了物联网设备、用户乃至整个城市之间的鸿沟，为智慧城市提供了血管。一些资源因素，例如未充分利用的频谱资源、巨大的带宽成本和有限的计算能力，推翻了智慧城市的好处，即智慧城市中的细胞和血管逐渐贫乏和拥挤。The recent deteriorating trends of population explosion, resource imbalance, and traffic congestion in the process of urbanization have put forward increasingly high requirements for citizens' high-quality life. With the unprecedented prosperity of 5G, Internet of Things (IoT) and Artificial Intelligence (AI), smart cities will become a new trend in urban development just around the corner. With the proliferation of smart cities, the amount of data from IoT devices will increase dramatically in 2021, reaching 850 Zettabytes. Billions of IoT devices are associated with smart cities, building various smart pods for entire cities. These IoT devices are typically deployed with moderate computing power, such as smart street lights, smart traffic lights, smart surveillance cameras, and smartphones. In addition, communication capabilities bridge the gap between IoT devices, users, and even entire cities, providing blood vessels for smart cities. Several resource factors, such as underutilized spectrum resources, huge bandwidth costs, and limited computing power, overturn the benefits of smart cities, where cells and blood vessels are gradually depleted and congested.

为了释放智慧城市的潜力，有许多研究趋势可以解决上述挑战。例如，认知物联网使物联网设备能够灵活感知和动态接入频谱，从而缓解智慧城市的频谱需求。边缘计算和雾计算将计算任务和服务从云服务器推送到网络边缘，进一步降低带宽消耗。但仍有许多问题有待解决。1)由于智慧城市对延迟的严格要求，对于频谱接入和计算分配的决策需要提前做出，并具有较高的精度，这些要求催生了大量关于人工智能受益策略的研究；2)构建智能决策的传统人工智能方法，通常依赖于提供海量数据，并在一个或几个云服务器上进行训练，这些问题进一步加剧了带宽成本、时间效率等问题；3)边缘智能包括将学习智能从一个或几个云服务器推送到网络边缘，但它们忽略了边缘服务器之间的协作特性，导致学习资源低效甚至学习性能下降；4)智慧城市运营的背后，存在着明显的社会等级制度，该层次结构由物联网设备、决定物联网设备运行的边缘服务器和决定边缘服务器操作的云服务器组成。现有的边缘智能也忽略了智慧城市的社会化决策。To unlock the potential of smart cities, there are a number of research trends that address the above challenges. For example, cognitive IoT enables IoT devices to flexibly sense and dynamically access spectrum, thereby alleviating spectrum requirements for smart cities. Edge computing and fog computing push computing tasks and services from cloud servers to the network edge, further reducing bandwidth consumption. But there are still many issues to be resolved. 1) Due to the strict requirements for delay in smart cities, decisions on spectrum access and computing allocation need to be made in advance and with high precision, these requirements have spawned a lot of research on AI benefit strategies; 2) Building intelligent decision-making The traditional artificial intelligence methods usually rely on providing massive data and training on one or several cloud servers, which further exacerbate the problems of bandwidth cost, time efficiency, etc.; 3) Edge intelligence includes transferring learning intelligence from one or several cloud servers Each cloud server is pushed to the edge of the network, but they ignore the collaborative characteristics between edge servers, resulting in inefficient learning resources and even decreased learning performance; 4) Behind the operation of smart cities, there is an obvious social hierarchy, which consists of It consists of IoT devices, edge servers that determine the operation of the IoT devices, and cloud servers that determine the operations of the edge servers. Existing edge intelligence also ignores social decision-making in smart cities.

发明内容SUMMARY OF THE INVENTION

针对以上技术问题，受人类社会高效协作的运行机制和人与人之间的社会互动的启发，本发明提出了一种面向智慧城市的社会化学习方法，它是一种社会化的、基于学习的和认知的方案，它根据智能体的特征构建等级，做出相互关联的决策，使信息交互流动，可以解决智慧城市的各种需求，例如合理的资源配置。为解决以上技术问题，本发明所采用的技术方案如下：In view of the above technical problems, inspired by the operation mechanism of efficient cooperation in human society and the social interaction between people, the present invention proposes a socialized learning method for smart cities, which is a socialized learning-based learning method. An intelligent and cognitive solution, which builds a hierarchy based on the characteristics of the agent, makes interrelated decisions, and enables the interactive flow of information, which can solve various needs of a smart city, such as reasonable resource allocation. For solving the above technical problems, the technical scheme adopted in the present invention is as follows:

一种面向智慧城市的社会化学习方法，包括如下步骤：A social learning method for smart cities, including the following steps:

S1，构建包括云服务器、边缘服务器和物联网设备的分层社会化学习系统，物联网设备通过无线网络与边缘服务器连接，边缘服务器与云服务器连接；S1, build a hierarchical social learning system including cloud servers, edge servers and IoT devices, where IoT devices are connected to edge servers through wireless networks, and edge servers are connected to cloud servers;

S2，基于深度强化学习在物联网设备、边缘服务器和云服务器中分别建立任务评估模型；S2, establish task evaluation models in IoT devices, edge servers and cloud servers based on deep reinforcement learning;

S3，物联网设备利用所有物联网设备获得的任务状态和信道状态对任务评估模型进行优化，根据优化后的任务评估模型获得任务处理基本决策，并将任务处理基本决策和优化后的任务评估模型发送给对应的边缘服务器；S3, the IoT device optimizes the task evaluation model by using the task state and channel state obtained by all the IoT devices, obtains the basic task processing decision according to the optimized task evaluation model, and combines the basic task processing decision and the optimized task evaluation model. Send to the corresponding edge server;

S4，边缘服务器利用联邦学习对物联网设备发送的任务评估模型进行边缘聚合，根据物联网设备发送的任务处理基本决策对边缘服务器上的任务评估模型进行优化获得任务处理高层决策，并将任务处理高层决策发送给物联网设备；S4, the edge server uses federated learning to perform edge aggregation on the task evaluation model sent by the IoT device, and optimizes the task evaluation model on the edge server according to the basic task processing decision sent by the IoT device to obtain a high-level task processing decision. High-level decisions are sent to IoT devices;

S5，边缘服务器利用迁移学习对物联网设备中的任务评估模型进行指导，并将任务处理高层决策和优化后的任务评估模型发送给云服务器；S5, the edge server uses migration learning to guide the task evaluation model in the IoT device, and sends the task processing high-level decision and the optimized task evaluation model to the cloud server;

S6，云服务器利用联邦学习将边缘服务器发送的任务评估模型进行云聚合，根据边缘服务器发送的任务处理高层决策和云服务器上的任务评估模型制定任务处理市级决策，将任务处理市级决策发送给边缘服务器，并利用迁移学习对边缘服务器中的任务评估模型进行指导。S6, the cloud server uses federated learning to aggregate the task evaluation model sent by the edge server to the cloud, and according to the task processing high-level decision sent by the edge server and the task evaluation model on the cloud server, the task processing city-level decision is made, and the task processing city-level decision is sent. To the edge server, and use transfer learning to guide the task evaluation model in the edge server.

在步骤S3中，所述任务状态包括任务的CPU周期和任务数据量，信道状态包括无线网络的授权信道增益、非授权信道增益和信道占用状态。In step S3, the task state includes the CPU cycle of the task and the amount of task data, and the channel state includes the authorized channel gain, the unlicensed channel gain and the channel occupancy state of the wireless network.

所述物联网设备包括主要用户和次要用户，主要用户通过授权信道与边缘服务器连接，次要用户通过授权信道或非授权信道与边缘服务器连接，当次要用户使用授权信道时，需不影响主要用户的连接。The IoT device includes a primary user and a secondary user. The primary user is connected to the edge server through an authorized channel, and the secondary user is connected to the edge server through an authorized channel or an unauthorized channel. When the secondary user uses the authorized channel, it is necessary to Primary user connection.

所述步骤S3包括如下步骤：The step S3 includes the following steps:

S3.1，每个物联网设备获取当前环境下的任务状态和信道状态，并根据对应的任务评估模型得到任务处理初步决策；S3.1, each IoT device obtains the task state and channel state in the current environment, and obtains a preliminary decision on task processing according to the corresponding task evaluation model;

S3.2，每个物联网设备综合所有物联网设备的任务处理初步决策，并以最小化总处理延迟和能耗的加权和为目标对任务评估模型进行一次优化；S3.2, each IoT device synthesizes the preliminary decision of task processing of all IoT devices, and optimizes the task evaluation model once with the goal of minimizing the weighted sum of total processing delay and energy consumption;

S3.3，物联网设备根据一次优化后的任务评估模型制定任务处理基本决策，并将任务处理基本决策和一次优化后的任务评估模型发送给对应的边缘服务器。S3.3, the IoT device makes basic task processing decisions according to an optimized task evaluation model, and sends the basic task processing decisions and an optimized task evaluation model to the corresponding edge server.

所述任务处理基本决策和任务处理初步决策均包括卸载决策、无线网络的信道选择和计算资源预算。Both the basic task processing decision and the preliminary task processing decision include offloading decision, channel selection of wireless network and computing resource budget.

所述步骤S4包括如下步骤：The step S4 includes the following steps:

S4.1，每个边缘服务器获取当前环境下的信道状态和边缘可用计算资源，利用联邦学习将接收到的一次优化的任务评估模型进行边缘聚合，并将边缘聚合后的任务评估模型发送对应的物联网设备；S4.1, each edge server obtains the channel state and available computing resources of the edge in the current environment, uses federated learning to perform edge aggregation on the received one-time optimized task evaluation model, and sends the edge-aggregated task evaluation model to the corresponding IoT devices;

S4.2，每个边缘服务器根据接收到的任务处理基本决策、步骤S4.1获取的信道状态和边缘可用计算资源以及边缘服务器上的任务评估模型得到任务处理高层初步决策；S4.2, each edge server obtains a high-level preliminary decision of task processing according to the basic decision of task processing received, the channel state and available computing resources of the edge obtained in step S4.1, and the task evaluation model on the edge server;

S4.3，每个边缘服务器综合所有边缘服务器的任务处理高层初步决策，并以成本最小化为目标对边缘服务器上的任务评估模型进行一次优化，根据一次优化后的任务评估模型得出任务处理高层决策；S4.3, each edge server synthesizes the high-level preliminary decision of the task processing of all edge servers, and optimizes the task evaluation model on the edge server with the goal of cost minimization, and obtains the task processing according to the optimized task evaluation model. high-level decision-making;

S4.4，边缘服务器将任务处理高层决策发送给对应的物联网设备，物联网设备利用任务处理高层决策对物联网设备上的任务评估模型进行二次优化。S4.4, the edge server sends the task processing high-level decision to the corresponding IoT device, and the IoT device uses the task processing high-level decision to perform secondary optimization on the task evaluation model on the IoT device.

所述任务处理高层决策和任务处理高层初步决策均包括基本决策的评估结果、协作边缘服务器选择和边缘计算资源贡献量。The task processing high-level decision and the task processing high-level preliminary decision both include the evaluation result of the basic decision, the cooperative edge server selection, and the contribution of edge computing resources.

所述成本等于边缘计算资源贡献量减去与其它边缘服务器合作的成本之间的差值。The cost is equal to the difference between the contribution of the edge computing resources minus the cost of cooperating with other edge servers.

所述步骤S6包括如下步骤：The step S6 includes the following steps:

S6.1，云服务器获取当前环境下的信道状态和云可用计算资源，利用联邦学习将接收到的边缘服务器发送的任务评估模型进行云聚合，并将云聚合后的任务评估模型发送边缘服务器；S6.1, the cloud server obtains the channel state and available computing resources of the cloud in the current environment, uses federated learning to perform cloud aggregation on the task evaluation model sent by the received edge server, and sends the cloud-aggregated task evaluation model to the edge server;

S6.2，云服务器将接收到的任务处理高层决策、步骤S5.1得到的信道状态和云可用计算资源输入云服务器上的任务评估模型得到任务处理市级决策，将任务处理市级决策发送给对应的边缘服务器，并利用迁移学习对边缘服务器中的任务评估模型进行指导；S6.2, the cloud server inputs the received task processing high-level decision, the channel state and cloud available computing resources obtained in step S5.1 into the task evaluation model on the cloud server to obtain the task processing city-level decision, and sends the task processing city-level decision. Give the corresponding edge server, and use transfer learning to guide the task evaluation model in the edge server;

S6.3，边缘服务器根据任务处理市级决策对边缘服务器上的任务评估模型进行二次优化。S6.3, the edge server performs secondary optimization on the task evaluation model on the edge server according to the task processing city-level decision.

所述任务处理市级决策包括高层决策的评估结果和云计算资源贡献量。The task processing city-level decision includes the evaluation result of the high-level decision and the contribution of cloud computing resources.

本发明的有益效果：Beneficial effects of the present invention:

本发明根据各智能体的计算能力不同构建了分层社会化学习系统，层内各智能体通过联邦学习提高层内智能体之间的协作能力，层间通过迁移学习实现了上层对下层的引导，提高了模型的性能；各智能体之间进行合作，单一智能体在进行决策的时候同时考量其它智能体的决策，在对任务卸载、频谱选择和计算资源分配进行决策时，可以共同优化传输时延、能量消耗和带宽利用率。The invention constructs a layered social learning system according to the different computing capabilities of each agent. Each agent in the layer improves the cooperation ability between the agents in the layer through federated learning, and realizes the guidance of the upper layer to the lower layer through migration learning between layers. , which improves the performance of the model; each agent cooperates, and a single agent considers the decisions of other agents when making decisions, and can jointly optimize transmission when making decisions on task offloading, spectrum selection, and computing resource allocation Latency, energy consumption and bandwidth utilization.

附图说明Description of drawings

为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

图1为本发明的基本架构图。FIG. 1 is a basic structure diagram of the present invention.

图2为云服务器、边缘服务器和物联网设备各层之间的关系。Figure 2 shows the relationship between the layers of cloud servers, edge servers, and IoT devices.

图3为本发明的流程图。Figure 3 is a flow chart of the present invention.

图4为本发明与随机卸载方案和本地卸载方案之间在平均SINR、平均时延和平均能耗之间的对比效果图。FIG. 4 is a comparison effect diagram between the present invention and the random offloading scheme and the local offloading scheme among the average SINR, the average delay and the average energy consumption.

具体实施方式Detailed ways

下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有付出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

随着机器智能的快速发展，可以认为许多机器就像人类一样构成了一个机器社会。对于人类社会，社会学习理论指出了社会特征对人类行为的限制作用。人类社会的运行机制对个人有激励作用，并承诺了关于个人之间的社会化特征的若干好处。社会中的人按照一定的标准被划分为不同的层次，社会中的社会关系使人类社会更加协调和高效。例如，在决策方面，高层次的人不需要从低层次的人那里收集所有数据。高层人士更关注来自低层人士的学习决策，协助他们做出更高层次的决策。也就是说，在现实社会中，总统不需要知道一个州的州长是如何决策的，他只需要学习决策结果。受人类社会高效协作运行机制的启发，本发明将其引入机器社会，提出社会化学习来解决智慧城市中的各种需求，如资源的合理分配。与边缘智能的其他使能技术不同，如联邦学习，该方案更具有人类社会的特征，如社会关系：合作和竞争。竞争与合作作为社会关系的代表，都是社会规则的组成部分。通过竞争可以提高个体的互动性、积极性和学习效率。此外，合作可以使私人知识公开化，以提高所有个人的能力和效用。With the rapid development of machine intelligence, it can be considered that many machines, just like humans, constitute a machine society. For human society, social learning theory points out the limiting effect of social characteristics on human behavior. The working mechanisms of human society are motivating to individuals and promise several benefits regarding the characteristics of socialization among individuals. People in society are divided into different levels according to certain standards, and social relations in society make human society more coordinated and efficient. For example, when it comes to decision-making, high-level people don't need to collect all the data from low-level people. Higher-level people pay more attention to learning decisions from lower-level people, helping them make higher-level decisions. That is to say, in the real society, the president does not need to know how the governor of a state makes decisions, he only needs to learn the results of the decisions. Inspired by the efficient cooperative operation mechanism of human society, the present invention introduces it into the machine society, and proposes socialized learning to solve various needs in smart cities, such as rational allocation of resources. Unlike other enabling technologies for edge intelligence, such as federated learning, this scheme is more characteristic of human societies, such as social relations: cooperation and competition. Competition and cooperation, as representatives of social relations, are both components of social rules. Through competition, individual interaction, enthusiasm and learning efficiency can be improved. In addition, collaboration can make private knowledge public to enhance the capabilities and utility of all individuals.

一种面向智慧城市的社会化学习方法，如图1-4所示，包括如下步骤：A social learning method for smart cities, as shown in Figure 1-4, includes the following steps:

S1，构建包括云服务器、边缘服务器和物联网设备的分层社会化学习系统，物联网设备通过无线网络与边缘服务器连接，边缘服务器与云服务器连接；S1, build a hierarchical social learning system including cloud servers, edge servers and IoT devices, where IoT devices are connected to edge servers through wireless networks, and edge servers are connected to cloud servers;

所述云服务器、边缘服务器和物联网设备均为智能体，由于智能体的计算能力、感知范围、通信方式等资源因素不同，因此将具有不同资源的云服务器、边缘服务器和物联网设备分别设置在分层社会化学习系统中的不同层，高层的云服务器具有强大的数据分析和计算能力，支持无所不在的、方便的、按需的网络访问，但长距离造成了很高的通信开销。位于底层的大规模物联网设备上设有多个时间敏感和计算密集型的任务需要处理，但它具有有限的感知能力，且受计算能力的限制。中间层是边缘服务器层，它比云服务器更接近底层物联网设备，可以提供中等的计算能力；通过连接物联网设备的无线网络、连接云服务器的骨干网络，边缘服务器可以通过协调和整合各种类型的资源，缓解最高和最低层之间的壁垒。The cloud server, edge server and IoT device are all intelligent bodies. Due to different resource factors such as the computing power, perception range, and communication method of the intelligent body, the cloud server, edge server and IoT device with different resources are set up respectively. At different layers in the hierarchical social learning system, high-level cloud servers have powerful data analysis and computing capabilities, and support ubiquitous, convenient, and on-demand network access, but long distances cause high communication overhead. The underlying large-scale IoT device has multiple time-sensitive and computationally intensive tasks to be processed, but it has limited perception capability and is limited by computational power. The middle layer is the edge server layer, which is closer to the underlying IoT devices than cloud servers and can provide moderate computing power; through the wireless network connecting IoT devices and the backbone network connecting cloud servers, edge servers can coordinate and integrate various types of resources that ease the barriers between the highest and lowest tiers.

所述无线网络的信道包括授权信道和非授权信道，物联网设备包括主要用户和次要用户，主要用户主要使用授权信道传输信号，次要用户主要使用非授权信道传输信号，当次要用户想要使用主要用户的授权信道卸载时，需要保证不影响主要用户的无线传输。The channels of the wireless network include authorized channels and unlicensed channels, and the IoT devices include primary users and secondary users. The primary user mainly uses the authorized channel to transmit signals, and the secondary user mainly uses the unlicensed channel to transmit signals. When offloading using the authorized channel of the primary user, it is necessary to ensure that the wireless transmission of the primary user is not affected.

S2，基于深度强化学习(Deep Reinforcement Learning，DRL)分别在物联网设备、边缘服务器和云服务器中分别建立任务评估模型；S2, based on Deep Reinforcement Learning (DRL), respectively establish task evaluation models in IoT devices, edge servers and cloud servers;

S3，物联网设备利用所有物联网设备获得的任务状态和信道状态对物联网设备上的任务评估模型进行优化，根据优化后的任务评估模型获得任务处理基本决策，并将任务处理基本决策和优化后的任务评估模型发送给对应的边缘服务器，包括如下步骤：S3, the IoT device optimizes the task evaluation model on the IoT device by using the task state and channel state obtained by all the IoT devices, obtains the basic task processing decision according to the optimized task evaluation model, and combines the basic task processing decision and optimization. The post-task evaluation model is sent to the corresponding edge server, including the following steps:

S3.1，每个物联网设备获取当前环境下的任务状态和信道状态，并根据物联网设备上的任务评估模型得到任务处理初步决策；S3.1, each IoT device obtains the task state and channel state in the current environment, and obtains a preliminary decision on task processing according to the task evaluation model on the IoT device;

所述任务状态包括任务的CPU周期和任务数据量，信道状态包括授权信道增益、非授权信道增益和信道占用状态，；The task state includes the CPU cycle of the task and the amount of task data, and the channel state includes the authorized channel gain, the unlicensed channel gain and the channel occupancy state,;

S3.2，每个物联网设备综合所有物联网设备的任务处理初步决策，并以最小化总处理延迟和能耗的加权和为目标对物联网设备上的任务评估模型进行一次优化；S3.2, each IoT device synthesizes the preliminary decision of task processing of all IoT devices, and optimizes the task evaluation model on the IoT device once with the goal of minimizing the weighted sum of total processing delay and energy consumption;

单个物联网设备在制定决策时同时考虑其它物联网设备的决策，可以实现智能体之间的合作。A single IoT device considers the decisions of other IoT devices when making decisions, enabling cooperation between agents.

S3.3，物联网设备根据一次优化后的任务评估模型制定任务处理基本决策，并将任务处理基本决策和一次优化后的任务评估模型发送给所连接的边缘服务器；S3.3, the IoT device makes basic task processing decisions according to an optimized task evaluation model, and sends the basic task processing decisions and an optimized task evaluation model to the connected edge server;

所述任务处理基本决策和任务处理初步决策均包括卸载决策、信道选择、计算资源预算，卸载决策是指将任务执行本地处理或是远程卸载，计算资源预算是指对执行任务所需要的计算资源的评估。Both the basic task processing decision and the preliminary task processing decision include offloading decision, channel selection, and computing resource budget. The offloading decision refers to performing local processing or remote offloading of the task, and the computing resource budget refers to the computing resources required to perform the task. evaluation of.

S4，边缘服务器利用联邦学习对物联网设备发送的任务评估模型进行边缘聚合，根据物联网设备的基本决策对边缘服务器上的任务评估模型进行优化，根据优化后的任务评估模型获得高层决策，并将高层决策和优化后的任务评估模型发送给云服务器，包括如下步骤：S4, the edge server uses federated learning to perform edge aggregation on the task evaluation model sent by the IoT device, optimizes the task evaluation model on the edge server according to the basic decision of the IoT device, and obtains high-level decisions according to the optimized task evaluation model. Send the high-level decision and the optimized task evaluation model to the cloud server, including the following steps:

S4.1，每个边缘服务器获取当前环境下的信道状态和边缘可用计算资源，利用联邦学习将所连接的物联网设备发送的任务评估模型进行边缘聚合，并将边缘聚合后的任务评估模型发送到对应的物联网设备；S4.1, each edge server obtains the channel state and available computing resources on the edge in the current environment, uses federated learning to perform edge aggregation on the task evaluation model sent by the connected IoT devices, and sends the edge-aggregated task evaluation model to the edge to the corresponding IoT device;

由于边缘服务器具有较强的计算能力，因此边缘服务器可以获取更精确的信道状态。利用联邦学习可以在保证数据交换过程中的信息安全、数据隐私和合法性的前提下，使同一层的设备之间进行高效的机器学习。Because the edge server has strong computing power, the edge server can obtain more accurate channel state. Using federated learning can enable efficient machine learning between devices at the same layer on the premise of ensuring information security, data privacy, and legality in the process of data exchange.

S4.2，每个边缘服务器根据接收到任务处理基本决策、步骤S4.1所获取的信道状态、边缘可用计算资源和边缘服务器上的任务评估模型得到任务处理高层初步决策；S4.2, each edge server obtains a high-level preliminary decision on task processing according to the basic decision of task processing received, the channel state obtained in step S4.1, the available computing resources on the edge, and the task evaluation model on the edge server;

S4.3，每个边缘服务器综合所有边缘服务器的任务处理高层初步决策，并以成本最小化为目标对边缘服务器上的任务评估模型进行一次优化，根据一次优化后的任务评估模型得出任务处理高层决策；S4.3, each edge server synthesizes the high-level preliminary decision of the task processing of all edge servers, and optimizes the task evaluation model on the edge server with the goal of cost minimization, and obtains the task processing according to the optimized task evaluation model. high-level decision-making;

所述成本的计算方法为将边缘计算资源贡献量减去与其它边缘服务器合作的成本。The cost is calculated by subtracting the cost of cooperating with other edge servers from the contribution of edge computing resources.

S4.4，边缘服务器将任务处理高层决策发送给对应的物联网设备，物联网设备利用任务处理高层决策对物联网设备上的任务评估模型进行二次优化；S4.4, the edge server sends the task processing high-level decision to the corresponding IoT device, and the IoT device uses the task processing high-level decision to perform secondary optimization on the task evaluation model on the IoT device;

所述任务处理高层决策和任务处理高层初步决策均包括基本决策的评估结果、协作边缘服务器选择和边缘计算资源贡献量，基本决策的评估结果是指是否批准物联网设备执行卸载决策，协作边缘服务器选择是指远程卸载任务时需要协作的边缘服务器集合也即将任务卸载到哪些边缘服务器进行处理，边缘计算资源贡献量是指远程卸载任务时每个边缘服务器根据各自的可用计算资源所分配的计算资源。由于每个边缘服务器的计算资源是有限的，因此通过边缘服务器之间的合作可以实现计算资源的合理分配。The task processing high-level decision and the task processing high-level preliminary decision both include the evaluation result of the basic decision, the selection of cooperative edge servers and the contribution of edge computing resources. Selection refers to the set of edge servers that need to cooperate when offloading tasks remotely, and which edge servers to offload tasks to for processing. The contribution of edge computing resources refers to the computing resources allocated by each edge server according to its own available computing resources when offloading tasks remotely. . Since the computing resources of each edge server are limited, the reasonable allocation of computing resources can be achieved through cooperation between edge servers.

S5，边缘服务器利用迁移学习对物联网设备中的任务评估模型进行指导，并将任务处理高层决策和边缘服务器的一次优化后的任务评估模型发送给云服务器；S5, the edge server uses migration learning to guide the task evaluation model in the IoT device, and sends the task processing high-level decision-making and an optimized task evaluation model of the edge server to the cloud server;

迁移学习作为跨层的学习方法，可以实现数据的垂直流动，加快训练过程，优化底层模型训练的学习效率，提高资源利用率。也可以不像大多数训练过程那样需要从头学习，大大节省了计算资源，时间效率高。本实施例中的迁移学习是受“教师-学生”结构的启发，“教师-学生”结构是社会关系中的一种特殊的合作，即师生合作。教师将所学知识分享给学生，实现知识的流动，提高学生的能力，显著减少学生学习相关知识所消耗的时间和资源。As a cross-layer learning method, transfer learning can realize the vertical flow of data, speed up the training process, optimize the learning efficiency of the underlying model training, and improve resource utilization. It can also not need to learn from scratch like most training processes, which greatly saves computing resources and is time-efficient. The transfer learning in this embodiment is inspired by the "teacher-student" structure, which is a special kind of cooperation in social relations, that is, teacher-student cooperation. Teachers share the knowledge they have learned with students, realize the flow of knowledge, improve students' abilities, and significantly reduce the time and resources consumed by students in learning related knowledge.

S6，云服务器利用联邦学习将边缘服务器发送的任务评估模型进行云聚合，根据边缘服务器发送的任务处理高层决策和云服务器上的任务评估模型制定任务处理市级决策，将任务处理市级决策发送给边缘服务器，并利用迁移学习对边缘服务器中的任务评估模型进行指导，实现云服务器对边缘服务器的引导，包括如下步骤：S6, the cloud server uses federated learning to aggregate the task evaluation model sent by the edge server to the cloud, and according to the task processing high-level decision sent by the edge server and the task evaluation model on the cloud server, the task processing city-level decision is made, and the task processing city-level decision is sent. Give the edge server, and use transfer learning to guide the task evaluation model in the edge server, and realize the guidance of the cloud server to the edge server, including the following steps:

S6.1，云服务器获取当前环境下的信道状态和云可用计算资源，利用联邦学习将接收到的边缘服务器发送的任务评估模型进行云聚合，并将云聚合后的任务评估模型发送边缘服务器；S6.1, the cloud server obtains the channel state and available computing resources of the cloud in the current environment, uses federated learning to perform cloud aggregation on the task evaluation model sent by the received edge server, and sends the cloud-aggregated task evaluation model to the edge server;

由于云服务器具有最强大的计算能力，因此云服务器可以获取更为精确的信道状态。Since the cloud server has the most powerful computing power, the cloud server can obtain a more accurate channel state.

S6.2，云服务器将接收到的任务处理高层决策、步骤S6.1得到的信道状态、云可用计算资源输入云服务器上的任务评估模型得到任务处理市级决策，将任务处理市级决策发送给对应的边缘服务器，并利用迁移学习对边缘服务器中的任务评估模型进行指导；S6.2, the cloud server inputs the received task processing high-level decision, the channel state obtained in step S6.1, and the cloud available computing resources into the task evaluation model on the cloud server to obtain the task processing city-level decision, and sends the task processing city-level decision. Give the corresponding edge server, and use transfer learning to guide the task evaluation model in the edge server;

所述任务处理市级决策包括高层决策的评估结果和云计算资源贡献量。高层决策的评估结果是指是否批准边缘服务器执行卸载决策，云计算资源贡献量是指通过云服务器执行远程卸载任务时需要云服务器贡献的计算资源。通过上层智能体的引导，可以将更多的信息传递到下层智能体，增加了数据储备，提高了数据的质量。The task processing city-level decision includes the evaluation result of the high-level decision and the contribution of cloud computing resources. The evaluation result of the high-level decision refers to whether to approve the edge server to perform the offloading decision, and the cloud computing resource contribution refers to the computing resources contributed by the cloud server when performing remote offloading tasks through the cloud server. Through the guidance of the upper-level agent, more information can be transmitted to the lower-level agent, which increases the data reserve and improves the quality of the data.

S6.3，边缘服务器根据云服务器所发送的任务处理市级决策对边缘服务器上的任务评估模型进行二次优化。S6.3, the edge server performs secondary optimization on the task evaluation model on the edge server according to the task processing city-level decision sent by the cloud server.

三层智能体之间社会化学习的流程为：物联网设备作为眼睛和耳朵，可以感知所在环境并对感知到的数据进行预处理，由于其感知范围的限制，物联网设备可以为其上的各种任务做出任务处理基本决策，并将基本决策上传给边缘服务器，边缘服务器对不同地理区域的物联网设备所传递的基本决策进行聚合处理，并根据自己所感知到的状态对物联网设备的基本决策进行评估制定出任务处理高层决策，同样的，边缘服务器做出任务处理高层决策后，将任务处理高层决策上传给云服务器，云服务器对接收到的智慧城市中的所有边缘服务器的任务处理高层决策进行聚合处理以制定出全面的任务处理市级决策，再通过迁移学习实现上层对下层的引导进而做出更为准确的决策，同时每个智能体都可以通过自主学习获得知识实现感知数据的共享，并通过学习同层或更高层的其它智能体的行为和相应的反馈采取动作也即决策。The process of social learning among the three-layer agents is as follows: IoT devices, acting as eyes and ears, can perceive the environment and preprocess the perceived data. Due to the limitation of its perception range, IoT devices can be used for Various tasks make basic decisions for task processing, and upload the basic decisions to the edge server. The edge server aggregates and processes the basic decisions transmitted by IoT devices in different geographical areas, and analyzes the IoT devices according to their perceived states. The basic decision-making is evaluated and the high-level decision of task processing is made. Similarly, after the edge server makes the high-level decision of task processing, it uploads the high-level decision of task processing to the cloud server, and the cloud server receives the tasks of all edge servers in the smart city. Process high-level decision-making and aggregate processing to formulate a comprehensive task processing city-level decision, and then use transfer learning to guide the upper layer to the lower layer to make more accurate decisions. At the same time, each agent can acquire knowledge through autonomous learning to realize perception Data is shared, and actions, that is, decisions, are taken by learning the behavior of other agents at the same level or higher and corresponding feedback.

本实施例中，所述边缘服务器只关注物联网设备做出的任务处理基本决策，而不论物联网设备是如何做出的这些任务处理基本决策，同样地，云服务器也只关注边缘服务器做出的任务处理高层决策，而不论如何做出的这些任务处理高层决策。所有模型在进行传输、优化或更新时是指通过操作模型参数实现的，解决了物联网设备由于其感知能力的限制，无法准确感知环境，或者感知到的数据中存在数据噪声或一定程度的不确定性，导致做出错误的卸载决策，甚至导致通信和计算资源消耗增加、时间效率降低的问题。由于错误的卸载决策可能会导致卸载机会的丧失及无法获得可用的频谱资源，因此增加了无线链路的拥塞、能耗及成本，降低了带宽利用率。所述联邦学习和迁移学习均为现有技术，本实施例不再赘述。In this embodiment, the edge server only pays attention to the basic task processing decisions made by the IoT device, regardless of how the IoT device makes these basic task processing decisions. Similarly, the cloud server also only pays attention to the basic decisions made by the edge server. The tasks deal with high-level decisions, regardless of how these tasks are made. All models are realized by operating model parameters when transmitting, optimizing or updating, which solves the problem that IoT devices cannot accurately perceive the environment due to the limitation of their perception capabilities, or there is data noise or a certain degree of inconsistency in the perceived data. Deterministic, leading to wrong offloading decisions, and even leading to increased consumption of communication and computing resources and decreased time efficiency. Since wrong offloading decisions may lead to loss of offloading opportunities and inability to obtain available spectrum resources, the congestion, energy consumption and cost of wireless links are increased, and bandwidth utilization is reduced. The federated learning and transfer learning are both existing technologies, and details are not repeated in this embodiment.

本实施例可以应用在智慧城市中的智能电网及智慧交通等领域，当应用在智能电网领域时，物联网设备相当于某一地理区域的监控电源设备如断路器、保险丝保护器、安培表、电压表等的摄像设备，边缘服务器相当于某一个地理区域的本地监控机，云服务器相当于有线连接本地监控机的总监控中心，由于摄像设备的计算能力有限无法处理全部的图像数据，可以利用本申请制定相关的图像处理决策以在摄像设备、本地监控机和总监控中心之间进行合理的资源分配。当应用在智慧交通领域时，物联网设备相当于某一地理区域的交通监控设备如固定或移动式摄像设备，边缘服务器相当于某一个地理区域的道路侧单元(Road Side Units,RSU)，云服务器相当于有线连接道路侧单元的交通监控中心。This embodiment can be applied to fields such as smart grids and smart transportation in smart cities. When applied to smart grids, IoT devices are equivalent to monitoring power supply devices in a certain geographical area, such as circuit breakers, fuse protectors, ammeters, For camera equipment such as voltmeters, the edge server is equivalent to a local monitoring machine in a certain geographical area, and the cloud server is equivalent to the general monitoring center connected to the local monitoring machine by wire. Due to the limited computing power of the camera equipment, it cannot process all the image data. The present application makes relevant image processing decisions to reasonably allocate resources among camera equipment, local monitoring machines and the general monitoring center. When applied in the field of smart transportation, IoT devices are equivalent to traffic monitoring equipment in a geographical area, such as fixed or mobile camera equipment, edge servers are equivalent to Road Side Units (RSU) in a geographical area, cloud The server is equivalent to the traffic monitoring center that is wired to the roadside unit.

如图4所示为本申请、随机卸载方案和本地卸载方案在干扰信噪比(Signal toInterference plus Noise Ratio，SINR)、时延和能耗方面的比较结果。本地卸载方案表示物联网设备仅在本地执行任务，而随机卸载方案则是随机决定是否卸载任务。图4(a)为主要用户(Primary Users，PUs)、次要用户(Secondary Users，SUs)与所有物联网设备也即主要用户和次要用户的加总的平均SINR对比结果，可以看出，本申请的平均SINR比传统方案高，平均增幅超过22.2％，本申请可以提高所有物联网设备的平均SINR，同时保证主要用户的传输速率不受太大影响，这直接证明了带宽利用率大大提高。这是因为本发明所提出的社会化学习方案可以确定执行任务的最优卸载和通信策略。FIG. 4 shows the comparison results of the present application, the random offloading scheme and the local offloading scheme in terms of Signal to Interference plus Noise Ratio (SINR), delay and energy consumption. The local offload scheme means that the IoT device only performs the task locally, while the random offload scheme randomly decides whether to offload the task or not. Figure 4(a) shows the comparison results of the average SINR of primary users (PUs), secondary users (SUs) and all IoT devices, that is, the total of primary users and secondary users. It can be seen that, The average SINR of this application is higher than that of the traditional solution, with an average increase of more than 22.2%. This application can improve the average SINR of all IoT devices, while ensuring that the transmission rate of major users is not greatly affected, which directly proves that the bandwidth utilization rate is greatly improved. . This is because the social learning scheme proposed by the present invention can determine the optimal offloading and communication strategies for performing tasks.

此外，为了进一步评估本申请的时间效率和能量效率，分别在图4(b)和图4(c)中给出了平均延迟和平均能量消耗，并与传统方案(包括随机卸载方案和本地卸载方案)相对比，从图中可以看出，本发明具有较低的平均延迟和平均能量消耗。在延迟方面，本申请的平均延迟比本地卸载方案低60.9％，比随机卸载方案低35.7％。在能耗方面，本申请的平均能耗比本地卸载方案低20％，比随机卸载方案低8.2％。这是由于物联网设备选择当前最优的卸载和通信策略来执行任务，平均延迟和能源消耗可以减少。在传统方案中，物联网设备随机选择策略或仅在本地执行任务，因此造成延迟和能耗高。In addition, to further evaluate the time efficiency and energy efficiency of the present application, the average delay and average energy consumption are presented in Fig. Compared with the scheme), it can be seen from the figure that the present invention has lower average delay and average energy consumption. In terms of latency, the average latency of this application is 60.9% lower than the local offloading scheme and 35.7% lower than the random offloading scheme. In terms of energy consumption, the average energy consumption of the present application is 20% lower than that of the local unloading scheme, and 8.2% lower than that of the random unloading scheme. This is due to the fact that IoT devices choose the currently optimal offloading and communication strategy to perform the task, the average latency and energy consumption can be reduced. In traditional schemes, IoT devices randomly choose policies or only perform tasks locally, thus causing high latency and energy consumption.

以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of the present invention. within the scope of protection.

Claims (9)

1. A social learning method for smart cities is characterized by comprising the following steps:

s1, constructing a hierarchical social learning system comprising a cloud server, an edge server and Internet of things equipment, wherein the Internet of things equipment is connected with the edge server through a wireless network, and the edge server is connected with the cloud server;

s2, respectively establishing task evaluation models in the Internet of things equipment, the edge server and the cloud server based on deep reinforcement learning;

s3, the Internet of things equipment optimizes the task evaluation model by using the task states and channel states obtained by all the Internet of things equipment, obtains a basic task processing decision according to the optimized task evaluation model, and sends the basic task processing decision and the optimized task evaluation model to the corresponding edge server;

the step S3 includes the following steps:

s3.1, each piece of Internet of things equipment acquires a task state and a channel state in the current environment and obtains a task processing preliminary decision according to a corresponding task evaluation model;

s3.2, each piece of Internet of things equipment integrates task processing preliminary decisions of all pieces of Internet of things equipment, and a task evaluation model is optimized once by taking the weighted sum of the minimized total processing delay and the energy consumption as a target;

s3.3, the Internet of things equipment makes a basic task processing decision according to the once optimized task evaluation model and sends the basic task processing decision and the once optimized task evaluation model to a corresponding edge server;

s4, the edge server performs edge aggregation on the task evaluation model sent by the Internet of things equipment by means of federal learning, optimizes the task evaluation model on the edge server according to the basic task processing decision sent by the Internet of things equipment to obtain a task processing high-level decision, and sends the task processing high-level decision to the Internet of things equipment;

s5, the edge server guides a task evaluation model in the Internet of things equipment by using transfer learning, and sends the task evaluation model after task processing high-level decision and optimization to the cloud server;

and S6, the cloud server performs cloud aggregation on the task evaluation model sent by the edge server by means of federal learning, makes a task processing city-level decision according to the task processing high-level decision sent by the edge server and the task evaluation model on the cloud server, sends the task processing city-level decision to the edge server, and guides the task evaluation model in the edge server by means of transfer learning.

2. The wisdom city-oriented social learning method of claim 1, wherein in step S3, the task state includes a CPU cycle of the task and a task data volume, and the channel state includes an authorized channel gain, an unauthorized channel gain, and a channel occupancy state of the wireless network.

3. The social learning method oriented to a smart city as claimed in claim 2, wherein the internet of things device comprises a primary user and a secondary user, the primary user is connected with the edge server through an authorized channel, the secondary user is connected with the edge server through an authorized channel or an unauthorized channel, and when the secondary user uses the authorized channel, the connection of the primary user is not affected.

4. The wisdom city-oriented social learning method of claim 1, wherein the task processing basic decision and the task processing preliminary decision each comprise an offload decision, a channel selection of a wireless network, and a computational resource budget.

5. The social learning method for smart cities as claimed in claim 1, wherein the step S4 comprises the steps of:

s4.1, each edge server acquires a channel state and edge available computing resources in the current environment, performs edge aggregation on the received once-optimized task evaluation model by using federal learning, and sends the task evaluation model subjected to edge aggregation to corresponding Internet of things equipment;

s4.2, each edge server obtains a task processing high-level preliminary decision according to the received task processing basic decision, the channel state and the edge available computing resources obtained in the step S4.1 and a task evaluation model on the edge server;

s4.3, each edge server integrates the task processing high-level preliminary decisions of all the edge servers, and performs primary optimization on the task evaluation model on the edge server by taking cost minimization as a target, and obtains the task processing high-level decisions according to the task evaluation model after primary optimization;

and S4.4, the edge server sends the task processing high-level decision to the corresponding Internet of things equipment, and the Internet of things equipment carries out secondary optimization on the task evaluation model on the Internet of things equipment by using the task processing high-level decision.

6. The wisdom city-oriented social learning method of claim 5, wherein the task processing high-level decision and the task processing high-level preliminary decision each comprise an evaluation result of a basic decision, collaborative edge server selection and an edge computing resource contribution.

7. The wisdom city-oriented social learning method of claim 6, wherein the cost is equal to a difference between an edge computing resource contribution minus a cost of cooperating with other edge servers.

8. The social learning method for smart cities as claimed in claim 1, wherein the step S6 comprises the steps of:

s6.1, the cloud server obtains a channel state and cloud available computing resources in the current environment, carries out cloud aggregation on the received task evaluation model sent by the edge server by means of federal learning, and sends the task evaluation model after cloud aggregation to the edge server;

s6.2, the cloud server inputs the received task processing high-level decision, the channel state obtained in the step S5.1 and the cloud available computing resources into a task evaluation model on the cloud server to obtain a task processing city-level decision, sends the task processing city-level decision to a corresponding edge server, and guides the task evaluation model in the edge server by using transfer learning;

and S6.3, performing secondary optimization on the task evaluation model on the edge server by the edge server according to the task processing city-level decision.

9. The wisdom city-oriented social learning method of claim 8, wherein the task processing city-level decision comprises an evaluation result of a high-level decision and a cloud computing resource contribution amount.

Images