


技术领域technical field
本发明涉及边缘计算技术领域,尤其涉及一种基于多种一致性协议的多边缘集群数据同步的方法及系统。The present invention relates to the technical field of edge computing, in particular to a method and system for data synchronization of multi-edge clusters based on multiple consensus protocols.
背景技术Background technique
随着AR/VR、高清视频、自动驾驶等新业务的孕育兴起,网络正在面临实时计算能力、超低时延、超大带宽等新的挑战。促进边缘计算产业发展,构建健康的生态环境,才能使终端用户获取新业务带来的极致体验,更加丰富的应用以及更安全可靠的使用。近年来,包括移动运营商、网络设备供应商、应用开发商,内容提供商在内的产业链各方纷纷加速边缘计算推进,促使这一技术获得了快速发展。边缘计算源自技术实践和商业实践,不仅是一项新兴技术和部署方式,更重要的是实现网络的底层开放,从而推动移动通信网络、互联网和物联网的深度融合。With the emergence of new services such as AR/VR, high-definition video, and autonomous driving, the network is facing new challenges such as real-time computing capabilities, ultra-low latency, and ultra-large bandwidth. Promoting the development of the edge computing industry and building a healthy ecological environment can enable end users to obtain the ultimate experience brought by new services, richer applications, and safer and more reliable use. In recent years, all parties in the industry chain, including mobile operators, network equipment suppliers, application developers, and content providers, have accelerated the advancement of edge computing, which has promoted the rapid development of this technology. Edge computing is derived from technical practice and business practice. It is not only an emerging technology and deployment method, but more importantly, it realizes the opening of the bottom layer of the network, thereby promoting the deep integration of mobile communication networks, the Internet and the Internet of Things.
传统云计算数据中心的数据同步机制只适合管理单个或少量大规模集群,无法很好地对布局分散、数量巨大的小规模边缘集群进行数据同步。The data synchronization mechanism of traditional cloud computing data centers is only suitable for managing a single or a small number of large-scale clusters, and cannot perform data synchronization well on small-scale edge clusters with scattered layouts and large numbers.
发明内容Contents of the invention
本发明实施例的目的是提供一种基于多种一致性协议的多边缘集群数据同步的方法及系统,以解决传统云计算数据中心的集群数据同步机制只适合单个或少量大规模边缘集群,无法高效地对布局分散、数量巨大的小规模边缘集群进行高效地数据同步的问题。The purpose of the embodiment of the present invention is to provide a multi-edge cluster data synchronization method and system based on multiple consistency protocols, so as to solve the problem that the cluster data synchronization mechanism of the traditional cloud computing data center is only suitable for a single or a small number of large-scale edge clusters, and cannot The problem of efficient data synchronization for small-scale edge clusters with scattered layout and huge number.
为了达到上述目的,本发明实施例所采用的技术方案如下:In order to achieve the above object, the technical scheme adopted in the embodiment of the present invention is as follows:
第一方面,本发明实施例提供一种基于多种一致性协议的多边缘集群数据同步的方法,每个边缘集群包括至少一个管理节点和至少一个工作节点,该方法包括:In the first aspect, an embodiment of the present invention provides a method for data synchronization of multi-edge clusters based on multiple consensus protocols, each edge cluster includes at least one management node and at least one working node, and the method includes:
边缘集群内部管理节点之间采用强一致性协议实现数据同步;The internal management nodes of the edge cluster adopt a strong consistency protocol to realize data synchronization;
边缘集群内部的管理节点与工作节点之间数据同步采用局域网弱一致性协议;The data synchronization between the management node and the working node in the edge cluster adopts the local area network weak consistency protocol;
不同边缘集群的管理节点之间数据同步采用广域网弱一致性协议。The data synchronization between the management nodes of different edge clusters adopts the weak consistency protocol of the wide area network.
进一步地,边缘集群内部管理节点之间采用强一致性协议实现数据同步时,只有当边缘集群中半数以上的管理节点数据达到一致,本次数据同步才算成功,否则即为失败。Furthermore, when the internal management nodes of the edge cluster adopt a strong consistency protocol to achieve data synchronization, only when the data of more than half of the management nodes in the edge cluster reaches consistency, the data synchronization will be considered successful, otherwise it will be a failure.
进一步地,所述管理节点的数量为奇数个。Further, the number of the management nodes is an odd number.
进一步地,通过局域网弱一致性协议以传染式的数据同步方法来实现同边缘集群管理节点与工作节点之间的数据同步。Furthermore, the data synchronization between the management nodes and working nodes of the same edge cluster is realized through the weak consistency protocol of the local area network and the contagious data synchronization method.
进一步地,通过广域网弱一致性协议以传染式的方法来实现不同边缘集群间管理节点的数据同步,广域网弱一致性协议不阻塞、容忍延迟,允许不同边缘集群间的请求转发。Furthermore, the data synchronization of management nodes between different edge clusters is realized in an infectious manner through the weak consistency protocol of the wide area network. The weak consistency protocol of the wide area network is non-blocking, tolerant of delay, and allows request forwarding between different edge clusters.
第二方面,本发明实施例还提供一种基于多种一致性协议的多边缘集群数据同步的系统,该系统包括:多个边缘集群,每个边缘集群包括至少一个管理节点和至少一个工作节点,边缘集群内部管理节点之间采用强一致性协议实现数据同步;边缘集群内部的管理节点与工作节点之间数据同步采用局域网弱一致性协议;不同边缘集群的管理节点之间数据同步采用广域网弱一致性协议。In the second aspect, the embodiment of the present invention also provides a multi-edge cluster data synchronization system based on multiple consensus protocols, the system includes: multiple edge clusters, each edge cluster includes at least one management node and at least one working node , the internal management nodes of the edge cluster adopt the strong consistency protocol to realize data synchronization; the data synchronization between the internal management nodes and the working nodes of the edge cluster adopts the LAN weak consistency protocol; the data synchronization between the management nodes of different edge clusters adopts the wide area network weak consistency protocol. consensus agreement.
根据以上技术方案,本发明的有益效果是:本发明将边缘集群内部管理节点之间采用强一致性协议实现数据同步;边缘集群内部的管理节点与工作节点之间数据同步采用局域网弱一致性协议;不同边缘集群的管理节点之间数据同步采用广域网弱一致性协议,能够对分散的、数量众多的边缘集群进行高效的数据同步,在确保边缘集群内部管理节点能够稳定运行的同时,边缘集群内、边缘集群间数据同步不会出现阻塞,还可以在不同边缘集群之前做请求转发,实现了边缘多边缘集群的统一管理。According to the above technical scheme, the beneficial effects of the present invention are: the present invention realizes data synchronization between the internal management nodes of the edge cluster using a strong consistency protocol; data synchronization between the internal management nodes and working nodes of the edge cluster adopts a weak local area network consistency protocol ;The data synchronization between the management nodes of different edge clusters adopts the weak consistency protocol of the wide area network, which can efficiently synchronize the data of scattered and large number of edge clusters. While ensuring the stable operation of the internal management nodes of the edge clusters, the edge clusters , Data synchronization between edge clusters will not be blocked, and request forwarding can be performed before different edge clusters, realizing unified management of edge multi-edge clusters.
附图说明Description of drawings
此处所说明的附图用来提供对本发明的进一步理解,构成本发明的一部分,本发明的示意性实施例及其说明用于解释本发明,并不构成对本发明的不当限定。在附图中:The accompanying drawings described here are used to provide a further understanding of the present invention, and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute improper limitations to the present invention. In the attached picture:
图1为本发明实施例中边缘同边缘集群管理节点之间强一致性数据同步、选举示意图;Figure 1 is a schematic diagram of strongly consistent data synchronization and election between edge and edge cluster management nodes in an embodiment of the present invention;
图2本发明实施例中边缘同边缘集群管理节点与工作节点之间局域网弱一致性数据同步、成员管理示意图;Fig. 2 is a schematic diagram of local area network weak consistency data synchronization and member management between edge-to-edge cluster management nodes and working nodes in an embodiment of the present invention;
图3本发明实施例中不同边缘集群管理节点之间广域网弱一致性数据同步、请求转发示意图。Fig. 3 is a schematic diagram of weakly consistent data synchronization and request forwarding in the wide area network between different edge cluster management nodes in the embodiment of the present invention.
具体实施方式Detailed ways
为使本申请的目的、技术方案和优点更加清楚,下面将结合本申请具体实施例及相应的附图对本申请技术方案进行清楚、完整地描述。显然,所描述的实施例仅是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and corresponding drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.
参考图1-图3,本发明实施例提供一种基于多种一致性协议的多边缘集群数据同步的方法,每个边缘集群包括至少一个管理节点和至少一个工作节点,该方法包括:Referring to Figures 1-3, an embodiment of the present invention provides a method for data synchronization of multi-edge clusters based on multiple consensus protocols, each edge cluster includes at least one management node and at least one working node, the method includes:
边缘集群内部管理节点之间采用强一致性协议实现数据同步;The internal management nodes of the edge cluster adopt a strong consistency protocol to realize data synchronization;
边缘集群内部的管理节点与工作节点之间数据同步采用局域网弱一致性协议;The data synchronization between the management node and the working node in the edge cluster adopts the local area network weak consistency protocol;
不同边缘集群的管理节点之间数据同步采用广域网弱一致性协议。The data synchronization between the management nodes of different edge clusters adopts the weak consistency protocol of the wide area network.
其中管理节点和工作节点可以是一台物理机,也可以是一台虚拟机,管理节点是边缘集群的控制节点,每个边缘集群中至少有一个管理节点来负责整个边缘集群的管理和控制;工作节点负责执行来自管理节点的工作负载。The management node and the working node can be a physical machine or a virtual machine. The management node is the control node of the edge cluster, and each edge cluster has at least one management node to be responsible for the management and control of the entire edge cluster; Worker nodes are responsible for executing workloads from management nodes.
本实施例中,构建系统时,首先用节点构建若干边缘集群,这些边缘集群通常位于不同的地理区域,每个边缘集群中都至少有一个管理节点和若干工作节点,组成边缘集群的所有节点都要符合边缘集群的认证和授权方式。为了保障管理节点的高可用,边缘集群中会部署多个管理节点(一般为奇数个,如1,3,5,7…)来对边缘集群进行管理,其中一个为主管理节点,其余为备份管理节点,主管理节点由边缘集群中的全体管理节点通过Raft协议选举产生。In this embodiment, when constructing the system, first build several edge clusters with nodes, these edge clusters are usually located in different geographical regions, each edge cluster has at least one management node and several working nodes, all the nodes that make up the edge cluster are It must comply with the authentication and authorization methods of edge clusters. In order to ensure the high availability of management nodes, multiple management nodes (usually an odd number, such as 1, 3, 5, 7...) will be deployed in the edge cluster to manage the edge cluster, one of which is the main management node, and the rest are backups Management node, the main management node is elected by all the management nodes in the edge cluster through the Raft protocol.
多个边缘集群的边缘集群内及边缘集群间的数据同步分为三个层次:同边缘集群管理节点之间的数据同步、同边缘集群管理节点与工作节点之间的数据同步以及不同边缘集群管理节点间的数据同步。Data synchronization within and between edge clusters of multiple edge clusters is divided into three levels: data synchronization between management nodes of the same edge cluster, data synchronization between management nodes and working nodes of the same edge cluster, and management of different edge clusters Data synchronization between nodes.
层次一:如图1所示,同边缘集群管理节点之间空间距离较近,网络质量较好,而且管理节点数量有限,采用强一致性协议进行数据同步:当同边缘集群半数以上的管理节点达到一致时即认为数据同步完成,边缘集群方可进行接下来的工作,否则边缘集群将会一直阻塞。具体步骤如下:Level 1: As shown in Figure 1, the spatial distance between the management nodes of the same edge cluster is relatively close, the network quality is good, and the number of management nodes is limited, and a strong consistency protocol is used for data synchronization: when more than half of the management nodes of the same edge cluster When the consistency is reached, it is considered that the data synchronization is completed, and the edge cluster can proceed with the next work, otherwise the edge cluster will always be blocked. Specific steps are as follows:
在启动之前,同边缘集群内所有的管理节点IP地址和端口信息都已经确定,在每个管理节点启动时,通过命令行参数或配置文件将所有的管理节点信息传入,所有的管理节点启动以后,会选出主管理节点管理该边缘集群,其他管理节点作为备用管理节点。Before starting, all management node IP addresses and port information in the same edge cluster have been determined. When each management node is started, all management node information is passed in through command line parameters or configuration files, and all management nodes are started. In the future, the main management node will be selected to manage the edge cluster, and other management nodes will be used as backup management nodes.
层次二:如图2所示,同边缘集群内管理节点与工作节点之间采用局域网弱一致性协议,通过“传染式”的方法实现数据同步,即数据变更节点挑选若干临近节点进行数据同步,之后就可以进行其他工作,而无需阻塞等待所有节点全部完成数据同步,才进行其他工作。具体步骤如下:Level 2: As shown in Figure 2, the local area network weak consistency protocol is adopted between the management nodes and the working nodes in the same edge cluster, and the data synchronization is realized through the "contagious" method, that is, the data change node selects several adjacent nodes for data synchronization, After that, other work can be performed without blocking and waiting for all nodes to complete data synchronization before performing other work. Specific steps are as follows:
(1)根据实际环境设置同边缘集群的工作节点,并启动;(1) Set up the working nodes of the same edge cluster according to the actual environment and start them;
(2)在知道任何一个管理节点的IP地址和端口的时,在工作节点主机上通过命令行便可加入管理节点形成边缘集群。(2) When knowing the IP address and port of any management node, you can join the management node to form an edge cluster through the command line on the working node host.
层次三:如图3所示,不同边缘集群管理节点之间采用广域网弱一致性协议,通过“传染式”的方法实现数据同步。需要进行数据同步的边缘集群首先随机通知临近的几个边缘集群进行数据同步,通知完成后就可以去执行其他任务而不需要阻塞。广域网弱一致性协议对网络延迟有更高的容忍度,适合于边缘集群空间距离大,网络质量不可控,网络延迟高的环境,同时广域网弱一致性协议还支持不同边缘集群间的访问请求转发。具体步骤如下:Level 3: As shown in Figure 3, different edge cluster management nodes adopt the weak consistency protocol of the wide area network, and realize data synchronization through the "contagious" method. The edge cluster that needs to perform data synchronization first randomly notifies several adjacent edge clusters to perform data synchronization. After the notification is completed, it can perform other tasks without blocking. The WAN weak consistency protocol has a higher tolerance for network delays, and is suitable for environments with large spatial distances between edge clusters, uncontrollable network quality, and high network delays. At the same time, the WAN weak consistency protocol also supports forwarding of access requests between different edge clusters . Specific steps are as follows:
不同边缘集群的管理节点之间,在知道其他边缘集群的任何一个管理节点的IP地址和端口,便可通过命令行加入其他边缘集群管理节点,形成多边缘集群。Between the management nodes of different edge clusters, if you know the IP address and port of any management node of other edge clusters, you can join other edge cluster management nodes through the command line to form a multi-edge cluster.
本发明实施例还提供一种基于多种一致性协议的多边缘集群数据同步的系统,该系统包括:多个边缘集群,每个边缘集群包括至少一个管理节点和至少一个工作节点,边缘集群内部管理节点之间采用强一致性协议实现数据同步;边缘集群内部的管理节点与工作节点之间数据同步采用局域网弱一致性协议;不同边缘集群的管理节点之间数据同步采用广域网弱一致性协议。The embodiment of the present invention also provides a multi-edge cluster data synchronization system based on multiple consensus protocols, the system includes: multiple edge clusters, each edge cluster includes at least one management node and at least one working node, and the edge cluster internal Strong consistency protocol is used between management nodes to realize data synchronization; data synchronization between management nodes and working nodes inside the edge cluster adopts weak consistency protocol of local area network; data synchronization between management nodes of different edge clusters adopts weak consistency protocol of wide area network.
以上所述仅为本发明的较佳实施例而已,并不用以限制本发明,凡在本发明的精神和原则之内,所作的任何修改、等同替换、改进等,均应包含在本发明的保护范围之内。The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of the present invention. within the scope of protection.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010788464.9ACN112055054B (en) | 2020-08-07 | 2020-08-07 | Method and system for multi-edge cluster data synchronization based on multiple consistency protocols |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010788464.9ACN112055054B (en) | 2020-08-07 | 2020-08-07 | Method and system for multi-edge cluster data synchronization based on multiple consistency protocols |
| Publication Number | Publication Date |
|---|---|
| CN112055054A CN112055054A (en) | 2020-12-08 |
| CN112055054Btrue CN112055054B (en) | 2023-04-07 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010788464.9AActiveCN112055054B (en) | 2020-08-07 | 2020-08-07 | Method and system for multi-edge cluster data synchronization based on multiple consistency protocols |
| Country | Link |
|---|---|
| CN (1) | CN112055054B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102436366A (en)* | 2011-08-22 | 2012-05-02 | 华为技术有限公司 | Data synchronization method and apparatus |
| CN102592420A (en)* | 2011-12-30 | 2012-07-18 | 北京理工大学 | Wireless sensor network testing node with microdrive and data synchronizing capacity |
| CN102681937A (en)* | 2012-05-15 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Correctness verifying method of cache consistency protocol |
| CN104360981A (en)* | 2014-11-12 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Design method of Cache coherence protocol for multi-core multi-processor platform |
| CN104915756A (en)* | 2015-05-22 | 2015-09-16 | 电信科学技术第五研究所 | Data consistency cloud auditing system and implementation method |
| CN106254413A (en)* | 2015-06-09 | 2016-12-21 | 哈曼国际工业有限公司 | Data syn-chronization |
| CN109271433A (en)* | 2018-09-03 | 2019-01-25 | 中新网络信息安全股份有限公司 | A kind of company-data synchronous method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8908564B2 (en)* | 2010-06-28 | 2014-12-09 | Avaya Inc. | Method for Media Access Control address learning and learning rate suppression |
| US8898728B2 (en)* | 2011-09-23 | 2014-11-25 | Oracle International Corporation | System and method of real-time change propagation and activation using a distributed object cache |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102436366A (en)* | 2011-08-22 | 2012-05-02 | 华为技术有限公司 | Data synchronization method and apparatus |
| CN102592420A (en)* | 2011-12-30 | 2012-07-18 | 北京理工大学 | Wireless sensor network testing node with microdrive and data synchronizing capacity |
| CN102681937A (en)* | 2012-05-15 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Correctness verifying method of cache consistency protocol |
| CN104360981A (en)* | 2014-11-12 | 2015-02-18 | 浪潮(北京)电子信息产业有限公司 | Design method of Cache coherence protocol for multi-core multi-processor platform |
| CN104915756A (en)* | 2015-05-22 | 2015-09-16 | 电信科学技术第五研究所 | Data consistency cloud auditing system and implementation method |
| CN106254413A (en)* | 2015-06-09 | 2016-12-21 | 哈曼国际工业有限公司 | Data syn-chronization |
| CN109271433A (en)* | 2018-09-03 | 2019-01-25 | 中新网络信息安全股份有限公司 | A kind of company-data synchronous method |
| Title |
|---|
| "基于RDMA和NVM的大数据系统一致性协议研究";吴昊,陈康,武永卫,郑纬民;《大数据》;20190531;全文* |
| "消息中间件元数据管理模块及发布订阅接口的设计与实现";姚思明;《中国优秀硕士学位论文全文数据库 (信息科技辑)》;20171231;全文* |
| Publication number | Publication date |
|---|---|
| CN112055054A (en) | 2020-12-08 |
| Publication | Publication Date | Title |
|---|---|---|
| US10911528B2 (en) | Managing replication of computing nodes for provided computer networks | |
| US11272044B2 (en) | Concurrent process execution | |
| Oktian et al. | Distributed SDN controller system: A survey on design choice | |
| US6233623B1 (en) | Replicated resource management system for managing resources in a distributed application and maintaining a relativistic view of state | |
| JP5498594B2 (en) | Consistency within the federation infrastructure | |
| CN103703724B (en) | A method of distributing resources | |
| US20010042139A1 (en) | Replicated resource management system for managing resources in a distributed application and maintaining a relativistic view of state | |
| US7711831B2 (en) | Methods, systems and computer program products for source address selection | |
| WO1997025673A9 (en) | Replicated resource management system for a distributed application maintaining a relativistic view of state | |
| WO2014150378A1 (en) | Distributed database management | |
| US9538440B2 (en) | Method and system of changing service points | |
| JP3798661B2 (en) | Method for processing a merge request received by a member of a group in a clustered computer system | |
| CN117296304A (en) | Managing data traffic for two simultaneous VPN connections using VPN clients | |
| CN111541608B (en) | Network communication method, system and related device | |
| WO2018121201A1 (en) | Distributed cluster service structure, node cooperation method and device, terminal and medium | |
| CN105162824A (en) | Host multi-path disposition method in cloud storage system, and system | |
| CN114885007B (en) | Method and electronic device for real-time strong consistency session synchronization | |
| CN112055054B (en) | Method and system for multi-edge cluster data synchronization based on multiple consistency protocols | |
| CN104601412A (en) | Independent test network system and establishing method thereof | |
| CN107306289B (en) | A method and device for load balancing based on cloud computing | |
| KR100742357B1 (en) | Intelligent Distributed Server System and Its Operation Method | |
| WO2012159339A1 (en) | Interface register method and device for network device to join cluster system | |
| CN113595749B (en) | Communication method and device for realizing multi-party network node intercommunication based on IRC platform | |
| JP6456304B2 (en) | Automated command and discovery process for network communications | |
| Shafaat et al. | Dealing with bootstrapping, maintenance, and network partitions and mergers in structured overlay networks |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |