CN115643271A - Method, device, server and medium for synchronizing multi-application data on cloud - Google Patents
Method, device, server and medium for synchronizing multi-application data on cloudDownload PDFInfo
- Publication number
- CN115643271A CN115643271ACN202211345921.2ACN202211345921ACN115643271ACN 115643271 ACN115643271 ACN 115643271ACN 202211345921 ACN202211345921 ACN 202211345921ACN 115643271 ACN115643271 ACN 115643271A
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- synchronization
- value
- abstract value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer And Data Communications (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请涉及数据传输技术领域，尤其涉及一种云上多应用数据同步方法、装置、服务器及介质。The present application relates to the technical field of data transmission, and in particular to a multi-application data synchronization method, device, server and medium on the cloud.
背景技术Background technique
云原生技术架构解决了传统应用数据同步在实现应用紧耦、弹性扩缩、可观测能力等方面需要专门研发的问题，而云应用数据同步由于软件平台化托管带来的故障排查困难，对一致性和安全性提出了更高的要求。The cloud-native technology architecture solves the problems of traditional application data synchronization that require special research and development in terms of application tight coupling, elastic scaling, and observability. However, cloud application data synchronization is difficult to troubleshoot due to software platform-based hosting. Higher requirements have been put forward for safety and security.
现有技术的数据同步技术主要包括应用紧耦合式数据同步、定时数据同步、消息中间件类异步数据同步和数据库实时数据同步。应用紧耦合类数据同步在业务逻辑中夹带数据同步逻辑实现数据同步。定时数据同步在应用业务功能以外设置定时器，通过定时器实现数据同步。消息中间件类的异步数据同步使用异步处理的方式实现数据同步。数据库实时数据同步借助数据库滚动事务日志实现字段的数据同步。Data synchronization technologies in the prior art mainly include application of tightly coupled data synchronization, timing data synchronization, message middleware type asynchronous data synchronization and database real-time data synchronization. The application of tightly coupled data synchronization implements data synchronization by entraining data synchronization logic in the business logic. Timed data synchronization sets a timer outside the application business function, and realizes data synchronization through the timer. The asynchronous data synchronization of message middleware uses asynchronous processing to realize data synchronization. Database real-time data synchronization realizes field data synchronization with the help of database rolling transaction logs.
上述四类数据同步方式适用于不同类型的业务系统，但是在数据一致性方面存在不同程度的不足，都有可能出现同步数据不一致的问题，进而影响后续相关业务的进行。The above four types of data synchronization methods are suitable for different types of business systems, but there are different degrees of deficiencies in data consistency, and there may be inconsistencies in the synchronization data, which will affect the subsequent related business.
发明内容Contents of the invention
本申请实施例提供了一种云上多应用数据同步方法、装置、服务器及介质，用以解决现有技术中数据同步的数据一致性较差的问题。Embodiments of the present application provide a multi-application data synchronization method, device, server, and medium on the cloud to solve the problem of poor data consistency in data synchronization in the prior art.
本申请提供了一种云上多应用数据同步方法，所述方法包括：The present application provides a multi-application data synchronization method on the cloud, the method comprising:
数据同步发起方获取待同步的数据表，通过哈希算法确定所述数据表的第一数据摘要值，将所述第一数据摘要值和所述数据表存放在数据同步请求信息中，将所述数据同步请求信息发送至数据同步接收方；The data synchronization initiator obtains the data table to be synchronized, determines the first data summary value of the data table through a hash algorithm, stores the first data summary value and the data table in the data synchronization request information, and stores the Send the above data synchronization request information to the data synchronization receiver;
数据同步接收方接收所述数据同步请求信息，获取其中携带的所述第一数据摘要值和数据表，通过哈希算法确定接收到的数据表的第二数据摘要值，判断所述第一数据摘要值和所述第二数据摘要值是否一致，如果是，对所述数据表进行导入操作。The data synchronization receiver receives the data synchronization request information, obtains the first data summary value and the data table carried in it, determines the second data summary value of the received data table through a hash algorithm, and judges the first data summary value Whether the summary value is consistent with the summary value of the second data, if yes, perform an import operation on the data table.
进一步地，所述数据同步发起方确定待同步的数据表，通过哈希算法确定所述数据表的第一数据摘要值，将所述第一数据摘要值和所述数据表存放在数据同步请求信息中包括：Further, the data synchronization initiator determines the data table to be synchronized, determines the first data summary value of the data table through a hash algorithm, and stores the first data summary value and the data table in the data synchronization request Information includes:
判断当前数据同步的业务场景是否为依赖业务场景，如果是，获取所述待同步的数据表对应的依赖数据表，通过哈希算法确定所述数据表的第一数据摘要值，以及所述依赖数据表的第三数据摘要值，将所述第一数据摘要值、第三数据摘要值和所述数据表存放在数据同步请求信息中。Judging whether the business scenario of the current data synchronization is a dependent business scenario, if so, obtaining the dependent data table corresponding to the data table to be synchronized, determining the first data digest value of the data table through a hash algorithm, and the dependent For the third data summary value of the data table, store the first data summary value, the third data summary value and the data table in the data synchronization request information.
进一步地，所述数据同步接收方接收所述同步请求信息，获取其中携带的所述第一数据摘要值和数据表，通过哈希算法确定接收到的数据表的第二数据摘要值，判断所述第一数据摘要值和所述第二数据摘要值是否一致，如果是，对所述数据表进行导入操作包括：Further, the data synchronization receiver receives the synchronization request information, obtains the first data summary value and the data table carried in it, determines the second data summary value of the received data table through a hash algorithm, and judges the Whether the first data summary value is consistent with the second data summary value, if yes, the import operation of the data table includes:
所述数据同步接收方接收所述同步请求信息，获取其中携带的所述第一数据摘要值、第三数据摘要值和数据表；The data synchronization receiver receives the synchronization request information, and obtains the first data summary value, the third data summary value and the data table carried therein;
根据预先配置的导入条件，确定与所述数据表对应的已导入的依赖数据表；Determine the imported dependent data table corresponding to the data table according to the pre-configured import condition;
通过哈希算法确定接收到的数据表的第二数据摘要值，以及所述已导入的依赖数据表的第四数据摘要值；Determine the second data digest value of the received data table and the fourth data digest value of the imported dependent data table through a hash algorithm;
判断所述第一数据摘要值和所述第二数据摘要值是否一致，并且所述第三数据摘要值和所述第四数据摘要值是否一致，如果均一致，对所述数据表进行导入操作。Judging whether the first data summary value is consistent with the second data summary value, and whether the third data summary value is consistent with the fourth data summary value, and if they are consistent, perform an import operation on the data table .
进一步地，所述对所述数据表进行导入操作之前，所述方法还包括：Further, before the import operation of the data table, the method also includes:
判断已导入数据中是否存在与所述数据表对应的被覆盖数据，如果是，对所述被覆盖数据进行备份，并保存至指定地址，将已导入数据中的所述被覆盖数据进行删除，对所述数据表进行导入操作。judging whether there is overwritten data corresponding to the data table in the imported data, if yes, backing up the overwritten data and saving it to a designated address, deleting the overwritten data in the imported data, Perform an import operation on the data table.
进一步地，所述方法还包括：Further, the method also includes:
对所述数据表进行导入操作之后，向所述数据同步发起方反馈数据已同步的提示信息。After the data table is imported, a prompt message that the data has been synchronized is fed back to the data synchronization initiator.
进一步地，所述方法还包括：Further, the method also includes:
若判断所述第一数据摘要值和所述第二数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送同步数据异常的提示信息。If it is judged that the first data summary value is inconsistent with the second data summary value, roll back the data table to the data synchronization initiator, and send a reminder of abnormal synchronization data to the manager of the data synchronization initiator information.
进一步地，所述方法还包括：Further, the method also includes:
若判断所述第三数据摘要值和所述第四数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送依赖的同步任务异常的提示信息。If it is determined that the third data summary value is inconsistent with the fourth data summary value, roll back the data table to the data synchronization initiator, and send a dependent synchronization task exception to the manager of the data synchronization initiator prompt information.
另一方面，本申请提供了一种云上多应用数据同步装置，所述装置包括：In another aspect, the present application provides a multi-application data synchronization device on the cloud, the device comprising:
发送模块，用于获取待同步的数据表，通过哈希算法确定所述数据表的第一数据摘要值，将所述第一数据摘要值和所述数据表存放在数据同步请求信息中，将所述数据同步请求信息发送至数据同步接收方；A sending module, configured to obtain a data table to be synchronized, determine a first data summary value of the data table through a hash algorithm, store the first data summary value and the data table in the data synchronization request information, and store the The data synchronization request information is sent to the data synchronization receiver;
接收模块，用于接收所述数据同步请求信息，获取其中携带的所述第一数据摘要值和数据表，通过哈希算法确定接收到的数据表的第二数据摘要值，判断所述第一数据摘要值和所述第二数据摘要值是否一致，如果是，对所述数据表进行导入操作。A receiving module, configured to receive the data synchronization request information, obtain the first data summary value and the data table carried in it, determine the second data summary value of the received data table through a hash algorithm, and judge the first Whether the data summary value is consistent with the second data summary value, and if so, perform an import operation on the data table.
进一步地，所述发送模块，具体用于判断当前数据同步的业务场景是否为依赖业务场景，如果是，获取所述待同步的数据表对应的依赖数据表，通过哈希算法确定所述数据表的第一数据摘要值，以及所述依赖数据表的第三数据摘要值，将所述第一数据摘要值、第三数据摘要值和所述数据表存放在数据同步请求信息中。Further, the sending module is specifically used to judge whether the current data synchronization business scenario is a dependent business scenario, and if so, obtain the dependent data table corresponding to the data table to be synchronized, and determine the data table through a hash algorithm and the third data summary value of the dependent data table, storing the first data summary value, the third data summary value and the data table in the data synchronization request information.
进一步地，所述接收模块，具体用于所述数据同步接收方接收所述同步请求信息，获取其中携带的所述第一数据摘要值、第三数据摘要值和数据表；根据预先配置的导入条件，确定与所述数据表对应的已导入的依赖数据表；通过哈希算法确定接收到的数据表的第二数据摘要值，以及所述已导入的依赖数据表的第四数据摘要值；判断所述第一数据摘要值和所述第二数据摘要值是否一致，并且所述第三数据摘要值和所述第四数据摘要值是否一致，如果均一致，对所述数据表进行导入操作。Further, the receiving module is specifically used for the data synchronization receiver to receive the synchronization request information, and obtain the first data summary value, the third data summary value and the data table carried in it; according to the pre-configured import The condition is to determine the imported dependent data table corresponding to the data table; determine the second data summary value of the received data table and the fourth data summary value of the imported dependent data table through a hash algorithm; Judging whether the first data summary value is consistent with the second data summary value, and whether the third data summary value is consistent with the fourth data summary value, and if they are consistent, perform an import operation on the data table .
进一步地，所述接收模块，还用于判断已导入数据中是否存在与所述数据表对应的被覆盖数据，如果是，对所述被覆盖数据进行备份，并保存至指定地址，将已导入数据中的所述被覆盖数据进行删除，对所述数据表进行导入操作。Further, the receiving module is also used to judge whether there is overwritten data corresponding to the data table in the imported data, and if so, back up the overwritten data and save it to a specified address, and save the overwritten data to the imported The overwritten data in the data is deleted, and the data table is imported.
进一步地，所述装置还包括：Further, the device also includes:
第一反馈模块，用于对所述数据表进行导入操作之后，向所述数据同步发起方反馈数据已同步的提示信息。The first feedback module is configured to feed back the prompt information that the data has been synchronized to the data synchronization initiator after the import operation is performed on the data table.
进一步地，所述装置还包括：Further, the device also includes:
第二反馈模块，用于若判断所述第一数据摘要值和所述第二数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送同步数据异常的提示信息。The second feedback module is used to roll back the data table to the data synchronization initiator and report to the management of the data synchronization initiator if it is judged that the first data summary value is inconsistent with the second data summary value. The staff sends a prompt message that the synchronization data is abnormal.
进一步地，所述装置还包括：Further, the device also includes:
第三反馈模块，用于若判断所述第三数据摘要值和所述第四数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送依赖的同步任务异常的提示信息。The third feedback module is used to roll back the data table to the data synchronization initiator if it is judged that the third data summary value is inconsistent with the fourth data summary value, and report to the management of the data synchronization initiator The staff sends a notification that the dependent synchronization task is abnormal.
再一方面，本申请提供了一种服务器，包括处理器、通信接口、存储器和通信总线，其中，处理器，通信接口，存储器通过通信总线完成相互间的通信；In another aspect, the present application provides a server, including a processor, a communication interface, a memory, and a communication bus, wherein, the processor, the communication interface, and the memory complete mutual communication through the communication bus;
存储器，用于存放计算机程序；memory for storing computer programs;
处理器，用于执行存储器上所存放的程序时，实现上述任一项所述的方法步骤。The processor is configured to implement the method steps described in any one of the above when executing the program stored in the memory.
再一方面，本申请提供了一种计算机可读存储介质，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现上述任一项所述的方法步骤。In another aspect, the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any one of the foregoing are implemented.
本申请提供了一种云上多应用数据同步方法、装置、服务器及介质，所述方法包括：数据同步发起方确定待同步的数据表，通过哈希算法确定所述数据表的第一数据摘要值，将所述第一数据摘要值和所述数据表存放在数据同步请求信息中，将所述数据同步请求信息发送至数据同步接收方；数据同步接收方接收所述同步请求信息，获取其中携带的所述第一数据摘要值和数据表，通过哈希算法确定接收到的数据表的第二数据摘要值，判断所述第一数据摘要值和所述第二数据摘要值是否一致，如果是，对所述数据表进行导入操作。The present application provides a multi-application data synchronization method, device, server and medium on the cloud. The method includes: the data synchronization initiator determines the data table to be synchronized, and determines the first data abstract of the data table through a hash algorithm value, store the first data abstract value and the data table in the data synchronization request information, and send the data synchronization request information to the data synchronization receiver; the data synchronization receiver receives the synchronization request information, and obtains the The first data summary value carried and the data table are used to determine the second data summary value of the received data table through a hash algorithm, and determine whether the first data summary value is consistent with the second data summary value, if Yes, perform an import operation on the data table.
上述的技术方案具有如下优点或有益效果：The above-mentioned technical scheme has the following advantages or beneficial effects:
本申请中，数据同步发起方在发起数据同步请求信息时，首先获取待同步的数据表，然后计算数据表的第一数据摘要值，将第一数据摘要值和数据表一并携带在数据同步请求信息中发送至数据同步接收方。数据同步接收方接收数据同步请求信息，获取其中携带的第一数据摘要值和数据表之后，首先计算接收到的数据表的第二数据摘要值。当判断第一数据摘要值和第二数据摘要值一致时，说明数据同步发起方发送的数据表与数据同步接收方接收到的数据表完全一致，进而导入数据表，从而严格保证数据表同步的一致性。In this application, when the data synchronization initiator initiates the data synchronization request information, it first obtains the data table to be synchronized, then calculates the first data summary value of the data table, and carries the first data summary value and the data table together in the data synchronization Sent to the data synchronization receiver in the request message. After the data synchronization receiver receives the data synchronization request information and obtains the first data summary value and the data table carried therein, it first calculates the second data summary value of the received data table. When it is judged that the first data summary value is consistent with the second data summary value, it means that the data table sent by the data synchronization initiator is completely consistent with the data table received by the data synchronization receiver, and then the data table is imported to strictly ensure the synchronization of the data table consistency.
附图说明Description of drawings
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简要介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域的普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.
图1为本申请提供的云上多应用数据同步过程示意图；FIG. 1 is a schematic diagram of the multi-application data synchronization process on the cloud provided by the present application;
图2为本申请提供的云上多应用数据同步架构图；Fig. 2 is a multi-application data synchronization architecture diagram on the cloud provided by the present application;
图3为本申请提供的同步装置的结构示意图；FIG. 3 is a schematic structural diagram of a synchronization device provided by the present application;
图4为本申请提供的两维平面基础设施架构示意图；FIG. 4 is a schematic diagram of the two-dimensional plane infrastructure architecture provided by the present application;
图5为本申请提供的数据同步原理架构图；FIG. 5 is a schematic diagram of the data synchronization principle provided by the present application;
图6为本申请提供的数据同步流程图；Figure 6 is a flow chart of data synchronization provided by this application;
图7为本申请提供的云上多应用数据同步装置结构示意图；FIG. 7 is a schematic structural diagram of a multi-application data synchronization device on the cloud provided by the present application;
图8为本申请提供的服务器结构示意图。FIG. 8 is a schematic diagram of the server structure provided by this application.
具体实施方式Detailed ways
为使本申请的目的和实施方式更加清楚，下面将结合本申请示例性实施例中的附图，对本申请示例性实施方式进行清楚、完整地描述，显然，描述的示例性实施例仅是本申请一部分实施例，而不是全部的实施例。In order to make the purpose and implementation of the application clearer, the following will clearly and completely describe the exemplary implementation of the application in conjunction with the accompanying drawings in the exemplary embodiment of the application. Obviously, the described exemplary embodiment is only the present application. Claim some of the examples, not all of them.
需要说明的是，本申请中对于术语的简要说明，仅是为了方便理解接下来描述的实施方式，而不是意图限定本申请的实施方式。除非另有说明，这些术语应当按照其普通和通常的含义理解。It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementations described below, and is not intended to limit the implementations of this application. These terms are to be understood according to their ordinary and usual meaning unless otherwise stated.
本申请中说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”等是用于区别类似或同类的对象或实体，而不必然意味着限定特定的顺序或先后次序，除非另外注明。应该理解这样使用的用语在适当情况下可以互换。The terms "first", "second", and "third" in the description and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean limiting specific sequential or sequential unless otherwise noted. It is to be understood that the terms so used are interchangeable under appropriate circumstances.
术语“包括”和“具有”以及他们的任何变形，意图在于覆盖但不排他的包含，例如，包含了一系列组件的产品或设备不必限于清楚地列出的所有组件，而是可包括没有清楚地列出的或对于这些产品或设备固有的其它组件。The terms "comprising" and "having", as well as any variations thereof, are intended to be inclusive but not exclusive, for example, a product or device comprising a series of components is not necessarily limited to all components expressly listed, but may include not expressly listed other components listed or inherent to these products or equipment.
术语“模块”是指任何已知或后来开发的硬件、软件、固件、人工智能、模糊逻辑或硬件或/和软件代码的组合，能够执行与该元件相关的功能。The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or/and software code capable of performing the function associated with that element.
最后应说明的是：以上各实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述各实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit it; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present application. scope.
为了方便解释，已经结合具体的实施方式进行了上述说明。但是，上述示例性的讨论不是意图穷尽或者将实施方式限定到上述公开的具体形式。根据上述的教导，可以得到多种修改和变形。上述实施方式的选择和描述是为了更好的解释原理以及实际的应用，从而使得本领域技术人员更好的使用所述实施方式以及适于具体使用考虑的各种不同的变形的实施方式。For convenience of explanation, the above description has been made in conjunction with specific implementation manners. However, the above exemplary discussion is not intended to be exhaustive or to limit the implementations to the precise forms disclosed above. Many modifications and variations are possible in light of the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various modified embodiments suitable for specific use considerations.
图1为本申请提供的云上多应用数据同步过程示意图，该过程包括以下步骤：Figure 1 is a schematic diagram of the multi-application data synchronization process on the cloud provided by this application, and the process includes the following steps:
S101：数据同步发起方获取待同步的数据表，通过哈希算法确定所述数据表的第一数据摘要值，将所述第一数据摘要值和所述数据表存放在数据同步请求信息中，将所述数据同步请求信息发送至数据同步接收方。S101: The data synchronization initiator obtains the data table to be synchronized, determines the first data summary value of the data table through a hash algorithm, stores the first data summary value and the data table in the data synchronization request information, Send the data synchronization request information to the data synchronization receiver.
S102：数据同步接收方接收所述数据同步请求信息，获取其中携带的所述第一数据摘要值和数据表，通过哈希算法确定接收到的数据表的第二数据摘要值，判断所述第一数据摘要值和所述第二数据摘要值是否一致，如果是，对所述数据表进行导入操作。S102: The data synchronization receiver receives the data synchronization request information, obtains the first data summary value and the data table carried in it, determines the second data summary value of the received data table through a hash algorithm, and determines the first data summary value Whether a data summary value is consistent with the second data summary value, and if so, perform an import operation on the data table.
本申请提供的云上多应用数据同步方法应用于云上多应用架构中的服务器，云上多应用架构中包括多个服务器，各服务器之间存在数据同步需求。将发起数据同步的服务器作为数据同步发起方，将数据同步的目的服务器作为数据同步接收方。The multi-application data synchronization method on the cloud provided by this application is applied to servers in the multi-application architecture on the cloud. The multi-application architecture on the cloud includes multiple servers, and there is a need for data synchronization among the servers. The server that initiates data synchronization is used as the data synchronization initiator, and the destination server of data synchronization is used as the data synchronization receiver.
数据同步发起方获取待同步的数据表之后，对待同步的数据表进行哈希计算，得到第一数据摘要值，然后将第一数据摘要值和待同步的数据表携带在数据同步请求信息中，将数据同步请求信息发送至数据同步接收方。数据同步接收方接收到数据同步请求信息之后，对数据同步请求信息进行解析，获取其中携带的第一数据摘要值和数据表，然后对接收到的数据表进行哈希计算，得到第二数据摘要值，并判断第一数据摘要值和第二数据摘要值是否一致，如果是，对数据表进行导入操作，完成数据同步，否则不进行数据导入。After the data synchronization initiator acquires the data table to be synchronized, it performs hash calculation on the data table to be synchronized to obtain the first data summary value, and then carries the first data summary value and the data table to be synchronized in the data synchronization request information, Send the data synchronization request information to the data synchronization receiver. After the data synchronization receiver receives the data synchronization request information, it analyzes the data synchronization request information, obtains the first data summary value and the data table carried in it, and then performs hash calculation on the received data table to obtain the second data summary Value, and judge whether the first data summary value is consistent with the second data summary value, if yes, perform an import operation on the data table to complete data synchronization, otherwise, do not perform data import.
本申请中，为了保证数据同步的安全性，数据同步发起方将第一数据摘要值和待同步的数据表携带在数据同步请求信息中之后，通过数字签名算法对数据同步请求信息添加数字签名。数据同步接收方接收到数据同步请求信息之后，首先对数据同步请求信息进行验签。若验签通过，对数据同步请求信息进行解析，获取其中携带的第一数据摘要值和数据表，然后对接收到的数据表进行哈希计算，得到第二数据摘要值，并判断第一数据摘要值和第二数据摘要值是否一致，如果是，对数据表进行导入操作，完成数据同步，否则不进行数据导入。若验签不通过，不进行后续对数据同步请求信息进行解析的过程。In this application, in order to ensure the security of data synchronization, the initiator of data synchronization carries the first data summary value and the data table to be synchronized in the data synchronization request information, and adds a digital signature to the data synchronization request information through a digital signature algorithm. After receiving the data synchronization request information, the data synchronization receiver first performs signature verification on the data synchronization request information. If the signature verification is passed, analyze the data synchronization request information, obtain the first data summary value and data table carried in it, and then perform hash calculation on the received data table to obtain the second data summary value, and judge the first data summary value Whether the summary value is consistent with the second data summary value, if yes, perform an import operation on the data table to complete data synchronization, otherwise, do not perform data import. If the signature verification fails, the subsequent process of parsing the data synchronization request information will not be performed.
本申请中，数据同步发起方在发起数据同步请求信息时，首先获取待同步的数据表，然后计算数据表的第一数据摘要值，将第一数据摘要值和数据表一并携带在数据同步请求信息中发送至数据同步接收方。数据同步接收方接收数据同步请求信息，获取其中携带的第一数据摘要值和数据表之后，首先计算接收到的数据表的第二数据摘要值。当判断第一数据摘要值和第二数据摘要值一致时，说明数据同步发起方发送的数据表与数据同步接收方接收到的数据表完全一致，进而导入数据表，从而严格保证数据表同步的一致性。In this application, when the data synchronization initiator initiates the data synchronization request information, it first obtains the data table to be synchronized, then calculates the first data summary value of the data table, and carries the first data summary value and the data table together in the data synchronization Sent to the data synchronization receiver in the request message. After the data synchronization receiver receives the data synchronization request information and obtains the first data summary value and the data table carried therein, it first calculates the second data summary value of the received data table. When it is judged that the first data summary value is consistent with the second data summary value, it means that the data table sent by the data synchronization initiator is completely consistent with the data table received by the data synchronization receiver, and then the data table is imported to strictly ensure the synchronization of the data table consistency.
为了进一步保证数据同步的一致性，本申请中，所述数据同步发起方确定待同步的数据表，通过哈希算法确定所述数据表的第一数据摘要值，将所述第一数据摘要值和所述数据表存放在数据同步请求信息中包括：In order to further ensure the consistency of data synchronization, in this application, the data synchronization initiator determines the data table to be synchronized, determines the first data summary value of the data table through a hash algorithm, and converts the first data summary value to And the data table stored in the data synchronization request information includes:
判断当前数据同步的业务场景是否为依赖业务场景，如果是，获取所述待同步的数据表对应的依赖数据表，通过哈希算法确定所述数据表的第一数据摘要值，以及所述依赖数据表的第三数据摘要值，将所述第一数据摘要值、第三数据摘要值和所述数据表存放在数据同步请求信息中。Judging whether the business scenario of the current data synchronization is a dependent business scenario, if so, obtaining the dependent data table corresponding to the data table to be synchronized, determining the first data digest value of the data table through a hash algorithm, and the dependent For the third data summary value of the data table, store the first data summary value, the third data summary value and the data table in the data synchronization request information.
数据同步发起方在向数据同步接收方发送数据同步请求信息之前，首先判断当前数据同步的业务场景是否为依赖业务场景，如果否，则按照上述方案进行数据同步，如果是依赖业务场景，根据预先确定的数据表之间的依赖关系，获取待同步的数据表对应的依赖数据表。对待同步的数据表进行哈希计算得到第一数据摘要值，对依赖数据表进行哈希计算，得到第三数据摘要值，将第一数据摘要值、第三数据摘要值和数据表存放在数据同步请求信息中。Before the data synchronization initiator sends the data synchronization request information to the data synchronization receiver, it first judges whether the current data synchronization business scenario is dependent on the business scenario, if not, performs data synchronization according to the above scheme, if it is dependent on the business scenario, according to the pre- The determined dependency relationship between the data tables is used to obtain the dependent data tables corresponding to the data tables to be synchronized. Perform hash calculation on the data table to be synchronized to obtain the first data summary value, perform hash calculation on the dependent data table to obtain the third data summary value, store the first data summary value, the third data summary value and the data table in the data Synchronization request information.
对于依赖业务场景，所述数据同步接收方接收所述同步请求信息，获取其中携带的所述第一数据摘要值和数据表，通过哈希算法确定接收到的数据表的第二数据摘要值，判断所述第一数据摘要值和所述第二数据摘要值是否一致，如果是，对所述数据表进行导入操作包括：For dependent business scenarios, the data synchronization receiver receives the synchronization request information, obtains the first data summary value and data table carried therein, and determines the second data summary value of the received data table through a hash algorithm, Judging whether the first data summary value is consistent with the second data summary value, if yes, performing an import operation on the data table includes:
所述数据同步接收方接收所述同步请求信息，获取其中携带的所述第一数据摘要值、第三数据摘要值和数据表；The data synchronization receiver receives the synchronization request information, and obtains the first data summary value, the third data summary value and the data table carried therein;
根据预先配置的导入条件，确定与所述数据表对应的已导入的依赖数据表；Determine the imported dependent data table corresponding to the data table according to the pre-configured import condition;
通过哈希算法确定接收到的数据表的第二数据摘要值，以及所述已导入的依赖数据表的第四数据摘要值；Determine the second data digest value of the received data table and the fourth data digest value of the imported dependent data table through a hash algorithm;
判断所述第一数据摘要值和所述第二数据摘要值是否一致，并且所述第三数据摘要值和所述第四数据摘要值是否一致，如果均一致，对所述数据表进行导入操作。Judging whether the first data summary value is consistent with the second data summary value, and whether the third data summary value is consistent with the fourth data summary value, and if they are consistent, perform an import operation on the data table .
本申请中，对于依赖业务场景，数据同步接收方接收到数据同步请求信息，并通过解析获取到其中携带的所述第一数据摘要值、第三数据摘要值和数据表之后，根据预先配置的导入条件，确定与数据表对应的已导入的依赖数据表；然后对已导入的依赖数据表进行哈希计算得到第四数据摘要值，对数据表进行哈希计算得到第二数据摘要值。In this application, for dependent business scenarios, the data synchronization receiver receives the data synchronization request information, and obtains the first data summary value, the third data summary value and the data table carried in it through parsing, and then according to the pre-configured The import condition determines the imported dependent data table corresponding to the data table; then performs hash calculation on the imported dependent data table to obtain a fourth data summary value, and performs hash calculation on the data table to obtain a second data summary value.
若第一数据摘要值和第二数据摘要值，说明当前同步的数据表一致，若第三数据摘要值和第四数据摘要值一致，说明当前同步的数据表所对应的依赖数据表一致。在当前同步的数据表一致，并且当前同步的数据表所对应的依赖数据表一致的情况下，对数据表进行导入操作，完成数据同步，否则不导入数据表。If the value of the first data digest is the same as the value of the second data digest, it means that the currently synchronized data table is consistent, and if the value of the third data digest is consistent with the value of the fourth data digest, it means that the dependent data table corresponding to the currently synchronized data table is consistent. When the currently synchronized data table is consistent and the dependent data table corresponding to the currently synchronized data table is consistent, the data table is imported to complete the data synchronization, otherwise the data table is not imported.
为了进一步保证数据同步的一致性，所述对所述数据表进行导入操作之前，所述方法还包括：In order to further ensure the consistency of data synchronization, before the import operation of the data table, the method further includes:
判断已导入数据中是否存在与所述数据表对应的被覆盖数据，如果是，对所述被覆盖数据进行备份，并保存至指定地址，将已导入数据中的所述被覆盖数据进行删除，对所述数据表进行导入操作。judging whether there is overwritten data corresponding to the data table in the imported data, if yes, backing up the overwritten data and saving it to a designated address, deleting the overwritten data in the imported data, Perform an import operation on the data table.
在一种情况下，若数据同步接收方对数据表校验通过，在数据表导入的过程中出现网络故障或者导入异常时，此时本地导入数据不完整，当需要重新导入数据表时，数据同步接收方的数据库中便存在被覆盖数据，此时需要对被覆盖数据进行备份，并保存至指定地址。当根据业务需求需要请求被覆盖数据时，可以直接到指定地址获取被覆盖数据并进行相应的业务。对被覆盖数据进行备份，并保存至指定地址之后，数据同步接收方删除数据库中被覆盖数据，然后再对数据表进行导入操作。从而保证数据同步的一致性。In one case, if the data synchronization recipient passes the verification of the data table, and a network failure or import exception occurs during the data table import process, the locally imported data is incomplete at this time. When the data table needs to be re-imported, the data The overwritten data exists in the database of the synchronization receiver. At this time, the overwritten data needs to be backed up and saved to the specified address. When requesting covered data according to business needs, you can directly go to the designated address to obtain the covered data and perform corresponding business. After backing up the overwritten data and saving it to the specified address, the data synchronization receiver deletes the overwritten data in the database, and then imports the data table. Thereby ensuring the consistency of data synchronization.
所述方法还包括：The method also includes:
对所述数据表进行导入操作之后，向所述数据同步发起方反馈数据已同步的提示信息。After the data table is imported, a prompt message that the data has been synchronized is fed back to the data synchronization initiator.
数据同步接收方对数据表进行导入操作之后，可以通过报文的形式向数据同步发起方反馈数据已同步的提示信息。以使数据同步发起方获得数据同步的状态。After the data synchronization receiver imports the data table, it can feed back the prompt information that the data has been synchronized to the data synchronization initiator in the form of a message. To enable the data synchronization initiator to obtain the status of data synchronization.
所述方法还包括：The method also includes:
若判断所述第一数据摘要值和所述第二数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送同步数据异常的提示信息。If it is judged that the first data summary value is inconsistent with the second data summary value, roll back the data table to the data synchronization initiator, and send a reminder of abnormal synchronization data to the manager of the data synchronization initiator information.
若判断第一数据摘要值和第二数据摘要值不一致，则不进行数据同步，也就是不导入数据表。第一数据摘要值和第二数据摘要值不一致说明同步数据异常，此时回退数据表至数据同步发起方，并通过报文的形式向数据同步发起方的管理人员发送同步数据异常的提示信息。以使数据同步发起方获得数据同步的异常状态，已经异常问题所在。If it is determined that the first data summary value is inconsistent with the second data summary value, no data synchronization is performed, that is, no data table is imported. The inconsistency between the first data summary value and the second data summary value indicates that the synchronization data is abnormal. At this time, the data table is rolled back to the data synchronization initiator, and a prompt message of the synchronization data exception is sent to the management personnel of the data synchronization initiator in the form of a message. . To make the data synchronization initiator obtain the abnormal state of data synchronization, the problem is already abnormal.
所述方法还包括：The method also includes:
若判断所述第三数据摘要值和所述第四数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送依赖的同步任务异常的提示信息。If it is determined that the third data summary value is inconsistent with the fourth data summary value, roll back the data table to the data synchronization initiator, and send a dependent synchronization task exception to the manager of the data synchronization initiator prompt information.
若判断第三数据摘要值和第四数据摘要值不一致，则不进行数据同步，也就是不导入数据表。第三数据摘要值和第四数据摘要值不一致说明依赖数据同步异常，此时回退数据表至数据同步发起方，并通过报文的形式向数据同步发起方的管理人员发送依赖的同步任务异常的提示信息。以使数据同步发起方获得数据同步的异常状态，已经异常问题所在。If it is judged that the third data summary value is inconsistent with the fourth data summary value, no data synchronization is performed, that is, no data table is imported. The inconsistency between the third data summary value and the fourth data summary value indicates that the dependent data synchronization is abnormal. At this time, the data table is rolled back to the data synchronization initiator, and the dependent synchronization task exception is sent to the management personnel of the data synchronization initiator in the form of a message. prompt information. In order for the data synchronization initiator to obtain the abnormal state of data synchronization, the problem is already abnormal.
下面对本申请提供的云上多应用数据同步过程进行详细说明。The following describes in detail the multi-application data synchronization process on the cloud provided by this application.
1、云上多应用数据同步基于API接口实现，本申请数据同步各处理环节采用纯异步执行方式处理，API接口的相应时间为毫秒级。1. Multi-application data synchronization on the cloud is realized based on the API interface. In this application, each processing link of data synchronization is processed in a purely asynchronous execution mode, and the corresponding time of the API interface is millisecond level.
2、本申请可以实现数据同步发起方多线程并发向多个数据同步接收方发送数据同步请求信息，CPU利用率高，数据同步的处理环节均以数据库表为单位采用多线程并发处理，业务处理并发度高，能充分利用CPU计算资源。2. This application can realize the multi-threaded concurrent data synchronization initiator to send data synchronization request information to multiple data synchronization receivers, the CPU utilization rate is high, and the data synchronization processing links are all processed by multi-threaded concurrent processing with the database table as the unit, business processing High concurrency, can make full use of CPU computing resources.
3、本申请为控制整体处理时间，单次数据同步体量控制在一定阈值范围内，例如100M内，超过100M分多次请求发送。单张表记录数支持“数量上限”配置，分报文发送。从而保证运行期应用占用有限内存，数据传输均为采用流传输，均是有限内存的处理过程。3. In order to control the overall processing time in this application, the volume of a single data synchronization is controlled within a certain threshold range, for example, within 100M, and more than 100M is sent in multiple requests. The number of records in a single table supports the "upper limit" configuration, which is sent in packets. In this way, it is ensured that the application occupies limited memory during the runtime, and data transmission is all performed by stream transmission, which is a processing process with limited memory.
4、在传输数据同步请求信息时通过数字签名算法提高数据传输的安全性，并且传输使用HTTPS网络传输，进一步提高数据传输安全性。整体数据摘要校验，保证了数据同步的一致性。4. When transmitting data synchronization request information, the security of data transmission is improved through the digital signature algorithm, and the transmission uses HTTPS network transmission to further improve the security of data transmission. The overall data summary verification ensures the consistency of data synchronization.
本申请使用云原生技术设计的多中心应用数据同步，采用两维平面微服务架构，在管理平面植入数据签名算法为核心的不可逆安全数据核验技术，在数据平面支持两层核验因子计算的方法实现了批量处理类系统在多中心热备架构下数据同步高一致性和数据安全的要求，同时具备与应用系统解耦，弹性扩缩能力和广泛的适用性。对金融类交易数据中高离散度交易要素的甄别和对这些交易要素的不可逆安全数据摘要及核验技术，确保了多中心数据同步的高一致性和数据安全。This application uses cloud-native technology to design multi-center application data synchronization, adopts two-dimensional plane micro-service architecture, implants data signature algorithm as the core irreversible security data verification technology in the management plane, and supports two-layer verification factor calculation method in the data plane It realizes the requirements of data synchronization, high consistency and data security for batch processing systems under the multi-center hot standby architecture, and at the same time has decoupling from application systems, elastic expansion and contraction capabilities and wide applicability. The identification of highly discrete transaction elements in financial transaction data and the irreversible security data summary and verification technology for these transaction elements ensure the high consistency and data security of multi-center data synchronization.
图2为本申请提供的云上多应用数据同步架构图，包括数据同步场次调度、数据同步场次服务以及各个业务系统。各个业务系统中包括数据库和独立于数据库配置的同步装置。Figure 2 is a diagram of the multi-application data synchronization architecture on the cloud provided by this application, including data synchronization session scheduling, data synchronization session service, and various business systems. Each business system includes a database and a synchronization device independent of database configuration.
图3为本申请提供的同步装置的结构示意图，同步装置主要分为四个工作模块，分别是配置管理模块、多中心管理模块、数据同步模块、任务监控模块。Fig. 3 is a schematic structural diagram of the synchronization device provided by the present application. The synchronization device is mainly divided into four working modules, namely a configuration management module, a multi-center management module, a data synchronization module, and a task monitoring module.
本申请所提出的同步装置使用HTTP通用API通讯协议，借助数据安全核验技术实现了云上应用的低耦合数据同步功能。具体的功能包括支持动态扩缩的两维平面中心管理和数据同步功能，下面分别详细介绍具体的工作原理。The synchronization device proposed in this application uses the HTTP general API communication protocol, and realizes the low-coupling data synchronization function of cloud applications with the help of data security verification technology. The specific functions include the two-dimensional plane center management and data synchronization functions that support dynamic expansion and contraction. The specific working principles are introduced in detail below.
图4为本申请提供的两维平面基础设施架构示意图，如图4所示，包括控制台meshconsole以及各个业务系统。每个业务系统中包括同步装置。图4中的虚线构成管理平面，实线构成数据平面。在管理平面植入数据签名算法为核心的不可逆安全数据核验技术，在数据平面支持两层核验因子计算。Fig. 4 is a schematic diagram of the two-dimensional planar infrastructure architecture provided by this application, as shown in Fig. 4, including the console meshconsole and various business systems. Each business system includes a synchronization device. The dotted line in Fig. 4 constitutes the management plane, and the solid line constitutes the data plane. The irreversible security data verification technology with data signature algorithm as the core is embedded in the management plane, and the two-layer verification factor calculation is supported in the data plane.
多中心管理模块配置如下：The multi-center management module is configured as follows:
(1)中心注册表和中心状态表。(1) Central registry and central state table.
中心注册表提供对以下信息的配置：中心ID，机器ID，IP，端口PORT。增加中心状态表MACHINE_ST，并支持对中心线路状态的主动探测，中心状态表可以记录各中心之间的连接状态。The center registry provides the configuration of the following information: center ID, machine ID, IP, port PORT. Increase the central state table MACHINE_ST, and support the active detection of the central line state, the central state table can record the connection state between the centers.
(2)中心建链和链路选择。(2) Center link building and link selection.
与其他中心的通讯使用HTTP短链接。给每个中心发同步报文时，使用轮询或指定机器等负载均衡策略向中心内一台机器发送数据同步请求。Communication with other centers uses HTTP short links. When sending a synchronization message to each center, use a load balancing strategy such as polling or specifying a machine to send a data synchronization request to a machine in the center.
(3)统一的数据同步API。(3) Unified data synchronization API.
a、状态探测接口：a. Status detection interface:
GET，参数为空，正常返回报文体{code：”0000“，msg：”“}，异常填写具体错误码和原因。GET, the parameter is empty, the message body {code: "0000", msg: ""} is returned normally, and the specific error code and reason are filled in for exceptions.
b、中心标识查询接口：b. Center logo query interface:
GET，参数为空，正常返回报文体{code：”0000“，msg：”“，centerIn：”master“}，异常同上。GET, the parameter is empty, the message body {code: "0000", msg: "", centerIn: "master"} is returned normally, and the exception is the same as above.
c、数据同步接收接口：c. Data synchronization receiving interface:
POST，请求报文体消息包括{eventid：xxx，dataTaskId：xxx，params：LISTMAP}，多媒体附件格式上送数据文件，应答同a，正常应答仅表示已接收。POST, the request body message includes {eventid: xxx, dataTaskId: xxx, params: LISTMAP}, the data file is sent in multimedia attachment format, the response is the same as a, and the normal response only means that it has been received.
d、异地中心应答接口：d. Remote center response interface:
POST，请求报文体消息包括{code：”0000“，msg：”“，eventid：xxx，dataTaskId：xxx}，应答同a。POST, the request body message includes {code: "0000", msg: "", eventid: xxx, dataTaskId: xxx}, and the response is the same as a.
数据同步场次调度如下：The schedule of data synchronization sessions is as follows:
场次使用作业调度系统调度，具体场次执行条件需要按照以下要求处理：The session is scheduled using the job scheduling system, and the specific execution conditions of the session need to be handled according to the following requirements:
a、依赖的业务场次已经执行成功。a. The dependent business sessions have been executed successfully.
b、对应的业务数据已经执行完毕。b. The corresponding business data has been executed.
c、当前划付日期对应的数据同步任务均已成功(用于限制重做等异常场景，避免同类数据同步冲突)。c. The data synchronization tasks corresponding to the current transfer date have all succeeded (used to limit abnormal scenarios such as redoing and avoid synchronization conflicts of similar data).
每个场次的功能在本质上均为数据任务的调用。The function of each scene is essentially the call of the data task.
a、通过数据同步任务配置定义数据同步内容。a. Define data synchronization content through data synchronization task configuration.
b、填写对应任务需要的参数。b. Fill in the parameters required by the corresponding task.
c、调用本提案所述装置发起数据同步。c. Call the device described in this proposal to initiate data synchronization.
数据同步模块配置如下：The configuration of the data synchronization module is as follows:
图5为本申请提供的数据同步原理架构图。FIG. 5 is a schematic diagram of the data synchronization principle provided by the present application.
(1)任务定义如下：(1) The task is defined as follows:
a、任务ID：开发分配，仅用于展示。a. Task ID: Assigned for development, only for display.
b、任务描述：开发定义，仅用于展示。b. Task description: development definition, for display only.
c、任务类型：数据传输、文件传输。c. Task type: data transfer, file transfer.
d、数据配置：[{数据库，实体名，表名，条件：{[{字段，类型，操作符，值}...]，SQL_WHERE}},......]说明：支持变量占位，发起时填入实际值，装置统一根据配置的主键组装成条件填入参数。d. Data configuration: [{database, entity name, table name, condition: {[{field, type, operator, value}...], SQL_WHERE}},...] Description: support variable account Bit, fill in the actual value when starting, and the device will assemble the condition and fill in the parameters according to the configured primary key.
e、金融字段列配置，使用时对于需要导出的目标表，按照d项配置的条件将金融字段列专门按照设定阈值范围内的数据条数导出，设定阈值范围例如是5000条至6000条，5000条至7000条等。汇总后整体使用SHA256哈希算法计算数据摘要值，并将该数据摘要值与对应的数据表通过哈希映射存放在数据同步请求信息中，发送给备中心，金融字段列属于必配置的属性。对于非金融类数据的同步，可以指定时间信息、状态信息等常发生变化字段达到相同的效果。e. Financial field column configuration. When using the target table that needs to be exported, the financial field column is exported according to the number of data within the set threshold range according to the conditions configured in item d. The set threshold range is, for example, 5000 to 6000 , 5000 to 7000 and so on. After summarizing, use the SHA256 hash algorithm to calculate the data summary value as a whole, and store the data summary value and the corresponding data table in the data synchronization request information through hash mapping, and send it to the backup center. The financial field column is a mandatory attribute. For the synchronization of non-financial data, frequently changing fields such as time information and status information can be specified to achieve the same effect.
(2)数据导出和发起同步如下：(2) Data export and synchronization are initiated as follows:
a、导出数据先生成文件落地。a. The exported data is first generated as a file and landed.
d、数据导出数据有两种方式：配置条件字段、或条件SQL。d. There are two ways to export data: configure conditional fields, or conditional SQL.
c、生成安全数据摘要。对于存在依赖的数据同步任务，等待依赖任务完成，在本次数据同步开始前，先按照将金融字段列专门按照设定阈值范围内的数据条数导出，汇总后整体使用SHA256哈希算法计算数据摘要值，作为摘要因子A，再计算本次新增数据的数据摘要，作为摘要因子B，并将两次计算的摘要值为摘要对放置在数据同步请求报文中；对于单纯无依赖的数据同步任务，将金融字段列专门按照设定阈值范围内的数据条数导出，汇总后整体使用SHA256哈希算法计算数据摘要值，并将该摘要值B与空摘要因子A_EMPTY配为一对，与对应的数据文件通过哈希映射存放在数据同步请求信息中，发送给备中心。c. Generate a safety data summary. For data synchronization tasks that have dependencies, wait for the completion of the dependent tasks. Before the start of this data synchronization, first export the financial field column according to the number of data items within the set threshold range, and use the SHA256 hash algorithm to calculate the data as a whole after summarizing The summary value is used as the summary factor A, and then the data summary of the newly added data is calculated as the summary factor B, and the summary value of the two calculations is placed in the summary pair in the data synchronization request message; for purely independent data Synchronize tasks, export the financial field column according to the number of data items within the set threshold range, and use the SHA256 hash algorithm to calculate the data summary value after summarization, and pair the summary value B with the empty summary factor A_EMPTY, and The corresponding data files are stored in the data synchronization request information through hash mapping and sent to the backup center.
(3)数据导入如下：(3) The data is imported as follows:
a、接收到导入文件后先将导入文件落地。对于存在依赖任务的数据同步任务，在导入前，先按照导入条件计算存量的数据信息摘要。如果数据为空，摘要因子设置A_EMPTY无需重复计算，如果存在数据，则计算摘要值A’。a. After receiving the import file, put the import file to the ground first. For data synchronization tasks with dependent tasks, before importing, calculate the summary of the stock data information according to the import conditions. If the data is empty, the summary factor setting A_EMPTY does not need to be recalculated, and if there is data, the summary value A' is calculated.
b、备中心对数据同步包容度：可配置依赖条件，满足条件使用数据覆盖。b. The tolerance of the backup center for data synchronization: the dependency conditions can be configured, and the data coverage can be used if the conditions are met.
c、导入前对被覆盖的数据做备份。备份完成后先删除，再插入同步数据。c. Make a backup of the overwritten data before importing. After the backup is complete, delete first, and then insert the synchronized data.
d、导入与数据摘要因子计算：为避免由于同步数据量过大，不采用批量提交的“大事务”方式导入数据，每条数据的导入均自动提交入库，如果在备份的导入环节出现异常，则使用备份的数据回退当前导入操作的数据，实现应用层面的数据一致性。导入完成后按照相同方法计算导入增量数据的数据信息摘要因子B’。d. Import and data summary factor calculation: In order to avoid the large amount of synchronized data, the "big transaction" method of batch submission is not used to import data. The import of each piece of data is automatically submitted to the database. If there is an exception in the import link of the backup , use the backup data to roll back the data of the current import operation to achieve data consistency at the application level. After the import is completed, calculate the data information summary factor B' of the imported incremental data in the same way.
e、安全核验：将接收侧计算的摘要因子A’和B’与报文中的A、B做比较，仅当二者同时相等时，视为数据一致。如果A与A’一致但，B不等于B’，则说明同步的数据存在问题，回退当前导入的全部数据，并将错误信息反馈给运维人员。如果A与A’不一致但B等于B’，说明数据同步的一致，但依赖的同步任务存在故障，回退当前导入的全部数据，并将错误信息反馈给运维人员。e. Security verification: compare the digest factors A' and B' calculated by the receiving side with A and B in the message, and only when the two are equal at the same time, the data is considered to be consistent. If A is consistent with A' but B is not equal to B', it means that there is a problem with the synchronized data, and all currently imported data will be rolled back, and the error information will be fed back to the operation and maintenance personnel. If A is inconsistent with A' but B is equal to B', it means that the data synchronization is consistent, but there is a failure in the dependent synchronization task, and all currently imported data will be rolled back, and the error information will be fed back to the operation and maintenance personnel.
图6为本申请提供的数据同步流程图，如图6所示，一次数据同步包括5个具体的数据同步步骤。从前往后分别是，对于数据同步的两侧，分别装载各自的技术参数，数据同步发起方(主中心)等待业务上的操作完成数据库操作，完成后数据同步装置将当前操作的增量数据生成数据文件，以及数据安全核验摘要信息，完成后通过HTTP协议将数据文件发送给数据同步的数据同步接收方(备中心)，数据同步接收方接收到请求后先将数据文件落地，完成数据安全核验后，开始具体的导入操作，最终将处理结果反馈给数据同步发起方。FIG. 6 is a flow chart of data synchronization provided by this application. As shown in FIG. 6 , one data synchronization includes five specific data synchronization steps. From front to back, respectively, for both sides of data synchronization, the respective technical parameters are loaded, and the data synchronization initiator (main center) waits for the operation on the business to complete the database operation. After completion, the data synchronization device will generate the incremental data of the current operation After the data file and the summary information of the data security verification are completed, the data file is sent to the data synchronization receiver (backup center) of the data synchronization through the HTTP protocol. After receiving the request, the data synchronization receiver first lands the data file and completes the data security verification After that, start the specific import operation, and finally feed back the processing result to the data synchronization initiator.
图7为本申请提供的云上多应用数据同步装置结构示意图，该装置包括：FIG. 7 is a schematic structural diagram of a multi-application data synchronization device on the cloud provided by the present application, which includes:
发送模块71，用于获取待同步的数据表，通过哈希算法确定所述数据表的第一数据摘要值，将所述第一数据摘要值和所述数据表存放在数据同步请求信息中，将所述数据同步请求信息发送至数据同步接收方；The sending
接收模块72，用于接收所述数据同步请求信息，获取其中携带的所述第一数据摘要值和数据表，通过哈希算法确定接收到的数据表的第二数据摘要值，判断所述第一数据摘要值和所述第二数据摘要值是否一致，如果是，对所述数据表进行导入操作。The receiving
所述发送模块71，具体用于判断当前数据同步的业务场景是否为依赖业务场景，如果是，获取所述待同步的数据表对应的依赖数据表，通过哈希算法确定所述数据表的第一数据摘要值，以及所述依赖数据表的第三数据摘要值，将所述第一数据摘要值、第三数据摘要值和所述数据表存放在数据同步请求信息中。The sending
所述接收模块72，具体用于所述数据同步接收方接收所述同步请求信息，获取其中携带的所述第一数据摘要值、第三数据摘要值和数据表；根据预先配置的导入条件，确定与所述数据表对应的已导入的依赖数据表；通过哈希算法确定接收到的数据表的第二数据摘要值，以及所述已导入的依赖数据表的第四数据摘要值；判断所述第一数据摘要值和所述第二数据摘要值是否一致，并且所述第三数据摘要值和所述第四数据摘要值是否一致，如果均一致，对所述数据表进行导入操作。The receiving
所述接收模块72，还用于判断已导入数据中是否存在与所述数据表对应的被覆盖数据，如果是，对所述被覆盖数据进行备份，并保存至指定地址，将已导入数据中的所述被覆盖数据进行删除，对所述数据表进行导入操作。The receiving
所述装置还包括：The device also includes:
第一反馈模块73，用于对所述数据表进行导入操作之后，向所述数据同步发起方反馈数据已同步的提示信息。The
所述装置还包括：The device also includes:
第二反馈模块74，用于若判断所述第一数据摘要值和所述第二数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送同步数据异常的提示信息。The
所述装置还包括：The device also includes:
第三反馈模块75，用于若判断所述第三数据摘要值和所述第四数据摘要值不一致，回退所述数据表至所述数据同步发起方，并向所述数据同步发起方的管理人员发送依赖的同步任务异常的提示信息。The
本申请还提供了一种服务器，如图8所示，包括：处理器301、通信接口302、存储器303和通信总线304，其中，处理器301，通信接口302，存储器303通过通信总线304完成相互间的通信；The present application also provides a server, as shown in FIG. 8 , including: a
所述存储器303中存储有计算机程序，当所述程序被所述处理器301执行时，使得所述处理器301执行以上任一方法步骤。A computer program is stored in the
上述服务器提到的通信总线可以是外设部件互连标准(Peripheral ComponentInterconnect，PCI)总线或扩展工业标准结构(Extended Industry StandardArchitecture，EISA)总线等。该通信总线可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。The communication bus mentioned in the above server may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
通信接口302用于上述服务器与其他设备之间的通信。The
存储器可以包括随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器(Non-Volatile Memory，NVM)，例如至少一个磁盘存储器。可选地，存储器还可以是至少一个位于远离前述处理器的存储装置。The memory may include a random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
上述处理器可以是通用处理器，包括中央处理器、网络处理器(NetworkProcessor，NP)等；还可以是数字信号处理器(Digital Signal Processing，DSP)、专用集成电路、现场可编程门陈列或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。Above-mentioned processor can be general-purpose processor, comprises central processing unit, network processor (NetworkProcessor, NP) etc.; It can also be digital signal processor (Digital Signal Processing, DSP), application-specific integrated circuit, field programmable gate display or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
本申请还提供了一种计算机存储可读存储介质，所述计算机可读存储介质内存储有可由服务器执行的计算机程序，当所述程序在所述服务器上运行时，使得所述服务器执行时实现以上任一方法步骤。The present application also provides a computer-readable storage medium, wherein a computer program executable by a server is stored in the computer-readable storage medium, and when the program is run on the server, the server is executed to realize any of the above method steps.
尽管已描述了本申请的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本申请范围的所有变更和修改。While preferred embodiments of the present application have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is appreciated. Therefore, the appended claims are intended to be construed to cover the preferred embodiment and all changes and modifications which fall within the scope of the application.
显然，本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样，倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内，则本申请也意图包含这些改动和变型在内。Obviously, those skilled in the art can make various changes and modifications to the application without departing from the spirit and scope of the application. In this way, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.
Claims (16)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211345921.2ACN115643271B (en) | 2022-10-31 | 2022-10-31 | A method, device, server and medium for synchronizing multi-application data on the cloud |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211345921.2ACN115643271B (en) | 2022-10-31 | 2022-10-31 | A method, device, server and medium for synchronizing multi-application data on the cloud |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115643271Atrue CN115643271A (en) | 2023-01-24 |
| CN115643271B CN115643271B (en) | 2025-09-23 |
Family
ID=84946774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211345921.2AActiveCN115643271B (en) | 2022-10-31 | 2022-10-31 | A method, device, server and medium for synchronizing multi-application data on the cloud |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115643271B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116738507A (en)* | 2023-08-14 | 2023-09-12 | 中科亿海微电子科技（苏州）有限公司 | Chip authentication method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108428141A (en)* | 2018-03-15 | 2018-08-21 | 华中农业大学 | A kind of foodstuff traceability information management system based on ERP system Yu block chain |
| CN109815291A (en)* | 2018-12-28 | 2019-05-28 | 北京城市网邻信息技术有限公司 | Method of data synchronization, device, electronic equipment and storage medium |
| CN110289947A (en)* | 2019-04-29 | 2019-09-27 | 北京开态智慧科技有限公司 | Data transmit consistency desired result method, apparatus, computer equipment and storage medium |
| CN112231367A (en)* | 2020-12-14 | 2021-01-15 | 腾讯科技（深圳）有限公司 | Business processing method and device based on artificial intelligence and electronic equipment |
| CN114116743A (en)* | 2021-11-23 | 2022-03-01 | 南京星云数字技术有限公司 | Data verification method and device, computer equipment and storage medium |
- 2022
- 2022-10-31CNCN202211345921.2Apatent/CN115643271B/enactiveActive
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108428141A (en)* | 2018-03-15 | 2018-08-21 | 华中农业大学 | A kind of foodstuff traceability information management system based on ERP system Yu block chain |
| CN109815291A (en)* | 2018-12-28 | 2019-05-28 | 北京城市网邻信息技术有限公司 | Method of data synchronization, device, electronic equipment and storage medium |
| CN110289947A (en)* | 2019-04-29 | 2019-09-27 | 北京开态智慧科技有限公司 | Data transmit consistency desired result method, apparatus, computer equipment and storage medium |
| CN112231367A (en)* | 2020-12-14 | 2021-01-15 | 腾讯科技（深圳）有限公司 | Business processing method and device based on artificial intelligence and electronic equipment |
| CN114116743A (en)* | 2021-11-23 | 2022-03-01 | 南京星云数字技术有限公司 | Data verification method and device, computer equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116738507A (en)* | 2023-08-14 | 2023-09-12 | 中科亿海微电子科技（苏州）有限公司 | Chip authentication method |
| CN116738507B (en)* | 2023-08-14 | 2023-11-10 | 中科亿海微电子科技（苏州）有限公司 | Chip authentication method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115643271B (en) | 2025-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10255343B2 (en) | Initialization protocol for a peer-to-peer replication environment | |
| CN106899648B (en) | A data processing method and device | |
| WO2019061720A1 (en) | Data synchronization method and system | |
| CN110096340B (en) | Timing task processing method and device | |
| WO2022134233A1 (en) | Consensus method and apparatus for blockchain, server and storage medium | |
| CN112468310B (en) | Streaming media cluster node management method and device and storage medium | |
| CN107682460A (en) | A kind of distributed storage trunked data communication method and system | |
| US20150186485A1 (en) | Robust communication system for guaranteed message sequencing with the detection of duplicate senders | |
| CN114900449B (en) | Resource information management method, system and device | |
| US20250036528A1 (en) | Systems and methods for failure recovery in at-most-once and exactly-once streaming data processing | |
| CN111680015A (en) | File resource processing method, apparatus, device and medium | |
| CN105373563B (en) | Database switching method and device | |
| CN108924008A (en) | A kind of dual controller data communications method, device, equipment and readable storage medium storing program for executing | |
| CN111259022A (en) | Information synchronization method, synchronization system, computer equipment and medium | |
| CN118796932A (en) | Data synchronization method, device, equipment and storage medium | |
| CN106033422A (en) | A database operation control method, device and system | |
| CN115643271A (en) | Method, device, server and medium for synchronizing multi-application data on cloud | |
| CN116028578A (en) | Block chain data synchronization method, device, electronic equipment and readable storage medium | |
| CN115455121A (en) | Real-time reliable data synchronous transmission method, equipment and medium | |
| CN112187916B (en) | A cross-system data synchronization method and device | |
| CN115426361A (en) | Distributed client packaging method and device, main server and storage medium | |
| CN118646758A (en) | A system and method for realizing remote data synchronization based on data monitoring | |
| CN116842090A (en) | Accounting system, method, equipment and storage medium | |
| CN113127219A (en) | Data storage method, data acquisition device and storage medium | |
| CN114553928A (en) | Business processing method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |