Background
Today, the requirement for system stability and high availability is becoming higher and higher, and especially in the field of financial business, it is becoming more important to be able to provide stable service capability to the outside. Therefore, each system realizes a multi-center deployment architecture so as to improve the stability and disaster tolerance of the system and ensure high availability.
In the prior art, when each central system receives a transaction request, data processing of the central system is performed first, and after the data processing is completed, result data is synchronized to other centers, and the result data is stored by the other centers. In the data synchronization process in the prior art, data synchronization is only performed on a database, the synchronization requirement on a service layer cannot be met, and the real-time consistency of data among centers cannot be ensured.
Disclosure of Invention
The embodiment of the application provides a cross-system data synchronization method and device, so as to meet the consistency of data processing among service layers of different systems.
In one aspect, an embodiment of the present application provides a cross-system data synchronization method, where the method includes:
a first server acquires a first synchronization message from a message queue of a second server, wherein the first server and the second server are servers in different systems, the first synchronization message comprises first synchronization data, and first processing result data of the first synchronization data is stored in the second server;
the first server analyzes the first synchronization message to obtain first synchronization data;
the first server takes a target object comprising first synchronization data of the first server as target synchronization data;
and the first server processes the target synchronous data, and determines and stores first processing result data of the target synchronous data.
Optionally, the obtaining, by the first server, the first synchronization packet from the message queue of the second server includes:
the first server receives the synchronous broadcast sent by the second server;
the first server sends a synchronous data acquisition request to the second server according to the synchronous broadcast;
and the first server receives the first synchronous message sent by the second server.
Optionally, after the first server parses the target synchronization packet, the method further includes:
if the first server fails to analyze the target synchronous message, caching the target synchronous message;
after the first server processes the target synchronization data, the method further includes:
and if the first server fails to process the target synchronous data, caching the target synchronous data.
Optionally, the processing, by the first server, the target synchronization data includes:
and the first server adds a synchronization identifier in the target synchronization data.
Optionally, the method further includes:
the first server receives a data synchronization request, wherein the data synchronization request comprises second synchronization data;
the first server processes the second synchronous data, and determines and stores second processing result data of the second synchronous data;
the first server generates a second synchronous message according to the second synchronous data;
and the first server adds the second synchronous message into a message queue of the first server.
On the other hand, an embodiment of the present invention further provides a cross-system data synchronization apparatus, where the apparatus includes:
the system comprises a receiving and sending unit, a processing unit and a processing unit, wherein the receiving and sending unit is used for acquiring a first synchronization message from a message queue of a second server, the first server and the second server are servers in different systems, the first synchronization message comprises first synchronization data, and first processing result data of the first synchronization data is stored in the second server;
the analysis unit is used for analyzing the first synchronization message to obtain first synchronization data;
a judging unit, configured to use a target object including first synchronization data of the first server as target synchronization data;
and the processing unit is used for processing the target synchronous data and determining and storing first processing result data of the target synchronous data.
Optionally, the transceiver unit is specifically configured to:
receiving the synchronous broadcast sent by the second server;
sending a synchronous data acquisition request to the second server according to the synchronous broadcast;
and receiving the first synchronous message sent by the second server.
Optionally, the apparatus further includes a cache unit, configured to:
if the target synchronous message is failed to be analyzed, caching the target synchronous message;
and if the target synchronous data is failed to be processed, caching the target synchronous data.
Optionally, the processing unit is specifically configured to:
and adding a synchronization identifier in the target synchronization data.
Optionally, the transceiver unit is further configured to receive a data synchronization request, where the data synchronization request includes second synchronization data;
the processing unit is further configured to process the second synchronization data, and determine and store second processing result data of the second synchronization data; generating a second synchronous message according to the second synchronous data; and adding the second synchronous message into a message queue of the first server.
In another aspect, the present application provides a computing device, including at least one processor and at least one memory, where the memory stores a computer program, and when the program is executed by the processor, the processor is caused to execute the steps of the cross-system data synchronization method provided by the present application.
In another aspect, an embodiment of the present application provides a storage medium, where the storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is caused to perform the steps of the cross-system data synchronization method provided by the embodiment of the present application.
The embodiment of the invention relates to two or more systems, and each system comprises at least one server. The first server and the second server are servers in different systems. And after receiving the first synchronization data, the second server processes the first synchronization data to obtain first processing result data, stores the first processing result data, generates a first synchronization message according to the first synchronization data and caches the first synchronization message in a message queue. The first server obtains a first synchronous message from the message queue of the second server, and analyzes the first synchronous message to obtain first synchronous data. And the target object comprises the first synchronization data of the first server as the target synchronization data of the first server. And processing the target synchronous data, and determining and storing first processing result data of the target synchronous data. In the embodiment of the invention, the second server directly synchronizes unprocessed original data, namely the first synchronization data, and directly delivers the first synchronization data to the service layer of the first server for data processing by means of the message middleware, so that the data synchronization requirement between the service layer of the first server and the service layer of the second server is ensured, and the real-time consistency of data processing is ensured.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments, but not all embodiments, of the technical solutions of the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments described in the present application without any creative effort belong to the protection scope of the technical solution of the present application.
The terms "first" and "second" in the description and claims of the present invention and the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the term "comprises" and any variations thereof, which are intended to cover non-exclusive protection. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The following briefly introduces some structures of the technical solutions of the embodiments of the present application, and it should be noted that the structures described below are only used for illustrating the embodiments of the present application and are not limited. In specific implementation, the technical scheme provided by the embodiment of the application can be flexibly applied according to actual needs.
To further illustrate the technical solutions provided by the embodiments of the present application, the following detailed description is made with reference to the accompanying drawings and the detailed description. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or figures, more or less operation steps may be included in the method based on the conventional or non-inventive labor. In steps where no necessary causal relationship exists logically, the order of execution of the steps is not limited to that provided by the embodiments of the present application.
Reference is made to fig. 1, which is a system architecture diagram of a cross-system data synchronization method provided by an embodiment of the present application. The embodiment of the invention provides a data processing method. As shown in fig. 1, a system architecture applicable to the embodiment of the present invention includes a plurality of central systems, where each central system includes a plurality of servers, each server may be a network device such as a computer, and each server may be an independent device or a server cluster formed by a plurality of servers. Preferably, the server can perform information processing by using a cloud computing technology. The communication between the databases and the center, and between the centers can be performed through INTERNET network, and also can be performed through Mobile communication systems such as Global System for Mobile Communications (GSM), Long Term Evolution (LTE) systems, and the like.
Specifically, the first server in the embodiment of the present invention may be a server of any central system in fig. 1, and the corresponding second server is a server in any other central system except the central system where the first server is located. Each central system in fig. 1 may synchronize data of the other central systems by using the data synchronization method in the embodiment of the present invention, so as to satisfy data consistency between service layers and data layers of different central systems.
In an alternative embodiment, the server of each central system in fig. 1 is a three-layer architecture, that is, each functional module is divided into three layers, namely, an interface layer (UI), a Business Logic Layer (BLL) and a Data Access Layer (DAL), the layers access each other by using interfaces, and an entity class (Model) of an object Model is used as a carrier for data transfer, the entity classes of different object models generally correspond to different tables of a database, and attributes of the entity classes are consistent with field names of the database table.
Fig. 2 shows a schematic structure of a three-layer architecture. Wherein, the main functions and the service logic of the system are processed in the service logic layer. The data access layer is mainly used for operating a non-original data operation layer, namely a data storage form such as a database or a text file, and particularly provides data service for a business logic layer or an interface layer. The service logic layer mainly operates for specific problems, and can also be understood as an operation on the data layer to process data service logic. The interface layer is mainly expressed in a WEB mode or a WINFORM mode, mainly receives the request of the user and data feedback, provides the access of the application program for the client and provides an interface for the interactive operation for the user.
Specifically, in the embodiment of the present invention, a tag (also referred to as "Topic") is newly created in the message middleware of each server of the central system, and is specifically used for storing data that needs to be synchronized across centers in the center. When the data in the center needs to be synchronized across centers, the data only needs to be cached to the synchronization Topic across centers of the message middleware. Other central systems may subscribe to the central system's synchronized Topic consuming synchronized data.
Each central system is newly provided with one or more data issuing services for packaging and issuing data to be synchronized according to a synchronization protocol, and if the issuing fails, the central system needs to locally record original data and has secondary processing capability of the local original data. When the data is issued, the source of the data to be synchronized needs to be judged, if the data comes from other centers, the data is not processed, and the data cycle synchronization is avoided.
Each center is added with one (or more) synchronous data subscription service(s) for subscribing and consuming the cross-center synchronous Topic data of other centers. When the synchronous Topic of other centers has data, the current center can directly acquire the synchronous data through the subscription service and process the synchronous data, so that the data can be circulated across the centers.
When the current center processes synchronous data, a synchronous mark needs to be added in the data to mark a data source as other centers, so as to avoid data cycle synchronization; if the processing fails, it is still ensured that the consumption progress of the cross-center synchronous data is not influenced, so as to avoid influencing the center operation of the issued data. For this reason, data that fails processing needs to be cached locally and has the ability to initiate processing again.
Of course, the architecture provided by the embodiment of the present application is not limited to the structures shown in fig. 1 and fig. 2, and the embodiment of the present application is not limited thereto. To further illustrate the technical solutions provided by the embodiments of the present application, the following detailed description is made with reference to the accompanying drawings and the detailed description. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or figures, more or less operation steps may be included in the method based on the conventional or non-inventive labor. In steps where no necessary causal relationship exists logically, the order of execution of the steps is not limited to that provided by the embodiments of the present application.
The cross-center synchronization method in the prior art only performs synchronization on a database level. However, since the synchronous data is not only stored in the database, but also needs to be refreshed on the service layer, the traditional scheme of performing data synchronization only on the database cannot solve the problem of refreshing the service layer of other centers, and if the database and the service layer are synchronized simultaneously, although the problem of central synchronization can be solved, the resource overhead such as cross-central data bandwidth is too large due to two synchronization rules, and once there is a synchronization problem, the problem of inconsistency between the database and the service layer in the center is easily caused.
In order to solve the above problem, an embodiment of the present invention provides a cross-system data synchronization method. Referring to fig. 3, a flowchart of a cross-system data synchronization method according to an embodiment of the present invention is shown, where the method can be applied to the scenario shown in fig. 1, and the flow of the method is described as follows.
Step 301, the first server obtains a first synchronization packet from a message queue of the second server.
The first server and the second server are servers in different systems, the first synchronization message contains first synchronization data, and the second server stores first processing result data of the first synchronization data.
Step 302, the first server analyzes the first synchronization packet to obtain first synchronization data.
Step 303, the first server uses the first synchronization data of the first server included in the target object as the target synchronization data.
Step 304, the first server processes the target synchronous data, and determines and stores first processing result data of the target synchronous data.
The embodiment of the invention relates to two or more systems, and each system comprises at least one server. The first server and the second server are servers in different systems. And after receiving the first synchronization data, the second server processes the first synchronization data to obtain first processing result data, stores the first processing result data, generates a first synchronization message according to the first synchronization data and caches the first synchronization message in a message queue. The first server obtains a first synchronous message from the message queue of the second server, and analyzes the first synchronous message to obtain first synchronous data. And the target object comprises the first synchronization data of the first server as the target synchronization data of the first server. And processing the target synchronous data, and determining and storing first processing result data of the target synchronous data. In the embodiment of the invention, the second server directly synchronizes unprocessed original data, namely the first synchronization data, and directly delivers the first synchronization data to the service layer of the first server for data processing by means of the message middleware, so that the data synchronization requirement between the service layer of the first server and the service layer of the second server is ensured, and the real-time consistency of data processing is ensured.
In the embodiment of the present invention, the server of each central system may be used as the first server or the second server. Namely, the server of each central system can be used as a distributor of the synchronous data and also can be used as a receiver of the synchronous data.
When the server is used as a publisher of the synchronized data, the data synchronization method in the embodiment of the present invention further includes:
a first server receives a data synchronization request, wherein the data synchronization request comprises second synchronization data;
the first server processes the second synchronous data, and determines and stores second processing result data of the second synchronous data;
the first server generates a second synchronous message according to the second synchronous data;
and the first server adds the second synchronous message into a message queue of the first server.
In a specific implementation process, after receiving the data synchronization request, the first server processes second synchronization data in the data synchronization request, and after the second synchronization data is processed, second processing result data of the second synchronization data is stored in the database. The first server generates a second synchronization message from the original data, namely the second synchronization data.
The specific sync message is generated according to a data sync protocol, which specifies the format of the sync message as shown in fig. 4. Besides the synchronous data, a synchronous message also has contents such as synchronous protocol version number, target service name, synchronous initiator and target, data type, data length and the like. The first server generates a target processing service name according to the service scene requirement, selects servers in all central systems or appointed central systems as target parties, and encapsulates second synchronous data according to a data synchronization protocol to output a second synchronous message.
And after the first server generates a second synchronous message according to the second synchronous data, adding the second synchronous message into the message middleware of the first server. The servers of the other centers may subscribe to the second synchronization data in the message middleware of the first server.
Further, the step of acquiring, by the first server, the first synchronization packet from the message queue of the second server includes:
the first server receives the synchronous broadcast sent by the second server;
the first server sends a synchronous data acquisition request to the second server according to the synchronous broadcast;
and the first server receives the first synchronous message sent by the second server.
In a specific implementation process, the manner in which the first server obtains the first synchronization packet in the embodiment of the present invention may be that the second server broadcasts to the servers of all the other central systems. The servers of the other central systems serve as first servers to receive the synchronous broadcast sent by the second server. The first server sends a synchronous data acquisition request to the second server according to the synchronous broadcast and receives a first synchronous message sent by the second server.
Another way to obtain the first synchronization packet may be that the first server actively sends a synchronization data obtaining request to the second server. Specifically, the first server sends a synchronous data acquisition request to the second server according to a certain frequency. And after receiving the synchronous data acquisition request, the second server sends the first synchronous message in the message middleware to the first server. The manner in which the first server obtains the first synchronization packet in the embodiment of the present invention is merely an example, and is not limited.
Further, after the first server analyzes the target synchronization packet, the method further includes:
if the first server fails to analyze the target synchronous message, caching the target synchronous message;
after the first server processes the target synchronization data, the method further includes:
and if the first server fails to process the target synchronous data, caching the target synchronous data.
In a specific implementation process, after receiving the first synchronization message, the first server analyzes the first synchronization message, and if the first synchronization message is successfully analyzed, the first synchronization data in the first synchronization message is processed. And judging whether the target object of the first synchronous message comprises the first server or not according to the content of the first synchronous message. And if not, the first server deletes the first synchronization data, which indicates that the first server does not need to synchronize the synchronization data in the first synchronization message.
On the other hand, if the first server fails to analyze the first synchronous message, the first synchronous message is cached first to be processed uniformly subsequently in order to prevent data loss. In addition, if the first server fails to process the target synchronous data, the target synchronous data is cached to wait for subsequent unified processing, so that the consumption progress of the cross-center synchronous data is not influenced, and the central operation of a second server issuing the synchronous data is not influenced.
Further, the processing of the target synchronization data by the first server includes:
and the first server adds a synchronization identifier in the target synchronization data.
In a specific implementation process, when the first server processes the target synchronous data, a synchronous identifier needs to be added to the target synchronous data to identify that a source of the synchronous data is other center heartbeats, so that data cycle synchronization is avoided.
The above-described process is described in detail below with specific examples. The scenario in the specific embodiment is that the merchant online query service system (hereinafter referred to as MOQS) uses the scheme, specifically: the upstream transmits merchant data to MOQS in a real-time RPC calling mode, and the MOQS needs to be stored in a warehouse and then loaded to redis. Since the MOQS is a multi-center deployment, in order to make the caller don't care about the deployment inside the multi-center, the data synchronization method in the embodiment of the present invention is adopted in the MOQS. When a service processing request is received, data processing of the central system is firstly carried out, after the data are successfully processed (warehousing + Redis loading), the data are synchronized to other central systems through the message middleware, and the other central systems carry out data processing. The specific process comprises the following steps:
the message queue of each central system is added with a cross-central Topic (label), and a new synchronous subscription service is added to subscribe the cross-central Topic.
After receiving the real-time synchronization request, the single center system performs data processing of the center system; and after the processing is finished, transmitting the data and the synchronization parameters to a synchronous publishing service, wherein the synchronized data in the scene is original real-time synchronization request data.
The synchronous data publishing service checks the synchronous data, packages the data which are not from other central system synchronization according to the cross-central protocol convention format, and publishes the data to the message queue of the central system.
And the synchronous subscription services of the other central systems acquire the synchronous data of the current central system, unpack the message according to the protocol requirement, and cache the synchronous message in the local file if the unpacking fails. In order to avoid increasing the network overhead of the cross-center system, synchronous data is obtained in batches each time.
And after unpacking, judging a source and a target, processing synchronous data of which the source is not the central system and the target is the central system, and discarding other data.
And taking out the synchronous data from the message, calling according to the target service name in the message, and acquiring a calling result.
If the calling fails, caching the synchronous message into a local file, and after the abnormal condition is processed, performing unified processing; if the calling is successful, the current synchronization message is successful, and the synchronization among the data centers is completed.
And aiming at the transaction which fails to be processed, a manual redo mode is provided, and secondary processing is carried out after the inter-center upgrading window period is closed or the abnormal problem is repaired, so that the failed data compensation is completed.
The following are embodiments of the apparatus of the present application, and for details not described in detail in the embodiments of the apparatus, reference may be made to the above-mentioned one-to-one corresponding method embodiments.
Referring to fig. 5, a block diagram of a cross-system data synchronization apparatus according to an embodiment of the present application is shown. The cross-system data synchronization device is implemented by hardware or a combination of hardware and software to become all or a part of the server in the central system of fig. 1. The device includes:
atransceiving unit 501, configured to obtain a first synchronization packet from a message queue of a second server, where the first server and the second server are servers in different systems, the first synchronization packet includes first synchronization data, and the second server stores first processing result data of the first synchronization data;
an analyzingunit 502, configured to analyze the first synchronization packet to obtain first synchronization data;
a determiningunit 503, configured to use the target object including the first synchronization data of the first server as target synchronization data;
theprocessing unit 504 is configured to process the target synchronization data, and determine and store first processing result data of the target synchronization data.
Optionally, thetransceiver unit 501 is specifically configured to:
receiving the synchronous broadcast sent by the second server;
sending a synchronous data acquisition request to the second server according to the synchronous broadcast;
and receiving the first synchronous message sent by the second server.
Optionally, the apparatus further includes acache unit 505, configured to:
if the target synchronous message is failed to be analyzed, caching the target synchronous message;
and if the target synchronous data is failed to be processed, caching the target synchronous data.
Optionally, theprocessing unit 504 is specifically configured to:
and adding a synchronization identifier in the target synchronization data.
Optionally, thetransceiver unit 501 is further configured to receive a data synchronization request, where the data synchronization request includes second synchronization data;
theprocessing unit 504 is further configured to process the second synchronization data, and determine and store second processing result data of the second synchronization data; generating a second synchronous message according to the second synchronous data; and adding the second synchronous message into a message queue of the first server.
Referring to fig. 6, a block diagram of a server according to an embodiment of the present application is shown. Theserver 1000 is implemented to become a server in the central system of fig. 1. Specifically, the method comprises the following steps:
theserver 1000 includes a Central Processing Unit (CPU)1001, asystem memory 1004 including a Random Access Memory (RAM)1002 and a Read Only Memory (ROM)1003, and asystem bus 1005 connecting thesystem memory 1004 and thecentral processing unit 1001. Theserver 1000 also includes a basic input/output system (I/O system) 1006, which facilitates the transfer of information between devices within the computer, and amass storage device 1007, which stores anoperating system 1013,application programs 1014, and other program modules 1015.
The basic input/output system 1006 includes adisplay 1008 for displaying information and aninput device 1009, such as a mouse, keyboard, etc., for user input of information. Wherein thedisplay 1008 andinput device 1009 are connected to thecentral processing unit 1001 through an input-output controller 1010 connected to thesystem bus 1005. The basic input/output system 1006 may also include an input/output controller 1010 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input-output controller 1010 also provides output to a display screen, a printer, or other type of output device.
Themass storage device 1007 is connected to thecentral processing unit 1001 through a mass storage controller (not shown) connected to thesystem bus 1005. Themass storage device 1007 and its associated computer-readable media provide non-volatile storage for theserver 1000. That is, themass storage device 1007 may include a computer readable medium (not shown) such as a hard disk or CD-ROM drive.
The computer readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the foregoing. Thesystem memory 1004 andmass storage device 1007 described above may be collectively referred to as memory.
Theserver 1000 may also operate as a remote computer connected to a network via a network, such as the internet, in accordance with various embodiments of the present invention. That is, theserver 1000 may be connected to thenetwork 1012 through thenetwork interface unit 1011 connected to thesystem bus 1005, or thenetwork interface unit 1011 may be used to connect to another type of network or a remote computer system (not shown).
The memory also includes one or more programs, which are stored in the memory, the one or more programs including instructions for performing the cross-system data synchronization method provided by the embodiments of the present invention.
It will be understood by those skilled in the art that all or part of the steps in the check-in method of the above embodiments may be implemented by a program instructing associated hardware, and the program may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
Those skilled in the art will appreciate that all or part of the steps in the check-in method of the above embodiments may be implemented by a program instructing associated hardware, and the program may be stored in a computer-readable storage medium, and the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.