CN107368503B - Data synchronization method and system based on button - Google Patents

Abstract

The invention provides a data synchronization method and system based on a button, wherein the method comprises the following steps: configuring data source information and parameter information through a client, and storing the data source configuration information and the parameter configuration information in a database; editing a key file comprising data circulation logic through a client-side key graphical interface; uploading the button file to a button execution engine; dynamically analyzing the button file; and transferring the corresponding data stream of the source data source to the target data source according to the data stream logic and the corresponding parameter configuration information and the data source configuration information. The system comprises a client and a data acquisition platform, wherein the client is used for configuring, editing and uploading the key file; and the data acquisition platform is used for receiving and dynamically analyzing the button file and transferring the corresponding data stream of the source data source to the target data source according to the data source configuration information. The invention can randomly designate the data stream transfer among a plurality of data sources, is rapid and convenient, and has low coupling degree among systems.

Description

Data synchronization method and system based on button

Technical Field

The invention relates to the technical field of data processing, in particular to a data synchronization method and system based on a button.

Background

At present, in most enterprises, with continuous expansion of enterprise services, enterprise systems are continuously increased and updated iteratively, and a lot of associations between the systems are also generated, just like spider webs, which is complicated and complicated. When the existing system can not meet the business requirements of the company, the system has to be designed and developed again. This, however, necessarily faces a very difficult task: how to achieve the consistency of new and old data. The general scheme is: when a new system is designed, the design of an old system is compatible, and the Worker is made to synchronize the data of the old system to the new system at regular time. This solution is certainly possible, but the time consumed for synchronizing data is very large. If a company has thousands of systems, a new system per liter level, the data is resynchronized. It takes too much time. Therefore, in addition to considering compatibility with an old system when designing a new system, it is necessary to find a scheme for completing data synchronization between systems.

At present, the schemes that are used more often adopt Sql statement data synchronization, timing synchronization between application programs, and call by inter-system RPC, for example: WebService, RMI, or RPC framework inside the enterprise, and some Etl tools are used for data conversion locally.

In an enterprise system with complicated complexity, data acquisition is carried out, then Sql sentences are adopted for importing, and the workload is large. If timing synchronization between applications is employed, there is a limit to the spread between applications within an enterprise. If remote RPC calls are used, the corresponding application server is inevitably stressed by the remote calls, and the integration is relatively complex. The Etl tool is used for data conversion, because the on-line database is directly operated, the authority limit and the safety problem are inevitable, and the method also generally does not conform to the specifications of the enterprise process.

Among Etl tools, there is an open source Etl tool named button, which is written in pure Java and has the characteristics of high efficiency and stability in data extraction. It allows managing data of different data sources by providing a graphical interface to implement the functions that the user wants to accomplish.

Disclosure of Invention

The technical problem to be solved by the present invention is to provide a method and a system for data synchronization based on a key for realizing fast data synchronization between systems, aiming at the defects of the prior art.

In order to solve the above technical problem, according to an aspect of the present invention, there is provided a method for data synchronization based on a key, including:

configuring a data source and parameters through a client, and storing data source configuration information and parameter configuration information in a database;

editing a key file comprising data circulation logic through a client-side key graphical interface;

uploading the button file to a button execution engine;

dynamically analyzing the button file; and

and transferring the corresponding data stream of the source data source to the target data source according to the data stream logic and the corresponding parameter configuration information and the data source configuration information.

Preferably, the data source configuration information includes a data source identification, a data source type, and an IP or URL link address of the data source; and/or the parameters comprise dynamic parameters required in data circulation.

Preferably, the step of dynamically parsing the button file includes:

dynamically loading the button file to generate a corresponding conversion object;

and assigning the data source information and the parameter information to the conversion object according to the data source configuration information and the parameter configuration information.

Preferably, the conversion object includes data source attribute information, parameter attribute information, and node attribute information;

the step of assigning the data source information to the conversion object comprises:

acquiring corresponding source data source information and target data source information in the data source configuration information according to the data source identification in the data source attribute information;

modifying the data source attribute information according to the source data source information and the target data source information;

the step of assigning the parameter information to the conversion object includes:

and searching the parameter configuration information according to the parameters needing to be assigned in the parameter attribute information, and modifying the parameter attribute information according to the searched specific parameters.

Preferably, the data source configuration information is recorded in a data source configuration table; and/or the parameter configuration information is recorded in a parameter table.

Preferably, the data source types include Orale, MySql, SQLServer, WebService, and SAP types.

Preferably, before uploading the button file to the button execution engine, the button file is tested locally, and completeness and accuracy of the acquired data are verified.

Preferably, after the corresponding data stream of the source data source is transferred to the target data source, the method further comprises the step of returning the execution information to the client.

In order to solve the above technical problem, according to another aspect of the present invention, there is provided a key-based data synchronization system, including:

the client is used for configuring, editing and uploading the key file, wherein the data source configuration information and the related parameter configuration information are stored in the database; and

and the data acquisition platform is used for receiving and dynamically analyzing the button file, and transferring the corresponding data stream of the source data source to the target data source according to the data stream logic and the corresponding parameter information and the data source information.

Preferably, the client includes:

the configuration module is used for configuring data source information and parameter information and storing the data source configuration information and related parameter configuration information in a database; and

and the editing module is used for editing the key file comprising the data circulation logic by providing a key graphical interface.

Preferably, the data acquisition platform comprises:

the key execution engine is used for dynamically analyzing the key file and completing data circulation;

the multiple APIs are used for the button execution engine to call and complete the analysis of the button file and the data circulation; and

and the database is used for providing data source configuration information and related parameter configuration information required in the key analysis process.

Preferably, the button execution engine includes:

the conversion object creating module is used for creating a corresponding conversion object according to the transmitted button file;

the information analysis module is used for acquiring database attribute information and parameter attribute information from the conversion object;

the assignment module is used for inquiring the data source configuration information and the parameter configuration information according to the database attribute information and the parameter attribute information, modifying the database attribute information of the conversion object according to the inquired matched database information, and modifying the parameter attribute information of the conversion object according to the inquired matched parameter information; and

and the scheduling module is connected with the APIs, calls the corresponding APIs to initialize and analyze the resource environment of the button file, completes assignment of the conversion object and executes the assigned conversion object, and therefore data circulation is completed.

According to a third aspect of the present invention, there is provided a data processing apparatus comprising: a memory; and a processor coupled to the memory, the processor configured to execute the above-mentioned key-based data synchronization method based on instructions stored in the memory.

According to a fourth aspect of the present invention, there is provided a computer readable storage medium storing computer instructions, which when executed by a processor, implement the data flow method of the above-mentioned key-based data synchronization system.

According to the scheme provided by the invention, data flow among a plurality of data sources can be randomly specified, namely, data synchronization is rapid and convenient, and the data synchronization process does not need to write responsible Sql sentences every time and does not need to make RPC remote call of timing tasks in application. During execution, the button file supports hot deployment, the uploading is effective, and the coupling degree between systems is low.

Drawings

The above and other objects, features and advantages of the present invention will become more apparent by describing embodiments of the present invention with reference to the following drawings, in which:

FIG. 1 is an overall design block for implementing data synchronization according to the present invention;

FIG. 2 is a schematic diagram of the data flow logic of the present invention;

FIG. 3 is a schematic block diagram of a data synchronization system based on a button according to the present invention;

FIG. 4 is a block diagram of the schematic structure of the execution of the button engine according to the present invention;

FIG. 5 is a schematic flow chart of the data synchronization method based on the button according to the present invention; and

FIG. 6 is a schematic flow chart of dynamically parsing a button file according to the present invention.

Detailed Description

The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details. Well-known methods, procedures, and procedures have not been described in detail so as not to obscure the present invention. The figures are not necessarily drawn to scale.

The flowcharts and block diagrams in the figures and block diagrams illustrate the possible architectures, functions, and operations of the systems, methods, and apparatuses according to the embodiments of the present invention, and may represent a module, a program segment, or merely a code segment, which is an executable instruction for implementing a specified logical function. It should also be noted that the executable instructions that implement the specified logical functions may be recombined to create new modules and program segments. The blocks of the drawings, and the order of the blocks, are thus provided to better illustrate the processes and steps of the embodiments and should not be taken as limiting the invention itself.

The invention provides a data synchronization system and method based on a button, as shown in fig. 1, in order to realize data synchronization, the invention needs to start from four aspects: configuration information, edit of a key file, analysis service of a key execution engine, and various key application program interfaces (abbreviated as key APIs) for completing the analysis service.

The configuration information includes various information and parameters required at the time of data synchronization. Mainly comprising data source information and some dynamic parameters used in the data synchronization process. The data source can be divided into a source data source and a target data source according to the data flow direction during synchronization. Whether a source data source or a target data source, the same data source information is required, such as data source identification, data source type, IP or URL link address of the data source. The data source identifier is used for distinguishing a data source, the data source type is data of a data type in the data source, such as oracle, MySql, SqlServer, WebService, SAP and the like, and the IP or URL link address of the data source provides the address of the data source.

The parameters are dynamic parameters, which are different according to the different key files, and therefore, each key file has corresponding dynamic parameters, such as: and conditions in SQL statements used in the data circulation process, parameter information of URLs and the like.

The configuration information is completed through a configuration page of the client, and the configured information is stored in a database in the data acquisition platform.

The editing of the key file is mainly completed on a key graphical interface provided by a client, and different nodes are set according to the requirement of data synchronization, so that a data circulation logic is formed. For example, as shown in fig. 2, the screen is an edit screen of a simple key file. The three nodes of the table input node, the SAP input node and the REST Client are respectively used for acquiring data of the data source. Taking the table input node as an example, the table input node defines a data source, i.e. data is obtained from a certain data table of a certain data source.

The JavaScript node can write a JS script. In practical application, the data in each row and each column in the data source may be subjected to checksum data format conversion. Such as: numerical data verification, date format verification, and string splitting, conversion between strings and other data types. This greatly reduces the amount of code passing verification in the program.

The table output node is used for selecting a service table, designating a column to be inserted and inserting data. The node may specify whether to insert in bulk or in single, or whether to process in the same transaction.

The three nodes of the table input node, the SAP input and the REST Client are used for acquiring data from corresponding source data sources, and therefore, corresponding database information as the source data sources, such as database types, e.g., Mysql, oracle, SQLServer, etc., need to be configured for the three nodes. Similarly, the table output node also needs to configure corresponding database information as a target data source. Thus, it can be understood that data obtained by a table input node from a source data source is inserted into a target data source corresponding to a table output node. The whole key file comprises a file and DB connection information which are formed by a plurality of nodes, and when the key file is loaded really, databases corresponding to the two nodes are replaced by real online databases according to configured data source configuration information, so that the real circulation of data is realized.

The key execution engine dynamically analyzes the key file, supports various key connection types such as Orale, MySql, SQLServer, WebService, SAP and the like, and can be matched with the data source type in the data source configuration information for connecting databases of different data types. Therefore, no matter what type of data source the button file relates to, the button execution engine can analyze the data source dynamically and in real time and support hot deployment, so that uploading of the button file is effective. The button execution engine defines and specifies the execution order in the execution process, such as: and performing assignment of parameters before execution, analysis of data sources and analysis of points in the execution process, outputting of return values after the analysis is finished, and the like.

The key API includes various key APIs commonly used in Etl tools, such as a key resource environment API and various conversion object APIs. The key APIs are mainly used for being called by a key engine to realize corresponding functions when the analysis service is carried out.

According to the design module, the invention provides a data synchronization system based on a button, as shown in fig. 3, comprising a client 1 and adata acquisition platform 2. The client 1 is used for configuring, editing and uploading a button file, wherein the configuration information comprises data source configuration information and related parameter configuration information. Thedata acquisition platform 2 is used for receiving the key file, dynamically analyzing the key file, and transferring the corresponding data stream of the source data source A to the target data source B according to the data stream logic and the corresponding parameter configuration information and the data source configuration information.

Specifically, as shown in fig. 4, the client includes a configuration module 11 and anediting module 12, where the configuration module 11 provides a configuration interface, and a user configures data source information and various dynamic parameter information required in a data flow process through the configuration interface, where the data source configuration information includes source data source information and target data source information. In one embodiment, the data source configuration table is generated when configuring the data source information, the parameter table is generated when configuring the parameter information, and both tables are stored in thedatabase 23 of thedata collection platform 2. Theediting module 12 provides a graphical user interface, and the user edits the key file through the graphical user interface, as described above, sets different nodes according to the data synchronization requirement, thereby forming a data flow logic.

Thedata acquisition platform 2 comprises abutton execution engine 21, a plurality ofAPIs 22 and adatabase 23, wherein thebutton execution engine 21 analyzes the button file by calling the API22 and completes data circulation. Specifically, thebutton execution engine 21 includes a transformationobject creation module 211, aninformation parsing module 212, anassignment module 213, and ascheduling module 214.

The conversionobject creating module 211 is configured to create a corresponding conversion object according to the sent key file.

Theinformation parsing module 212 is configured to obtain database attribute information and parameter attribute information from the conversion object.

Theassignment module 213 is configured to query the data source configuration information and the parameter configuration information according to the database attribute information and the parameter attribute information, modify the database attribute information of the conversion object according to the queried matched database information, and modify the parameter attribute information of the conversion object according to the queried matched parameter information.

Thescheduling module 214 is connected to a plurality of APIs, and invokes corresponding APIs to initialize and analyze a resource environment of the key file, complete assignment of the conversion object, and execute the assigned conversion object, thereby completing data transfer.

The API22 includes APIs for various functions, such as: the initialization API is used for initializing the resource environment for analyzing the keyboard file; the assignment API is used for finishing the dynamic assignment of the corresponding attribute information of the conversion object; and the execution API of the conversion object is used for executing the assigned conversion object so as to complete the data circulation.

As shown in fig. 5, the method for data synchronization based on a key specifically includes:

step S1, configuring data source information and related parameter information, where the data source information includes source data source information and target data source information.

A plurality of data sources can be configured by using a configuration interface provided by the client 1, and each data source has a unique data source identification. The data source configuration information includes, but is not limited to, a data source identification, a data source type, an IP or URL link address of the data source. Through the data source identification, other required database information, such as IP addresses and the like, can be obtained, and then the corresponding database is connected.

Step S2, editing a button file comprising data circulation logic through a button graphical interface; the key file has a data source identifier, and the data source identifier is consistent with the data source identifier in the configuration information.

After the key file is generated and before the key file is uploaded to a key execution engine, testing the key file is executed locally, wherein the testing process comprises the following steps:

test data is acquired through the table input nodes, for example: table1 data in test1 database; .

The completeness and the accuracy of the acquired data are verified by comparing the acquired data with preset required data; for example, the data in table1 is compared with the data in the Excel requirement file, mainly with the columns in the Excel requirement file, to see whether the data in the requirement file is imported and correctly imported.

And if the check is passed, namely the table input node acquires the correct and required data, inserting the acquired data into a data table of the designated database in the table data nodes. Such as table2 of the test2 database. Therefore, the data of table1 in the test1 database is inserted into table2 of the test2 database according to the specified columns and rows, and the log information in the operation process is returned to the client. The client checks whether the exception exists or not and whether the data circulation is correctly executed or not. If the data flow is correctly executed, step S2 may be executed to upload the key file to the key execution engine.

In step S3, the key file is uploaded to thekey execution engine 21. When the information configuration is performed in step 1, the data source information and the parameter information may be configured as a data source configuration table and a parameter table, respectively, and stored in thedatabase 23 of the data acquisition platform.

And step S4, dynamically analyzing the button file. First, thekey execution engine 21 initializes the resource environment for parsing the key file by calling the initialization API. Then, the conversionobject creating module 211 of the button execution engine creates a conversion object according to the button file, and the assignment module calls an assignment API through thescheduling module 214 to complete dynamic assignment of the corresponding attribute information of the conversion object. The conversion object comprises data source attribute information, parameter attribute information and node attribute information, the currently obtained data source attribute information is data source information during testing and is not data source information of real streaming number, so that the original test data source information needs to be modified into database information of real streaming data.

With respect to the parameters, for example: when the key file contains the table input node, the query statement in the key file is a Sql statement with parameters, and the condition parameters used in the Sql statement are endowed with corresponding parameter information values during configuration. For the Rest Client node, some request parameters are also needed, which are all ready at this time, i.e. they are given the appropriate values before running.

Step S5, after the above-mentioned work is completed, the execution API of the transformation object is called, and the assigned transformation object is executed, so as to achieve the purpose of transferring the corresponding data stream of the source data source to the target data source.

In addition, in order to make the user know the data circulation situation, after the corresponding data flow of the source data source is transferred to the target data source, the method also comprises the step of returning the execution information, namely sending the log information of the recording operation process back to the client.

Specifically, as shown in fig. 6, the dynamic parsing of the key file in step S4 includes the following steps:

step S41, calling an initialization API, and initializing a resource environment executed by the button file;

step S42, dynamically loading the button file, and generating a corresponding conversion object, wherein the conversion object comprises data source attribute information, parameter attribute information and node attribute information;

step S43, calling API to analyze data source information, and according to the data source identification in the data source attribute information of the conversion object, acquiring the source data source information and the target data source information corresponding to the data source identification in the data source configuration table;

step S44, modifying the data source attribute information according to the source data source information and the target data source information, and finishing assignment of the data source information;

step S45, searching the parameter table according to the parameters needing assignment in the parameter attribute information;

and step S46, according to the found specific parameters, modifying the parameter attribute information and finishing the assignment of the parameters.

The invention can configure different types of data sources, thereby realizing data flow between different data sources. And the data conversion among a plurality of data sources can be designated at will, the method is quick and convenient, the Sql responsible for each data synchronous writing is not needed, and the remote call of the timing task RPC in the application is not needed.

The invention can dynamically analyze the button file, thereby supporting hot deployment and being effective when uploading. The invention can configure different database types, support the connection of various data and solve the problem of system connection types in the system, thereby having low coupling degree among the systems.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (12)

1. A data synchronization method based on a button comprises the following steps:

configuring a data source and related parameters through a client, and storing data source configuration information and parameter configuration information in a database;

editing a key file comprising data circulation logic through a client-side key graphical interface;

uploading the button file to a button execution engine;

dynamically loading the button file and generating a corresponding conversion object, wherein the conversion object comprises data source attribute information and parameter attribute information;

acquiring corresponding source data source information and target data source information in data source configuration information according to the data source identification in the data source attribute information;

modifying the data source attribute information according to the source data source information and the target data source information;

searching parameter configuration information according to the parameters needing to be assigned in the parameter attribute information, and modifying the parameter attribute information according to the searched specific parameters; and

and transferring the corresponding data stream of the source data source to the target data source according to the data stream logic and the corresponding parameter configuration information and the data source configuration information.

2. The key-based data synchronization method of claim 1, wherein the data source configuration information includes a data source identification, a data source type, an IP or URL link address of the data source; and/or the parameters comprise dynamic parameters required in data circulation.

3. The key-based data synchronization method of claim 2, wherein the data source configuration information is recorded in a data source configuration table; and/or the parameter configuration information is recorded in a parameter table.

4. The key-based data synchronization method of claim 2, wherein the data source type includes one or a combination of any of oracle, MySql, SQLServer, WebService, and SAP.

5. The key-based data synchronization method of claim 1, wherein before uploading the key file to the key execution engine, locally performing a test on the key file and verifying the completeness and accuracy of the acquired data.

6. The key-based data synchronization method of claim 1, wherein after forwarding the corresponding data stream of the source data source to the target data source, further comprising the step of returning execution information to the client.

7. A key-based data synchronization system, comprising:

the client is used for configuring, editing and uploading the key file; and

a data acquisition platform for dynamically loading the button file and generating a corresponding conversion object, wherein the conversion object comprises data source attribute information and parameter attribute information,

acquiring corresponding source data source information and target data source information from the data source configuration information according to the data source identification in the data source attribute information,

modifying the data source attribute information according to the source data source information and the target data source information,

searching parameter configuration information according to the parameters needing to be assigned in the parameter attribute information, modifying the parameter attribute information according to the searched specific parameters, and transferring the corresponding data stream of the source data source to the target data source according to the data stream logic and the corresponding parameter configuration information and the data source configuration information.

8. The key-based data synchronization system of claim 7 wherein the client comprises:

the configuration module is used for configuring a data source and relevant parameters and storing data source configuration information and parameter configuration information in a database;

and the editing module is used for editing the key file comprising the data circulation logic by providing a key graphical interface.

9. The key-based data synchronization system of claim 7 wherein the data acquisition platform comprises:

the key execution engine is used for dynamically analyzing the key file and completing data circulation;

the multiple APIs are used for the button execution engine to call to complete the analysis of the button file and the data circulation; and

and the database is used for providing data source configuration information and parameter configuration information required in the key analysis process.

10. The key-based data synchronization system of claim 9 wherein the key execution engine comprises:

the conversion object creating module is used for creating a corresponding conversion object according to the transmitted button file;

the information analysis module is used for acquiring database attribute information and parameter attribute information from the conversion object;

the assignment module is used for inquiring the data source configuration information and the parameter configuration information according to the database attribute information and the parameter attribute information, modifying the database attribute information of the conversion object according to the inquired matched database information, and modifying the parameter attribute information of the conversion object according to the inquired matched parameter information;

and the scheduling module is connected with the APIs, calls the corresponding APIs to initialize and analyze the resource environment of the button file, completes assignment of the conversion object and executes the assigned conversion object, and therefore data circulation is completed.

11. A data processing apparatus comprising: a memory; and a processor coupled to the memory, the processor configured to perform the method for Kettle-based data synchronization of any of claims 1 to 6 based on instructions stored in the memory.

12. A computer readable storage medium storing computer instructions which, when executed by a processor, implement the method for key-based data synchronization of any one of claims 1 to 6.

Images