CN114020850A - Database data synchronization method, device, equipment and readable storage medium - Google Patents
Database data synchronization method, device, equipment and readable storage mediumDownload PDFInfo
- Publication number
- CN114020850A CN114020850ACN202210002655.7ACN202210002655ACN114020850ACN 114020850 ACN114020850 ACN 114020850ACN 202210002655 ACN202210002655 ACN 202210002655ACN 114020850 ACN114020850 ACN 114020850A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronized
- synchronization
- database
- binlog
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The invention relates to the technical field of database sub-database data synchronization and data sharing, in particular to a database data synchronization method, a database data synchronization device, database data synchronization equipment and a readable storage medium.
Background
MySQL is the most popular relational database in the Internet industry, and most business data of Internet IT companies are stored in MySQL. Due to the needs of database sub-database data synchronization, data sharing, data warehouse construction to form enterprise data assets and the like, the demands of timing distribution and synchronization of business data in MySQL to other data storage media and the like are derived. The existing method for synchronizing data in a MySQL data source to a target data storage medium needs to establish an interactive protocol of MySQL, the technical threshold is high, and the method for completing data synchronization through the interactive protocol is too complex, namely, the problem of high learning cost exists when an open source component is used.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The application mainly aims to provide a database data synchronization method, a database data synchronization device, a database data synchronization equipment and a readable storage medium, and aims to solve the technical problem that in the prior art, the implementation of a method for completing data synchronization through an interactive protocol is too complex, namely, the use of an open source component has high learning cost.
In order to achieve the above object, the present invention provides a database data synchronization method, where the database data synchronization method includes the following steps:
acquiring a binlog log of first data to be synchronized in an original database and synchronous recording information of second data to be synchronized in a target database;
judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information;
and when the first data to be synchronized and the second data to be synchronized are judged to be not synchronized, synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information and the first data to be synchronized, and updating the synchronization record information according to synchronization content.
Further, the step of determining whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information includes:
acquiring the latest binlog position in the binlog log;
judging whether the latest binlog position is the same as the last synchronous position recorded in the synchronous recording information;
when the latest binlog position is different from the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are not synchronized;
and when the latest binlog position is equal to the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are synchronized, and not performing data synchronization operation.
Further, the step of synchronizing second data to be synchronized in the target database based on the binlog log, the synchronization record information, and the first data to be synchronized includes:
judging whether the synchronous recording information is empty or not;
if the synchronous record information is not empty, generating a synchronous data position collection according to the binlog log and the synchronous record information;
and carrying out data synchronization on the second data to be synchronized according to the synchronous data position collection and the first data to be synchronized.
Further, after the step of determining whether the synchronization record information is empty, the method includes:
if the synchronous record information is empty, performing full synchronization, wherein the full synchronization is as follows: and clearing the second data to be synchronized, reading the first data to be synchronized, and sending the first data to be synchronized to the target database as the second data to be synchronized.
Further, the binlog log at least comprises one of a latest binlog position, a latest binlog file name and a latest update time, the synchronization record information at least comprises one of a last synchronization file name, a last synchronization time and the last synchronization position, and the step of determining whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information comprises:
judging whether the latest binlog position is equal to the last synchronization position, judging whether the latest binlog file name is equal to the last synchronization file name, and judging whether the latest updating time is equal to the last synchronization time;
and when the latest binlog position is equal to the last synchronization position, the latest binlog file name is equal to the last synchronization file name, and the latest updating time is equal to the last synchronization time, judging that the binlog log is synchronous with the synchronization record information.
Further, before the step of obtaining the binlog log of the first data to be synchronized in the original database and the synchronization record information of the second data to be synchronized in the target database, the method includes:
and establishing a synchronous link between a target application program and the target database, and executing the step of synchronizing the second data to be synchronized in the target database through the synchronous link.
Further, before the step of establishing a synchronous link between the target application and the target database, the method comprises:
loading a preset MySqlCDC program to the target application program;
acquiring the binlog log of the first data to be synchronized in the original database and the synchronous recording information of the second data to be synchronized in the target database by the preset MySqlCDC program;
and setting the acquisition frequency of the binlog log and the acquisition frequency of the synchronous recording information based on user operation.
In addition, to achieve the above object, the present invention further provides a database data synchronization apparatus, which is applied to a database sub-database data synchronization or data sharing system, and the database data synchronization apparatus includes
The reading module is used for acquiring a binlog log of first data to be synchronized in an original database and synchronous recording information of second data to be synchronized in a target database;
the judging module is used for judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronous recording information;
and the synchronization module is used for synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information and the first data to be synchronized when the first data to be synchronized and the second data to be synchronized are judged to be asynchronous, and updating the synchronization record information according to synchronization content.
In addition, to achieve the above object, the present invention also provides a database data synchronization apparatus, including: the system comprises a memory, a processor and a database data synchronization program stored on the memory and capable of running on the processor, wherein the database data synchronization program realizes the steps of the database data synchronization method when being executed by the processor.
In addition, to achieve the above object, the present invention further provides a readable storage medium, on which a database data synchronization program is stored, and the database data synchronization program, when executed by a processor, implements the steps of the database data synchronization method as described above.
Compared with the technical means of completing the data synchronization in the MySQL data source to the target data storage medium by establishing an interactive protocol of MySQL in the prior art, the method and the device for synchronizing the data in the MySQL data source to the target data storage medium acquire the binlog log of the target database and the synchronous record information of the target database in the application program, obtain a data collection to be synchronized through comparison and analysis, synchronize the data collection to be synchronized to the target database to complete a data synchronization task, reduce the technical threshold of the technology of synchronizing the data in the MySQL data source to the target data storage medium, reduce the use cost of a user and improve the use experience of the user.
Drawings
FIG. 1 is a schematic diagram of an apparatus architecture of a hardware operating environment according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a database data synchronization method according to a first embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, fig. 1 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present invention.
The device of the embodiment of the invention can be a server, and can also be an electronic terminal device with data receiving, data processing and data transmitting functions, such as a PC, a portable computer and the like.
As shown in fig. 1, the apparatus may include: aprocessor 1001, such as a CPU, anetwork interface 1004, auser interface 1003, amemory 1005, acommunication bus 1002. Wherein acommunication bus 1002 is used to enable connective communication between these components. Theuser interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and theoptional user interface 1003 may also include a standard wired interface, a wireless interface. Thenetwork interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). Thememory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). Thememory 1005 may alternatively be a storage device separate from theprocessor 1001.
Optionally, the device may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, WiFi modules, and so forth. Such as light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display screen according to the brightness of ambient light, and a proximity sensor that may turn off the display screen and/or the backlight when the mobile terminal is moved to the ear. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally, three axes), detect the magnitude and direction of gravity when the mobile terminal is stationary, and can be used for applications (such as horizontal and vertical screen switching, related games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer and tapping) and the like for recognizing the attitude of the mobile terminal; of course, the mobile terminal may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which are not described herein again.
Those skilled in the art will appreciate that the configuration of the apparatus shown in fig. 1 is not intended to be limiting of the apparatus and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, amemory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a database data synchronization program.
In the terminal shown in fig. 1, thenetwork interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; theuser interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and theprocessor 1001 may be configured to invoke a database data synchronization program stored in thememory 1005 and perform the following operations:
acquiring a binlog log of first data to be synchronized in an original database and synchronous recording information of second data to be synchronized in a target database;
judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information;
and when the first data to be synchronized and the second data to be synchronized are judged to be not synchronized, synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information and the first data to be synchronized, and updating the synchronization record information according to synchronization content.
Further, theprocessor 1001 may call the database data synchronization program stored in thememory 1005, and also perform the following operations:
the step of judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information comprises:
acquiring the latest binlog position in the binlog log;
judging whether the latest binlog position is the same as the last synchronous position recorded in the synchronous recording information;
when the latest binlog position is different from the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are not synchronized;
and when the latest binlog position is equal to the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are synchronized, and not performing data synchronization operation.
Further, theprocessor 1001 may call the database data synchronization program stored in thememory 1005, and also perform the following operations:
the step of synchronizing second data to be synchronized in the target database based on the binlog log, the synchronization record information, and the first data to be synchronized comprises:
judging whether the synchronous recording information is empty or not;
if the synchronous record information is not empty, generating a synchronous data position collection according to the binlog log and the synchronous record information;
and carrying out data synchronization on the second data to be synchronized according to the synchronous data position collection and the first data to be synchronized.
Further, theprocessor 1001 may call the database data synchronization program stored in thememory 1005, and also perform the following operations:
after the step of judging whether the synchronous recording information is empty, the method comprises the following steps:
if the synchronous record information is empty, performing full synchronization, wherein the full synchronization is as follows: and clearing the second data to be synchronized, reading the first data to be synchronized, and sending the first data to be synchronized to the target database as the second data to be synchronized.
Further, theprocessor 1001 may call the database data synchronization program stored in thememory 1005, and also perform the following operations:
the binlog log at least comprises one of a latest binlog position, a latest binlog file name and a latest update time, the synchronization record information at least comprises one of a last synchronization file name, a last synchronization time and the last synchronization position, and the step of judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information comprises the steps of:
judging whether the latest binlog position is equal to the last synchronization position, judging whether the latest binlog file name is equal to the last synchronization file name, and judging whether the latest updating time is equal to the last synchronization time;
and when the latest binlog position is equal to the last synchronization position, the latest binlog file name is equal to the last synchronization file name, and the latest updating time is equal to the last synchronization time, judging that the binlog log is synchronous with the synchronization record information.
Further, theprocessor 1001 may call the database data synchronization program stored in thememory 1005, and also perform the following operations:
before the step of obtaining the binlog log of the first data to be synchronized in the original database and the synchronization record information of the second data to be synchronized in the target database, the method comprises the following steps:
and establishing a synchronous link between a target application program and the target database, and executing the step of synchronizing the second data to be synchronized in the target database through the synchronous link.
Further, theprocessor 1001 may call the database data synchronization program stored in thememory 1005, and also perform the following operations:
prior to the step of establishing a synchronous link of a target application with the target database, comprising:
loading a preset MySqlCDC program to the target application program;
acquiring the binlog log of the first data to be synchronized in the original database and the synchronous recording information of the second data to be synchronized in the target database by the preset MySqlCDC program;
and setting the acquisition frequency of the binlog log and the acquisition frequency of the synchronous recording information based on user operation.
Referring to fig. 2, a first embodiment of the database data synchronization method of the present invention includes:
step S10, acquiring the binlog log of the first data to be synchronized in the original database and the synchronous recording information of the second data to be synchronized in the target database;
in this embodiment, the original database is usually a MySQL database, the first data to be synchronized refers to data that needs to be synchronized from the original database to the target database, for example, there are 3 data tables in the MySQL database, which are table 1, table 2, and table 3, the first data to be synchronized may be any one, multiple, or all of the three data tables, and when data change occurs in each data table, that is, when data update is performed in the original database, specific change steps or update contents are recorded by a binlog (binary log). The target database may be a relational database management system such as MySQL, maridb, supercona Server, PostgreSQL, and the like, where the second data to be synchronized corresponds to the first data to be synchronized, if the second data to be synchronized is table 1 and table 2, the first data to be synchronized is table 1 and table 2, and similarly, if the second data to be synchronized is table 1, table 2, and table 3, the first data to be synchronized is also table 1, table 2, and table 3, and the synchronization record information is synchronization content of the last data synchronization performed by the target database. In addition, there may be a plurality of target databases during data synchronization, and since the synchronization process performed on each target database is the same, the following description will be given by taking a target database synchronization process as an example.
Further, before the step of obtaining the binlog log of the first data to be synchronized in the original database and the synchronous record information of the second data to be synchronized in the target database, a synchronous link between the target application program and the target database is established, and the step of synchronizing the second data to be synchronized in the target database is executed through the synchronous link.
It can be understood that, the target application corresponds to the target database, in this embodiment, the data updating step of the target database is specifically completed by the target program, and therefore, before data synchronization is performed, a synchronization link between the target application and the target database needs to be established, where the synchronization link is used for data transmission from the target application to the target database during subsequent update of the target database.
Further, before the step of establishing a synchronous link between a target application program and the target database, loading a preset MySqlCDC program to the target application program; acquiring the binlog log of the first data to be synchronized in the original database and the synchronous recording information of the second data to be synchronized in the target database by the preset MySqlCDC program; and setting the acquisition frequency of the binlog log and the acquisition frequency of the synchronous recording information based on user operation.
In this embodiment, the preset myqlcdc program is loaded to the target application program, that is, the data synchronization component and the application program are integrated, and the preset myqlcdc program completes reading of the binlog log and the synchronization record information and subsequent generation of this time synchronization data, and a specific generation manner is further described later.
Meanwhile, a user can set the data synchronization frequency according to actual requirements, for example, the synchronization record information of the binlog log of the original database and the target database is acquired once every 24 hours, or the acquisition step is triggered when existing data of the original database is updated. Specifically, a handler program is loaded in a target application program, a timing scheduling frame is installed by using an Install-Package handler (task scheduling frame) command, and the task of acquiring two database synchronization tables for data synchronization is executed according to a certain frequency, wherein the specific command statement is as follows: recurringjobjob ("myrringjobjob" () = > DataDistributionTask (arg), cron.
Step S20, determining whether the first data to be synchronized and the second data to be synchronized are synchronized by the binlog log and the synchronization record information;
further, acquiring the latest binlog position in the binlog log; judging whether the latest binlog position is the same as the last synchronous position recorded in the synchronous recording information; when the latest binlog position is different from the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are not synchronized; and when the latest binlog position is equal to the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are synchronized, and not performing data synchronization operation.
Specifically, the preset mysqldcdc program will obtain the latest log record of the binlog (binary log file) of the original database, such as the position of the latest binlog, which is used to record the sequence number of the data change event of the original database, and similarly, obtain the synchronization record information of the target database, where the last synchronization position recorded in the synchronization record information is the position of the event in the binlog log executed last by the target database, and compare the last synchronization position with the binlog position, such as: and if the latest binlog position is 418 and the last synchronization position is 325, determining that the first data to be synchronized and the second data to be synchronized are not synchronized, if the latest binlog position is 418 and the last synchronization position is 325, determining that the first data to be synchronized and the second data to be synchronized are synchronized, and no data synchronization operation is required. The binlog position represents a serial number of an executed event of the database, and it can be understood that the present application is directed to synchronizing data in a source database to a target database, so the executed event is usually recorded as an operation such as deletion or addition performed in the database, and the object data of the operation, if data 1 is added to table 1, the existing serial number of an event corresponding to the event is the position. Whether the first data to be synchronized and the second data to be synchronized are synchronized can be judged through the positions of events which are already executed by the two databases under the normal condition.
Furthermore, the binlog log at least comprises one of a latest binlog position, a latest binlog file name and a latest update time, the synchronization record information at least comprises one of a last synchronization file name, a last synchronization time and the last synchronization position, and the step of determining whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information comprises: judging whether the latest binlog position is equal to the last synchronization position, judging whether the latest binlog file name is equal to the last synchronization file name, and judging whether the latest updating time is equal to the last synchronization time; and when the latest binlog position is equal to the last synchronization position, the latest binlog file name is equal to the last synchronization file name, and the latest updating time is equal to the last synchronization time, judging that the binlog log is synchronous with the synchronization record information.
Specifically, when the first data to be synchronized and the second data to be synchronized are judged to be synchronized, whether the first data to be synchronized and the second data to be synchronized are judged according to the file name and the time corresponding to the position in addition to whether the first data to be synchronized and the second data to be synchronized are judged according to the position. The specific determination steps are the same as the positions, and are not described herein again.
Step S30, when it is determined that the first data to be synchronized and the second data to be synchronized are not synchronized, synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information, and the first data to be synchronized, and updating the synchronization record information according to synchronization content.
When the first to-be-synchronized data and the second to-be-synchronized data are judged to be asynchronous, further judging whether the synchronous recording information is empty; if the synchronous record information is not empty, generating a synchronous data position collection according to the binlog log and the synchronous record information; and carrying out data synchronization on the second data to be synchronized according to the synchronous data position collection and the first data to be synchronized.
Specifically, based on the above example that the latest binlog position is 418 and the last synchronization position is 325, it is determined that the first data to be synchronized and the second data to be synchronized are not synchronized, and the last synchronization position recorded in the synchronization record information is 325, the synchronization record information is not null, and a synchronization data position set is generated according to the binlog and the synchronization record information, where the synchronization data position set is from position 326 to position 418 if the latest binlog position is 418 and the last synchronization position is 325. The event with the position of 326 to 418 in the binlog of the original database is replayed in the target database, that is, the event with the position of 326 to 418 in the original database is executed in the target database, so that the synchronization of the data of the original database to the target database is completed, meanwhile, the last synchronization position in the synchronization record information of the target database is updated to 418 from the original 326, if the data synchronization task is executed again under the condition that the original database does not perform data deletion and other operations, the first data to be synchronized and the second data to be synchronized are judged to be synchronized, and further data synchronization task is not needed.
Further, after the step of determining whether the synchronization record information is empty, if the synchronization record information is empty, performing full synchronization, where the full synchronization is: and clearing the second data to be synchronized, reading the first data to be synchronized, and sending the first data to be synchronized to the target database as the second data to be synchronized.
Specifically, when it is determined that the target database does not have the synchronization record information, the synchronization record information is determined to be empty, the second data to be synchronized of the target database is emptied, if the second data to be synchronized is table 1, clearing the table 1 of the second data to be synchronized, reading the data stored in the table 1 of the first data to be synchronized of the original database, the data stored in the first table 1 of the data to be synchronized is inserted into the second table 1 of the data to be synchronized of the target database in batch by MySqlBulkCopy to complete data distribution, the last synchronization position in the synchronization record information of the target database is also updated, and in addition, the synchronous record information is also distributed to the original database, and when a plurality of target databases exist, the synchronous condition of different target databases can be known through different synchronous record information in the original database.
It can be understood that, in this embodiment, the mysql cdc program is loaded on the target application program, the synchronization component is integrated with the target program, the mysql cdc program synchronizes data from the original database to the target database of the target application program, specifically, a dataset to be synchronized is obtained by obtaining synchronization record information of the binlog log of the target database and the target database, and a dataset to be synchronized is synchronized to the target database by comparison and analysis, so as to complete a data synchronization task. A complex MySQL interaction protocol does not need to be established, and the problem that a user pays high learning cost to learn an open source framework with the same complex function can be avoided. The problem that a complex open source framework is not matched with user requirements is solved, and users are helped to develop timing data distribution application suitable for the user requirements at low cost.
In addition, this embodiment further provides a database data synchronization device, where the database data synchronization device is applied to a database sub-database data synchronization or data sharing system, and the database data synchronization device includes
The reading module is used for acquiring a binlog log of first data to be synchronized in an original database and synchronous recording information of second data to be synchronized in a target database;
the judging module is used for judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronous recording information;
and the synchronization module is used for synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information and the first data to be synchronized when the first data to be synchronized and the second data to be synchronized are judged to be asynchronous, and updating the synchronization record information according to synchronization content.
The database data synchronization device provided by the invention adopts the database data synchronization method in the embodiment, and solves the technical problem of high learning cost in the prior art that the data in the MySQL data source is synchronized to the target data storage medium by establishing the MySQL interaction protocol. Compared with the prior art, the beneficial effects of the database data synchronization device provided by the embodiment of the invention are the same as the beneficial effects of the database data synchronization method provided by the embodiment, and other technical features of the database data synchronization device are the same as those disclosed by the embodiment method, which are not repeated herein.
In addition, this embodiment further provides a database data synchronization device, where the database data synchronization device includes: the system comprises a memory, a processor and a database data synchronization program stored on the memory and capable of running on the processor, wherein the database data synchronization program realizes the steps of the database data synchronization method when being executed by the processor.
The specific implementation of the database data synchronization device of the present invention is substantially the same as the embodiments of the database data synchronization method described above, and will not be described herein again.
In addition, the present embodiment also provides a readable storage medium, where a database data synchronization program is stored, and when executed by a processor, the database data synchronization program implements the steps of the database data synchronization method as described above.
The specific implementation of the medium of the present invention is basically the same as that of the above-mentioned database data synchronization method, and is not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A database data synchronization method is characterized by comprising the following steps:
acquiring a binlog log of first data to be synchronized in an original database and synchronous recording information of second data to be synchronized in a target database;
judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information;
and when the first data to be synchronized and the second data to be synchronized are judged to be not synchronized, synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information and the first data to be synchronized, and updating the synchronization record information according to synchronization content.
2. The database data synchronization method according to claim 1, wherein the step of determining whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information comprises:
acquiring the latest binlog position in the binlog log;
judging whether the latest binlog position is the same as the last synchronous position recorded in the synchronous recording information;
when the latest binlog position is different from the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are not synchronized;
and when the latest binlog position is equal to the last synchronization position, judging that the first data to be synchronized and the second data to be synchronized are synchronized, and not performing data synchronization operation.
3. The database data synchronization method of claim 2, wherein the step of synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information, and the first data to be synchronized comprises:
judging whether the synchronous recording information is empty or not;
if the synchronous record information is not empty, generating a synchronous data position collection according to the binlog log and the synchronous record information;
and carrying out data synchronization on the second data to be synchronized according to the synchronous data position collection and the first data to be synchronized.
4. The database data synchronization method of claim 3, wherein after the step of determining whether the synchronization record information is empty, comprising:
if the synchronous record information is empty, performing full synchronization, wherein the full synchronization is as follows: and clearing the second data to be synchronized, reading the first data to be synchronized, and sending the first data to be synchronized to the target database as the second data to be synchronized.
5. The database data synchronization method according to claim 1, wherein the binlog log includes at least one of a latest binlog position, a latest binlog file name and a latest update time, the synchronization record information includes at least one of a last synchronization file name, a last synchronization time and a last synchronization position, and the step of determining whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronization record information includes:
judging whether the latest binlog position is equal to the last synchronization position, judging whether the latest binlog file name is equal to the last synchronization file name, and judging whether the latest updating time is equal to the last synchronization time;
and when the latest binlog position is equal to the last synchronization position, the latest binlog file name is equal to the last synchronization file name, and the latest updating time is equal to the last synchronization time, judging that the binlog log is synchronous with the synchronization record information.
6. The database data synchronization method of claim 1, wherein before the step of obtaining the binlog log of the first data to be synchronized in the original database and the synchronization record information of the second data to be synchronized in the target database, comprising:
and establishing a synchronous link between a target application program and the target database, and executing the step of synchronizing the second data to be synchronized in the target database through the synchronous link.
7. The database data synchronization method of claim 6, prior to the step of establishing a synchronization link of a target application with the target database, comprising:
loading a preset MySqlCDC program to the target application program;
acquiring the binlog log of the first data to be synchronized in the original database and the synchronous recording information of the second data to be synchronized in the target database by the preset MySqlCDC program;
and setting the acquisition frequency of the binlog log and the acquisition frequency of the synchronous recording information based on user operation.
8. The database data synchronization device is applied to a database sub-database data synchronization or data sharing system, and comprises
The reading module is used for acquiring a binlog log of first data to be synchronized in an original database and synchronous recording information of second data to be synchronized in a target database;
the judging module is used for judging whether the first data to be synchronized and the second data to be synchronized are synchronized through the binlog log and the synchronous recording information;
and the synchronization module is used for synchronizing the second data to be synchronized in the target database based on the binlog log, the synchronization record information and the first data to be synchronized when the first data to be synchronized and the second data to be synchronized are judged to be asynchronous, and updating the synchronization record information according to synchronization content.
9. A database data synchronization apparatus, characterized in that the database data synchronization apparatus comprises: memory, a processor and a database data synchronization program stored on the memory and executable on the processor, the database data synchronization program when executed by the processor implementing the steps of the database data synchronization method of any of claims 1 to 7.
10. A readable storage medium, having stored thereon a database data synchronization program, which when executed by a processor implements the steps of the database data synchronization method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210002655.7ACN114020850B (en) | 2022-01-05 | 2022-01-05 | Database data synchronization method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210002655.7ACN114020850B (en) | 2022-01-05 | 2022-01-05 | Database data synchronization method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114020850Atrue CN114020850A (en) | 2022-02-08 |
| CN114020850B CN114020850B (en) | 2022-04-08 |
Family
ID=80069281
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210002655.7AActiveCN114020850B (en) | 2022-01-05 | 2022-01-05 | Database data synchronization method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114020850B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114676112A (en)* | 2022-03-24 | 2022-06-28 | 平安科技（深圳）有限公司 | Data synchronization method and storage medium |
| CN115277681A (en)* | 2022-07-29 | 2022-11-01 | 山石网科通信技术股份有限公司 | File synchronization method and device for avoiding synchronization process conflict |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6526417B1 (en)* | 2000-01-25 | 2003-02-25 | International Business Machines Corporation | System and method for change accumulation unmerged update reduction |
| CN103034650A (en)* | 2011-09-29 | 2013-04-10 | 北京新媒传信科技有限公司 | System and method for processing data |
| CN103221949A (en)* | 2010-07-27 | 2013-07-24 | 甲骨文国际公司 | MYSQL database heterogeneous log based replication |
| CN103530290A (en)* | 2012-07-03 | 2014-01-22 | 深圳市腾讯计算机系统有限公司 | Method and system for data migration among databases |
| CN104346479A (en)* | 2014-11-26 | 2015-02-11 | 北京奇虎科技有限公司 | Database synchronization method and database synchronization device |
| CN105955989A (en)* | 2015-12-31 | 2016-09-21 | 无锡华云数据技术服务有限公司 | Method for establishing master and slave servers of cloud platform database |
| US20180150540A1 (en)* | 2016-11-29 | 2018-05-31 | Sap Se | Synchronization Mechanism for Serialized Data Log Replay in Database Systems |
| US20180246948A1 (en)* | 2017-02-28 | 2018-08-30 | Sap Se | Replay of Redo Log Records in Persistency or Main Memory of Database Systems |
| CN110209735A (en)* | 2019-05-05 | 2019-09-06 | 深圳市腾讯计算机系统有限公司 | Database backup method, calculates equipment and storage medium at DB Backup device |
| CN111008241A (en)* | 2019-11-14 | 2020-04-14 | 微民保险代理有限公司 | Data synchronization method and device, storage medium and computer equipment |
| CN111309742A (en)* | 2020-01-20 | 2020-06-19 | 深圳壹账通智能科技有限公司 | Data synchronization method, device, computer equipment and storage medium |
| CN111639132A (en)* | 2020-05-29 | 2020-09-08 | 中国联合网络通信集团有限公司 | Log synchronization method and device |
| CN112434043A (en)* | 2020-12-02 | 2021-03-02 | 新华三大数据技术有限公司 | Data synchronization method, device, electronic equipment and medium |
| CN113220794A (en)* | 2016-02-29 | 2021-08-06 | 株式会社特迈数据 | Method, server and computer readable medium for synchronizing query results between heterogeneous databases |
- 2022
- 2022-01-05CNCN202210002655.7Apatent/CN114020850B/enactiveActive
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6526417B1 (en)* | 2000-01-25 | 2003-02-25 | International Business Machines Corporation | System and method for change accumulation unmerged update reduction |
| CN103221949A (en)* | 2010-07-27 | 2013-07-24 | 甲骨文国际公司 | MYSQL database heterogeneous log based replication |
| CN103034650A (en)* | 2011-09-29 | 2013-04-10 | 北京新媒传信科技有限公司 | System and method for processing data |
| CN103530290A (en)* | 2012-07-03 | 2014-01-22 | 深圳市腾讯计算机系统有限公司 | Method and system for data migration among databases |
| CN104346479A (en)* | 2014-11-26 | 2015-02-11 | 北京奇虎科技有限公司 | Database synchronization method and database synchronization device |
| CN105955989A (en)* | 2015-12-31 | 2016-09-21 | 无锡华云数据技术服务有限公司 | Method for establishing master and slave servers of cloud platform database |
| CN113220794A (en)* | 2016-02-29 | 2021-08-06 | 株式会社特迈数据 | Method, server and computer readable medium for synchronizing query results between heterogeneous databases |
| US20180150540A1 (en)* | 2016-11-29 | 2018-05-31 | Sap Se | Synchronization Mechanism for Serialized Data Log Replay in Database Systems |
| US20180246948A1 (en)* | 2017-02-28 | 2018-08-30 | Sap Se | Replay of Redo Log Records in Persistency or Main Memory of Database Systems |
| CN110209735A (en)* | 2019-05-05 | 2019-09-06 | 深圳市腾讯计算机系统有限公司 | Database backup method, calculates equipment and storage medium at DB Backup device |
| CN111008241A (en)* | 2019-11-14 | 2020-04-14 | 微民保险代理有限公司 | Data synchronization method and device, storage medium and computer equipment |
| CN111309742A (en)* | 2020-01-20 | 2020-06-19 | 深圳壹账通智能科技有限公司 | Data synchronization method, device, computer equipment and storage medium |
| CN111639132A (en)* | 2020-05-29 | 2020-09-08 | 中国联合网络通信集团有限公司 | Log synchronization method and device |
| CN112434043A (en)* | 2020-12-02 | 2021-03-02 | 新华三大数据技术有限公司 | Data synchronization method, device, electronic equipment and medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114676112A (en)* | 2022-03-24 | 2022-06-28 | 平安科技（深圳）有限公司 | Data synchronization method and storage medium |
| CN115277681A (en)* | 2022-07-29 | 2022-11-01 | 山石网科通信技术股份有限公司 | File synchronization method and device for avoiding synchronization process conflict |
| CN115277681B (en)* | 2022-07-29 | 2024-04-02 | 山石网科通信技术股份有限公司 | File synchronization method and device for avoiding conflict in synchronization process |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114020850B (en) | 2022-04-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109976620B (en) | Method, device, equipment and storage medium for determining list item display attribute information | |
| US11023716B2 (en) | Method and device for generating stickers | |
| US11762931B2 (en) | Feedback method and apparatus based on online document comment, and non-transitory computer-readable storage medium | |
| CN107679058B (en) | Data synchronization method, device and computer readable storage medium | |
| CN114020850B (en) | Database data synchronization method, device, equipment and readable storage medium | |
| CN108401026B (en) | Block chain-based data storage method, system and computer-readable storage medium | |
| CN106101256B (en) | Method and apparatus for synchrodata | |
| CN111813685B (en) | Automatic test method and device | |
| CN110619100B (en) | Method and apparatus for acquiring data | |
| CN115113989B (en) | Transaction execution method, device, computing equipment and storage medium | |
| CN107330087B (en) | Page file generation method and device | |
| CN115757400B (en) | Data table processing method, device, electronic equipment and computer readable medium | |
| CN110196662B (en) | Method, device, terminal and storage medium for displaying synchronization state | |
| CN111782235A (en) | Data upgrading and querying method and device | |
| CN113220954A (en) | Information display method and device and projection equipment | |
| CN113656050A (en) | Method and device for generating version number | |
| CN113515717B (en) | Navigation page updating method and device, electronic equipment and storage medium | |
| CN114253520A (en) | Interface code generation method and device | |
| CN112015459A (en) | Method and device for upgrading image file, terminal and readable storage medium | |
| CN111787043A (en) | Data request method and device | |
| CN111626802A (en) | Method and apparatus for processing information | |
| CN112422648B (en) | Data synchronization method and system | |
| CN118069174A (en) | A data processing method, a data interaction method and a related device | |
| CN115185911A (en) | Partial document data processing method based on text structured segmentation and cloud storage | |
| CN111784376B (en) | Method and device for processing information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |