CN115952172B - Data matching method and device based on database temporary table - Google Patents
Data matching method and device based on database temporary tableDownload PDFInfo
- Publication number
- CN115952172B CN115952172BCN202310215543.4ACN202310215543ACN115952172BCN 115952172 BCN115952172 BCN 115952172BCN 202310215543 ACN202310215543 ACN 202310215543ACN 115952172 BCN115952172 BCN 115952172B
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- temporary
- matching
- thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034methodMethods0.000titleclaimsabstractdescription30
- 230000008569processEffects0.000description6
- 230000000694effectsEffects0.000description5
- 238000004590computer programMethods0.000description3
- 230000006870functionEffects0.000description3
- 230000001360synchronised effectEffects0.000description3
- 230000008859changeEffects0.000description2
- 125000004122cyclic groupChemical group0.000description1
- 230000007547defectEffects0.000description1
- 238000010586diagramMethods0.000description1
- 238000012986modificationMethods0.000description1
- 230000004048modificationEffects0.000description1
- 238000012544monitoring processMethods0.000description1
- 230000003287optical effectEffects0.000description1
Images
Classifications
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The invention relates to the technical field of database processing, in particular to a data matching method and device based on a temporary table of a database.
Background
In the prior art, any tables in the two databases are compared one by one, so that the method has the defects of low speed, large occupied memory and the like, namely in the prior art, the data matching is generally carried out in a cyclic comparison mode.
In addition, in the data processing process, many current service scenarios depend on the functions of data synchronization and subscription, the link length is different, if some of the environments lose part of the data, the service functions may not be available, and the means for verifying the accuracy of the data content is lacking, so that the current problems mainly include:
1. the invisibility of the synchronous task does not have unified log information, the synchronous task can be traced, the change data is more, and all the data change logs can not be stored.
2. The data loss often causes the problem and finds out the reason again, and sometimes the data loss is passive and lacks monitoring and early warning.
3. The complexity of the link, more synchronous data tables and larger data volume are compared, and no scheme at present can compare the data difference between tables efficiently and accurately.
Thus, how to perform efficient and secure matching of data tables is a technical challenge, and there is a need to minimize the impact on the performance of the system during the data matching process.
Disclosure of Invention
The present invention proposes the following technical solution to one or more of the above technical drawbacks of the prior art.
A data matching method based on a temporary table of a database, the method comprising:
the method comprises the steps of obtaining a first data table of a source database, a second data table of a target database and a data matching range;
judging whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value or not, if not, directly matching the data of the first data table in the data matching range with the data of the second data table in the data matching range to obtain a matching result, and if so, performing temporary table matching;
and a temporary table matching step, namely establishing a first temporary data table in the source database, establishing a second temporary data table and a third temporary data table in the target database, and performing matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result.
Further, the operation of matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result is as follows: and calculating the MD5 value of the data of the first data table in the data matching range, storing the MD5 value of the data of the second data table in the data matching range in a first temporary data table, storing the MD5 value in a second temporary data table, inserting the MD5 value in the first temporary data table into a third temporary data table, and carrying out left link or internal link on the second temporary data table and the third temporary data table to obtain a matching result.
Still further, the matching result includes at least one of: the first data table and the second data table are the same data, the first data table and the second data table are different data, the second data table is more missing data than the first data table, and the second data table is more data than the first data table.
Further, after the matching is completed, initializing a diff thread, a missing thread, an extra thread and a match thread, wherein the diff thread is used for outputting different data in the first data table and the second data table, the missing thread is used for outputting missing data in the second data table compared with the first data table, the extra thread is used for outputting more data in the second data table compared with the first data table, and the match thread is used for outputting the same data in the first data table and the second data table.
Furthermore, initializing corresponding diff queues, missing queues, extra queues and match queues in the memory for the diff threads, missing threads, extra threads and match threads is used for realizing a producer and consumer relation pool, and the size of the memory occupied by the relation pool is as follows:
if the number of the pins is not equal,
is greater than->
，
Then
；
Otherwise the first set of parameters is selected,
；
wherein ,
=1, 2, 3, 4 represent diff, missing, extra, and match queues, respectively, +_>
Representing the memory size occupied by the corresponding queue implementation producer and consumer relationship pool,/->
Representing the amount of data generated per time of the corresponding queue,/->
Representing the amount of data consumed per time of the corresponding queue,/->
Indicating the total data that the corresponding queue needs to outputQuantity (S)>
Representing the total time required for the total data amount output by the corresponding queue.
The invention also provides a data matching device based on the temporary table of the database, which comprises:
the acquisition unit acquires a first data table of the source database, a second data table of the target database and a data matching range;
the judging unit is used for judging whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value or not, if not, directly matching the data of the first data table in the data matching range with the data of the second data table in the data matching range to obtain a matching result, and if so, performing temporary table matching;
and the temporary table matching unit is used for establishing a first temporary data table in the source database, establishing a second temporary data table and a third temporary data table in the target database, and performing matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result.
Further, the operation of matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result is as follows: and calculating the MD5 value of the data of the first data table in the data matching range, storing the MD5 value in the first temporary data table, calculating the MD5 value of the data of the second data table in the data matching range, storing the MD5 value in the first temporary data table in the second temporary data table, inserting the MD5 value in the first temporary data table into the third temporary data table, and carrying out left link or internal link on the second temporary data table and the third temporary data table to obtain a matching result.
Still further, the matching result includes at least one of: the first data table and the second data table are the same data, the first data table and the second data table are different data, the second data table is more missing data than the first data table, and the second data table is more data than the first data table.
Further, after the matching is completed, initializing a diff thread, a missing thread, an extra thread and a match thread, wherein the diff thread is used for outputting different data in the first data table and the second data table, the missing thread is used for outputting missing data in the second data table compared with the first data table, the extra thread is used for outputting more data in the second data table compared with the first data table, and the match thread is used for outputting the same data in the first data table and the second data table.
Furthermore, initializing corresponding diff queues, missing queues, extra queues and match queues in the memory for the diff threads, missing threads, extra threads and match threads is used for realizing a producer and consumer relation pool, and the size of the memory occupied by the relation pool is as follows:
if the number of the pins is not equal,
is greater than->
，
Then
；
Otherwise the first set of parameters is selected,
；
wherein ,
=1, 2, 3, 4 represent diff, missing, extra, and match queues, respectively, +_>
Representing the memory size occupied by the corresponding queue implementation producer and consumer relationship pool,/->
Representing the amount of data generated per time of the corresponding queue,/->
Representing the amount of data consumed per time of the corresponding queue,/->
Indicating the total data amount that the corresponding queue needs to output, < >>
Representing the total time required for the total data amount output by the corresponding queue.
The invention also proposes a computer readable storage medium having stored thereon computer program code which, when executed by a computer, performs any of the methods described above.
The invention has the technical effects that: the invention discloses a data matching method, a device and a storage medium based on a temporary database table, wherein the method comprises the following steps: the method comprises the steps of obtaining a first data table of a source database, a second data table of a target database and a data matching range; judging whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value or not, if not, directly matching the data of the first data table in the data matching range with the data of the second data table in the data matching range to obtain a matching result, and if so, performing temporary table matching; and a temporary table matching step, namely establishing a first temporary data table in the source database, establishing a second temporary data table and a third temporary data table in the target database, and performing matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result. In the invention, the matching range of the data is designated by the user, so that all data in all data tables are prevented from being matched, the calculated amount during data matching is reduced, and the data matching efficiency is improved. The space is saved, after the client side exits from the session, the temporary table can be automatically drop, and no data information occupies the database space; privacy, the client establishes a temporary table which is only used for specific transactions, and the table has special purpose and privacy and does not need to be shared for other transactions; the invention further improves the data security and improves the data matching efficiency by calculating the MD5 value of the data in the corresponding data matching range of the first data table and the second data table, and inserting the MD5 value in the first temporary data table of the source database (namely the source end) into the third temporary data table positioned on the target database (namely the target segment), and matching the MD5 value in the second temporary data table and the third temporary data table by the target data, thereby completing the matching of the data in the corresponding data matching range of the first data table and the second data table, and further improving the data security and the data matching efficiency by storing the MD5 value of the corresponding data in the temporary data table instead of the data per se.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings.
Fig. 1 is a flowchart of a data matching method based on a database temporary table according to an embodiment of the present invention.
Fig. 2 is a block diagram of a data matching apparatus based on a database temporary table according to an embodiment of the present invention.
Detailed Description
The present application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 shows a data matching method based on a temporary table of a database, the method comprising:
in the step S101, a first data table of the source database and a second data table of the target database, and a data matching range are obtained, where the data matching range may be two data tables with the same table name specified by the user, or may be two data tables with different table names, or may be a row or a column in two data tables, for example, a 5 th row in the first data table matches a 7 th row in the second data table, or a 3 rd column in the first data table matches a 7 th column in the second data table.
Step S102, judging whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value, if not, directly matching the data of the first data table in the data matching range with the data of the second data table in the data matching range to obtain a matching result, and if so, performing temporary table matching;
and step S103, a first temporary data table is established in the source database, a second temporary data table and a third temporary data table are established in the target database, and matching is performed based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result.
In the invention, a first data table of a source database, a second data table of a target database and a data matching range are firstly obtained, then whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value is judged, if not, the data of the first data table in the data matching range is directly matched with the data of the second data table in the data matching range to obtain a matching result, if so, a first temporary data table is built in the source database, a second temporary data table and a third temporary data table are built in the target database, and the matching result is obtained based on the first temporary data table, the second temporary data table and the third temporary data table. In the invention, the matching range of the data is designated by the user, so that all the data in all the data tables are prevented from being matched, the calculated amount during data matching is reduced, the data matching efficiency is improved, and the matching range of the data can be set by the user in a GUI (graphical user interface), command line and other modes. In the invention, the matching is also based on judging the size of the matched data volume, when the data volume is smaller, the matching is directly carried out, when the data volume is larger, the matching is carried out based on a temporary table, and when the temporary table is matched, the computing matching is carried out based on an MD5 value, and as the temporary matching is adopted, the following technical effects are achieved: the space is saved, after the client side exits from the session, the temporary table can be automatically drop, and no data information occupies the database space; privacy, the client establishes a temporary table which is only used for specific transactions, and the table has special purpose and privacy and does not need to be shared for other transactions; the efficiency is high, and the temporary table established by the client has independent operation and read-write performance, so that the processing speed and the processing efficiency are higher, which is another important invention point of the invention.
In a further embodiment, the first temporary data table, the second temporary data table and the third temporary data table are set in the memory to be accessible only by the corresponding process that created them, and other processes cannot access them, i.e. the clients (the source database, the client of the target database) establish the temporary table to serve only specific transactions, which table has the special purpose and the privacy, and does not need to be shared to other transactions, which improves the security of the data, which is another important invention point of the present invention.
In a further embodiment, the operation of matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result is: and calculating the MD5 value of the data of the first data table in the data matching range, storing the MD5 value of the data of the second data table in the data matching range in a first temporary data table, storing the MD5 value of the data of the second data table in a second temporary data table, inserting the MD5 value in the first temporary data table into a third temporary data table, and carrying out left link, internal link or right link on the second temporary data table and the third temporary data table to obtain a matching result.
In the invention, the MD5 value of the data in the corresponding data matching range of the first data table and the second data table is calculated and then written into the first temporary data table and the second temporary data table, and the MD5 value in the first temporary data table of the source database (namely the source end) is inserted into the third temporary data table positioned on the target database (namely the target section), and the MD5 value in the second temporary table and the third temporary table is matched on the target data, so that the matching of the data in the corresponding data matching range of the first data table and the second data table is completed.
In a further embodiment, the matching result comprises at least one of: the first data table and the second data table are the same data, the first data table and the second data table are different data, the second data table is more missing data than the first data table, and the second data table is more data than the first data table. Based on the matching results, the data synchronization of the source end and the target end can be performed.
In a further embodiment, after the matching is completed, a diff thread, a missing thread, an extra thread and a match thread are initialized, the diff thread is used for outputting data different from the first data table and the second data table, the missing thread is used for outputting data missing from the second data table than the first data table, the extra thread is used for outputting data more from the second data table than the first data table, and the match thread is used for outputting data identical to the first data table and the second data table. In the invention, by initializing the corresponding threads, the threads can run in parallel, thereby realizing the output of different matched data results and improving the data output efficiency, which is another important invention point of the invention.
In a further embodiment, for the diff thread, the missing thread, the extra thread, and the match thread, initializing corresponding diff queues, missing queues, extra queues, and match queues in a memory, where the relationship pool is used to implement a producer-consumer relationship pool, and the memory size occupied by the relationship pool is:
if the number of the pins is not equal,
is greater than->
，/>
Then
；
Otherwise the first set of parameters is selected,
；
wherein ,
=1, 2, 3, 4 represent diff, missing, extra, and match queues, respectively, +_>
Representing the memory size occupied by the corresponding queue implementation producer and consumer relationship pool,/->
Representing the amount of data generated per time of the corresponding queue,/->
Representing the amount of data consumed per time of the corresponding queue,/->
Indicating the total data amount that the corresponding queue needs to output, < >>
Representing the total time required for the total data amount output by the corresponding queue.
In the invention, in order to prevent the loss during data output, a producer consumer working mode is simulated through the initialized corresponding queue, so as to achieve the technical effects of data peak clipping, valley filling and decoupling.
Fig. 2 shows a data matching device based on a temporary table of a database of the present invention, the device comprising:
the obtaining unit 201 obtains a first data table of the source database and a second data table of the target database, and a data matching range, where the data matching range may be two data tables with the same table names specified by a user, or may be two data tables with different table names, or may be a row or a column in the two data tables, for example, a 5 th row in the first data table matches a 7 th row in the second data table, or a 3 rd column in the first data table matches a 7 th column in the second data table.
The judging unit 202 is configured to judge whether the data amounts of the first data table and the second data table located in the data matching range are both greater than a first threshold, if not, directly match the data of the first data table located in the data matching range with the data of the second data table located in the data matching range to obtain a matching result, and if so, perform temporary table matching;
and a temporary table matching unit 203, configured to establish a first temporary data table in the source database, establish a second temporary data table and a third temporary data table in the target database, and perform matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result.
In the invention, a first data table of a source database, a second data table of a target database and a data matching range are firstly obtained, then whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value is judged, if not, the data of the first data table in the data matching range is directly matched with the data of the second data table in the data matching range to obtain a matching result, if so, a first temporary data table is built in the source database, a second temporary data table and a third temporary data table are built in the target database, and the matching result is obtained based on the first temporary data table, the second temporary data table and the third temporary data table. In the invention, the matching range of the data is designated by the user, so that all the data in all the data tables are prevented from being matched, the calculated amount during data matching is reduced, the data matching efficiency is improved, and the matching range of the data can be set by the user in a GUI (graphical user interface), command line and other modes. In the invention, the matching is also based on judging the size of the matched data volume, when the data volume is smaller, the matching is directly carried out, when the data volume is larger, the matching is carried out based on a temporary table, and when the temporary table is matched, the computing matching is carried out based on an MD5 value, and as the temporary matching is adopted, the following technical effects are achieved: the space is saved, after the client side exits from the session, the temporary table can be automatically drop, and no data information occupies the database space; privacy, the client establishes a temporary table which is only used for specific transactions, and the table has special purpose and privacy and does not need to be shared for other transactions; the efficiency is high, and the temporary table established by the client has independent operation and read-write performance, so that the processing speed and the processing efficiency are higher, which is another important invention point of the invention.
In a further embodiment, the first temporary data table, the second temporary data table and the third temporary data table are set in the memory to be accessible only by the corresponding process that created them, and other processes cannot access them, i.e. the clients (the source database, the client of the target database) establish the temporary table to serve only specific transactions, which table has the special purpose and the privacy, and does not need to be shared to other transactions, which improves the security of the data, which is another important invention point of the present invention.
In a further embodiment, the operation of matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result is: and calculating the MD5 value of the data of the first data table in the data matching range, storing the MD5 value of the data of the second data table in the data matching range in a first temporary data table, storing the MD5 value of the data of the second data table in a second temporary data table, inserting the MD5 value in the first temporary data table into a third temporary data table, and carrying out left link, internal link or right link on the second temporary data table and the third temporary data table to obtain a matching result.
In the invention, the MD5 value of the data in the corresponding data matching range of the first data table and the second data table is calculated and then written into the first temporary data table and the second temporary data table, and the MD5 value in the first temporary data table of the source database (namely the source end) is inserted into the third temporary data table positioned on the target database (namely the target section), and the MD5 value in the second temporary table and the third temporary table is matched on the target data, so that the matching of the data in the corresponding data matching range of the first data table and the second data table is completed.
In a further embodiment, the matching result comprises at least one of: the first data table and the second data table are the same data, the first data table and the second data table are different data, the second data table is more missing data than the first data table, and the second data table is more data than the first data table. Based on the matching results, the data synchronization of the source end and the target end can be performed.
In a further embodiment, after the matching is completed, a diff thread, a missing thread, an extra thread and a match thread are initialized, the diff thread is used for outputting data different from the first data table and the second data table, the missing thread is used for outputting data missing from the second data table than the first data table, the extra thread is used for outputting data more from the second data table than the first data table, and the match thread is used for outputting data identical to the first data table and the second data table. In the invention, by initializing the corresponding threads, the threads can run in parallel, thereby realizing the output of different matched data results and improving the data output efficiency, which is another important invention point of the invention.
In a further embodiment, for the diff thread, the missing thread, the extra thread, and the match thread, initializing corresponding diff queues, missing queues, extra queues, and match queues in a memory, where the relationship pool is used to implement a producer-consumer relationship pool, and the memory size occupied by the relationship pool is:
if the number of the pins is not equal,
is greater than->
，
Then
；/>
Otherwise the first set of parameters is selected,
；
wherein ,
=1, 2, 3, 4 represent diff queue, missing queue, extra queue and match queue respectively,
representing the memory size occupied by the corresponding queue implementation producer and consumer relationship pool,/->
Representing the amount of data generated per time of the corresponding queue,/->
Representing the amount of data consumed per time of the corresponding queue,/->
Indicating the total data amount that the corresponding queue needs to output, < >>
Representing the total time required for the total data amount output by the corresponding queue.
In the invention, in order to prevent the loss during data output, a producer consumer working mode is simulated through the initialized corresponding queue, so as to achieve the technical effects of data peak clipping, valley filling and decoupling.
In one embodiment of the invention a computer storage medium is provided, on which a computer program is stored, which computer storage medium may be a hard disk, DVD, CD, flash memory or the like, which computer program, when being executed by a processor, carries out the above-mentioned method.
For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present application.
From the above description of embodiments, it will be apparent to those skilled in the art that the present application may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the apparatus described in the embodiments or some parts of the embodiments of the present application.
Finally, what should be said is: the above embodiments are merely for illustrating the technical aspects of the present invention, and it should be understood by those skilled in the art that although the present invention has been described in detail with reference to the above embodiments: modifications and equivalents may be made thereto without departing from the spirit and scope of the invention, which is intended to be encompassed by the claims.
Claims (8)
1. A method for matching data based on a temporary database table, the method comprising:
the method comprises the steps of obtaining a first data table of a source database, a second data table of a target database and a data matching range;
judging whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value or not, if not, directly matching the data of the first data table in the data matching range with the data of the second data table in the data matching range to obtain a matching result, and if so, performing temporary table matching;
a temporary table matching step, namely, a first temporary data table is established in the source database, a second temporary data table and a third temporary data table are established in the target database, and matching is carried out on the basis of the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result; the operation of matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result is as follows: and calculating the MD5 value of the data of the first data table in the data matching range, storing the MD5 value in a first temporary data table, calculating the MD5 value of the data of the second data table in the data matching range, storing the MD5 value in a second temporary data table, inserting the MD5 value in the first temporary data table into a third temporary data table, and carrying out left link or internal link on the second temporary data table and the third temporary data table to obtain a matching result.
2. The method of claim 1, wherein the matching result comprises at least one of: the first data table and the second data table are the same data, the first data table and the second data table are different data, the second data table is more missing data than the first data table, and the second data table is more data than the first data table.
3. The method of claim 2, wherein after the matching is completed, a diff thread, a missing thread, an extra thread, and a match thread are initialized, the diff thread is used to output data in the first data table that is different from the data in the second data table, the missing thread is used to output data in the second data table that is missing from the first data table, the extra thread is used to output data in the second data table that is more than the first data table, and the match thread is used to output data in the first data table that is the same as the data in the second data table.
4. The method of claim 3, wherein initializing corresponding diff, missing, extra, and match queues in a memory for the diff, missing, extra, and match threads is used to implement a producer-consumer relationship pool, the relationship pool having a memory size:
if the number of the pins is not equal,
is greater than->
，
Then
；
Otherwise the first set of parameters is selected,
；
wherein ,
=1, 2, 3, 4 represent diff, missing, extra, and match queues, respectively, +_>
Representing the memory size occupied by the corresponding queue implementation producer and consumer relationship pool,/->
Representing the amount of data generated per time of the corresponding queue,/->
Representing the amount of data consumed per time of the corresponding queue,/->
Indicating the total data amount that the corresponding queue needs to output, < >>
Representing the total time required for the total data amount output by the corresponding queue.
5. A data matching device based on a temporary table of a database, the device comprising:
the acquisition unit acquires a first data table of the source database, a second data table of the target database and a data matching range;
the judging unit is used for judging whether the data quantity of the first data table and the second data table in the data matching range is larger than a first threshold value or not, if not, directly matching the data of the first data table in the data matching range with the data of the second data table in the data matching range to obtain a matching result, and if so, performing temporary table matching;
a temporary table matching unit, which is used for establishing a first temporary data table in the source database, establishing a second temporary data table and a third temporary data table in the target database, and performing matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result; the operation of matching based on the first temporary data table, the second temporary data table and the third temporary data table to obtain a matching result is as follows: and calculating the MD5 value of the data of the first data table in the data matching range, storing the MD5 value in a first temporary data table, calculating the MD5 value of the data of the second data table in the data matching range, storing the MD5 value in a second temporary data table, inserting the MD5 value in the first temporary data table into a third temporary data table, and carrying out left link or internal link on the second temporary data table and the third temporary data table to obtain a matching result.
6. The apparatus of claim 5, wherein the match result comprises at least one of: the first data table and the second data table are the same data, the first data table and the second data table are different data, the second data table is more missing data than the first data table, and the second data table is more data than the first data table.
7. The apparatus of claim 6, wherein after the matching is completed, a diff thread, a missing thread, an extra thread, and a match thread are initialized, the diff thread is used to output data in the first data table that is different from the data in the second data table, the missing thread is used to output data in the second data table that is missing from the first data table, the extra thread is used to output data in the second data table that is more than the first data table, and the match thread is used to output data in the first data table that is the same as the data in the second data table.
8. The apparatus of claim 7, wherein initializing corresponding diff, missing, extra, and match queues in memory for the diff, missing, extra, and match threads is used to implement a producer-consumer relationship pool, the relationship pool occupying a memory of the size:
if the number of the pins is not equal,
is greater than->
，
Then
；
Otherwise the first set of parameters is selected,
；
wherein ,
=1, 2, 3, 4 represent diff, missing, extra, and match queues, respectively, +_>
Representing the memory size occupied by the corresponding queue implementation producer and consumer relationship pool,/->
Representing the amount of data generated per time of the corresponding queue,/->
Representing the amount of data consumed per time of the corresponding queue,/->
Indicating the total data amount that the corresponding queue needs to output, < >>
Representing the total time required for the total data amount output by the corresponding queue. />
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310215543.4ACN115952172B (en) | 2023-03-08 | 2023-03-08 | Data matching method and device based on database temporary table |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310215543.4ACN115952172B (en) | 2023-03-08 | 2023-03-08 | Data matching method and device based on database temporary table |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115952172A CN115952172A (en) | 2023-04-11 |
| CN115952172Btrue CN115952172B (en) | 2023-05-26 |
Family
ID=85891154
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310215543.4AActiveCN115952172B (en) | 2023-03-08 | 2023-03-08 | Data matching method and device based on database temporary table |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115952172B (en) |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103530375A (en)* | 2013-10-15 | 2014-01-22 | 北京国双科技有限公司 | Method and device for data source matching |
| CN110770722B (en)* | 2017-06-29 | 2023-08-18 | 北京清影机器视觉技术有限公司 | Two-dimensional data matching method, device and logic circuit |
| CN107665233B (en)* | 2017-07-24 | 2020-07-31 | 深圳壹账通智能科技有限公司 | Database data processing method and device, computer equipment and storage medium |
| CN109408535B (en)* | 2018-09-28 | 2024-04-09 | 中国平安财产保险股份有限公司 | Large data volume matching method, device, computer equipment and storage medium |
| CN110347747A (en)* | 2019-06-14 | 2019-10-18 | 平安科技（深圳）有限公司 | Database data synchronic method, system, computer equipment and storage medium |
| CN110865982A (en)* | 2019-11-19 | 2020-03-06 | 深信服科技股份有限公司 | Data matching method and device, electronic equipment and storage medium |
| CN111949524A (en)* | 2020-08-03 | 2020-11-17 | 北京锐安科技有限公司 | A data interface testing method, device, server and storage medium |
| CN114448869A (en)* | 2020-10-19 | 2022-05-06 | 南京中兴新软件有限责任公司 | Data table synchronization method and device, data exchange equipment and storage medium |
| CN113360503B (en)* | 2021-06-18 | 2023-02-07 | 建信金融科技有限责任公司 | Test data tracking method and device for distributed database |
- 2023
- 2023-03-08CNCN202310215543.4Apatent/CN115952172B/enactiveActive
Also Published As
| Publication number | Publication date |
|---|---|
| CN115952172A (en) | 2023-04-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3678346B1 (en) | Blockchain smart contract verification method and apparatus, and storage medium | |
| CN110442560B (en) | Log replay method, device, server and storage medium | |
| CN113434396A (en) | Interface test method, device, equipment, storage medium and program product | |
| CN111553652B (en) | Service processing method and device | |
| EP3933639B1 (en) | Transaction processing method, apparatus, and electronic device for blockchain | |
| CN109815141B (en) | A test method and device | |
| CN115618239B (en) | Management method, system, terminal and medium for deep learning framework training | |
| US8286141B2 (en) | Instruction-trace generation program, instruction-trace generating device, and instruction-trace generating method | |
| CN112732427B (en) | Data processing method, system and related device based on Redis cluster | |
| CN115952172B (en) | Data matching method and device based on database temporary table | |
| CN114968725A (en) | Task dependency relationship correction method and device, computer equipment and storage medium | |
| CN113342647A (en) | Test data generation method and device | |
| CN118363799A (en) | Method, device, equipment and medium for testing performance problem recurrence of solid state disk | |
| CN117171030A (en) | Method, device, equipment and storage medium for detecting software running environment | |
| CN114443101B (en) | System advanced audit strategy update method, system, terminal and storage medium | |
| CN113077344B (en) | Block chain-based transaction method, device, electronic equipment and storage medium | |
| CN110209512B (en) | Data checking method and device based on multiple data sources | |
| US11513884B2 (en) | Information processing apparatus, control method, and program for flexibly managing event history | |
| CN112685080B (en) | Open source component duplicate checking method, system, device and readable storage medium | |
| CN115858324B (en) | AI-based IT equipment fault processing method, apparatus, equipment and medium | |
| CN112181539B (en) | File processing method, device, equipment and medium | |
| CN118277267B (en) | A method and device for optimizing configuration of dual regularization engines | |
| CN114647562B (en) | A thread analysis method and device | |
| CN119988345A (en) | Method and device for generating rollback log and determining active data table in transaction rollback | |
| CN117370211A (en) | Test method, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |