CN111198910A - Data fusion method and device - Google Patents
Data fusion method and deviceDownload PDFInfo
- Publication number
- CN111198910A CN111198910ACN201811368415.9ACN201811368415ACN111198910ACN 111198910 ACN111198910 ACN 111198910ACN 201811368415 ACN201811368415 ACN 201811368415ACN 111198910 ACN111198910 ACN 111198910A
- Authority
- CN
- China
- Prior art keywords
- field
- data
- target table
- target
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data fusion method and apparatus.
Background
With the development of scientific technology, the data sources for generating data are increasing. In the face of data from different data sources, how to fuse the data of each data source to obtain useful information from the data is a focus of attention in academic and industrial fields.
The current data fusion scheme completely realizes data fusion through codes, however, when the types of data are more, the code implementation completely depends on the codes, which results in huge code amount, and if the types of the data change, the codes for realizing the fusion also need to be modified, so the flexibility of realizing the data fusion completely depending on the codes is also lower.
Disclosure of Invention
In view of this, the present application provides a data fusion method and apparatus, so as to solve the problems of large code amount and low flexibility of implementing data fusion completely depending on codes.
According to a first aspect of embodiments of the present application, there is provided a data fusion method, including:
for each source table, determining mapping configuration information corresponding to the source table based on a stored target table field, wherein the mapping configuration information comprises a mapping relation between the source table field and the target table field and a conversion rule corresponding to the source table field; converting the data of the source table field in the source table into a target table corresponding to the source table by using a conversion rule corresponding to the source table field according to the mapping relation between the source table field and the target table field;
fusing the data of the target table field in each target table obtained by conversion into a final target table according to a main key field of a preset target table;
and the structures of the target table corresponding to each source table and the final target table are consistent.
According to a second aspect of the embodiments of the present application, there is provided a data fusion apparatus, including:
the determining module is used for determining mapping configuration information corresponding to each source table based on a stored target table field, wherein the mapping configuration information comprises a mapping relation between the source table field and the target table field and a conversion rule corresponding to the source table field;
the conversion module is used for converting the data of the source table field in the source table into a target table corresponding to the source table by using a conversion rule corresponding to the source table field according to the mapping relation between the source table field and the target table field;
the fusion module is used for fusing the data of the target table fields in each target table obtained by conversion into a final target table according to the primary key fields of the target tables configured in advance;
and the structures of the target table corresponding to each source table and the final target table are consistent.
According to a third aspect of embodiments herein, there is provided an electronic device, the device comprising a readable storage medium and a processor;
wherein the readable storage medium is configured to store machine executable instructions;
the processor is configured to read the machine executable instructions on the readable storage medium and execute the instructions to implement the steps of the method according to the first aspect.
By applying the embodiment of the application, for each source table, mapping configuration information (including mapping relationships between source table fields and target table fields and conversion rules corresponding to the source table fields) corresponding to the source table is determined based on the stored target table fields, so that data in the source table is converted into the target table according to the mapping configuration information, and because the target table corresponding to each source table and the final target table are consistent in structure, the converted data in the target table fields in each target table can be fused into the final target table according to the pre-configured primary key fields of the target table, so that data fusion is realized.
Based on the above description, it can be seen that, by determining the mapping configuration information corresponding to each source table to implement data fusion, the code amount of data fusion can be reduced, and even if the dimension (i.e. the source table field) of the source table changes, only the mapping configuration information needs to be changed, and the code for implementing data fusion does not need to be modified, so that the flexibility of data fusion can be improved.
Drawings
FIG. 1A is a flow chart illustrating an embodiment of a data fusion method according to an exemplary embodiment of the present application;
FIG. 1B is a diagram of a priority configuration page according to the embodiment shown in FIG. 1A;
FIG. 1C is a diagram of a primary key field configuration page according to the embodiment shown in FIG. 1A;
FIG. 2A is a flow chart illustrating an embodiment of another data fusion method according to an exemplary embodiment of the present application;
FIG. 2B is a diagram of a mapping configuration page according to the embodiment shown in FIG. 2A;
FIG. 2C is a diagram of a mapping test page according to the embodiment of FIG. 2A;
FIG. 3 is a diagram illustrating a hardware configuration of an electronic device according to an exemplary embodiment of the present application;
fig. 4 is a block diagram of an embodiment of a data fusion device according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
The data fusion performed at present generally involves the fusion of multi-source heterogeneous data, which refers to: the data generated by each data source is recorded by a data table containing a plurality of dimensions (i.e. a plurality of attributes), the data of each dimension in the data table has a respective value range and data type, and since the services served by each data source are different, the attributes contained in each corresponding data table are also different. In the related technology, the data fusion completely depending on the code is low in flexibility and large in code amount.
In order to solve the above problems, the present application provides a data fusion method, for each source table, mapping configuration information (including a mapping relationship between a source table field and a target table field and a conversion rule corresponding to the source table field) corresponding to the source table is determined based on a stored target table field, so as to convert data in the source table to a target table according to the mapping configuration information, and since structures of the target table corresponding to each source table and a final target table are consistent, data of the target table field in each target table obtained by conversion can be fused to the final target table according to a pre-configured primary key field of the target table, so as to implement data fusion.
Based on the above description, it can be seen that, by determining the mapping configuration information corresponding to each source table to implement data fusion, the code amount of data fusion can be reduced, and even if the dimension (i.e. the source table field) of the source table changes, only the mapping configuration information needs to be changed, and the code for implementing data fusion does not need to be modified, so that the flexibility of data fusion can be improved.
The technical solution of the present application will be described in detail with specific examples.
Fig. 1A is a flowchart illustrating an embodiment of a data fusion method according to an exemplary embodiment of the present application, where the data fusion method may be applied to an electronic device, and as shown in fig. 1A, the data fusion method includes the following steps:
step 101: and for each source table, determining mapping configuration information corresponding to the source table based on the stored target table fields, wherein the mapping configuration information comprises the mapping relation between the source table fields and the target table fields and conversion rules corresponding to the source table fields.
The source table field refers to a field included in a source table, the target table field refers to a field included in a target table, a conversion rule in the mapping configuration information may be a Chinese name-changing pinyin (e.g., cn2py (name)), or a gender extracted from an identity card, or an age extracted from an identity card, or a data type conversion, and the like, and a mapping relationship between the source table field and the target table field may be implemented by an SQL language, such as City as address from a, which indicates that the source table field City in the source table a is mapped by the target table field address.
It should be noted that the source table fields in each source table may be consistent or inconsistent, but each source table needs to have a primary key field that is used for uniquely identifying one record and is the same for subsequent data fusion, where the primary key field belongs to the source table field. For example, the source table to be fused is a floating population table and a student archive table, and the source table fields in the floating population table are identity cards, names, ages, sexes and addresses; the source table fields in the student archive table comprise schools, classes, school numbers, names, identity cards and genders, and the main key fields in the floating population table and the student archive table are used for uniquely identifying one record and are identity cards.
For the process of determining the mapping configuration information corresponding to the source table based on the stored target table field, reference may be made to the following description of the embodiment shown in fig. 2A, and details of the process are not described here.
Step 102: and converting the data of the source table field in the source table into the target table corresponding to the source table by using the conversion rule corresponding to the source table field according to the mapping relation between the source table field and the target table field corresponding to the source table.
Where each source table corresponds to a target table, which may be generated using stored target table fields. It should be noted that the structures of the target tables corresponding to each source table are consistent, that is, the fields and field types of the target tables are consistent.
In an exemplary scenario, as shown in table 1, an exemplary employee source table is shown, where the source table fields have names, ages, sexes, positions, and numbers, and the corresponding target table fields in the target table have names, ages, and cities, it is assumed that mapping configuration information corresponding to table 1 includes mapping relationships between names and names, a conversion rule corresponding to names is a chinese name-pinyin, a mapping relationship between ages and ages, and a conversion rule corresponding to ages is a copy. The target table obtained after conversion according to the mapping configuration information corresponding to table 1 is shown in table 2.
| Name (I) | Age (age) | Sex | Position of employment | Numbering | |
| Zhang three | 25 | For male | Teacher's | 1 | |
| Li four | 26 | Doctor | 2 | ||
| 42 | Doctor | 3 |
TABLE 1
| Name | Age | City |
| Zhangsan | 25 | |
| Lisi | 26 | |
| 42 |
TABLE 2
Step 103: and fusing the data contained in the target table field in each target table obtained by conversion into a final target table according to the primary key field of the pre-configured target table.
In one embodiment, prior to performingstep 103, the field priority may be preconfigured, which may be implemented by: when a priority configuration command input from the outside is received, determining target table fields selected from the outside, and acquiring and displaying source table fields corresponding to the target table fields from mapping configuration information corresponding to each source table; and receiving and storing externally input priority configuration information aiming at the source table field corresponding to the target table field.
The user can send a priority configuration command by triggering a configuration priority button, the priority configuration page is displayed after the priority configuration command is received by the equipment, and the equipment displays a source table field corresponding to a target table field after the user selects the target table field in the page, so that the user can perform priority configuration. In addition, when the source table fields are displayed, the source table name to which each source table field belongs and the mapping rule corresponding to each source table field can be displayed, and therefore a user can conveniently carry out priority configuration according to actual requirements.
In an exemplary scenario, as shown in fig. 1B, the destination table field selected by the user in the priority configuration page is "AGE", the "AGE" field corresponds to "AGE", "PATIENT _ AGE", and "AGE" of the source table fields in the three source tables, respectively, and the user performs priority configuration by adjusting the "top", "up", and "down" buttons corresponding to each source table field, and the larger the value of the priority is, the more preferred the data of the source table field is to be fetched in the fusion.
In one embodiment, beforestep 103 is performed, the primary key field needs to be configured, which may be implemented as: when receiving externally input primary key field configuration commands, displaying the stored target table fields, and when receiving externally selected target table fields, determining the target table fields as the primary key fields.
The user can send a main key field configuration command by triggering a 'configure main key field' button, and the device displays the stored target table field after receiving the main key field configuration command, so that the user can select the target table field as the main key field according to actual requirements. As shown in FIG. 1C, for an exemplary primary key field configuration page, the target table fields shown in FIG. 1C have "NAME", "AGE" and "EDUBACK", from which the user can select one target table field having a mapping relationship in each source table as the primary key field.
It should be noted that, because the primary key field is used to uniquely identify a record, the user needs to select a target table field having a mapping relationship in the mapping configuration information of each source table as the primary key field in the target table field, so as to facilitate subsequent merging and use.
In an embodiment, for the process ofstep 103, the data contained in the primary key field may be obtained from each target table, and the obtained data is added to the final target table, and then for each data contained in the primary key field, if the data appears in only one target table, a record corresponding to the data is obtained from the one target table, and the record corresponding to the data is added to the final target table; and if the data appears in a plurality of target tables, acquiring records corresponding to the data from the plurality of target tables, determining data of each target field except the primary key field from the acquired records, and adding the determined data of each target table field corresponding to the data to a final target table.
And the final target table is consistent with the structure of the target table obtained through conversion. If the data contained in the primary key field appears in only one target table, only one record corresponding to the data is shown, the record corresponding to the data is directly added to the final target table, and if the data contained in the primary key field appears in a plurality of target tables, a plurality of records corresponding to the data are shown, the data of each target field except the primary key field needs to be determined from the plurality of records and added, so that the fusion purpose is achieved.
In an embodiment, for the data process of determining each target field except the primary key field from the obtained plurality of records, a record containing data of the target table field may be determined from the obtained plurality of records for each target table field except the primary key field, and when one record is determined, the data of the target table field in the one record is determined as the data of the target table field; when a plurality of records are determined, determining priority configuration information of a source table field corresponding to the target table field, selecting the source table field with the highest priority from the priority configuration information, acquiring the record corresponding to the source table field from the determined plurality of records, and determining data of the target table field in the record as data of the target table field; the record corresponding to the source table field is located in the target table corresponding to the source table where the source table field with the highest priority is located.
In an exemplary scenario, as shown in tables 3 and 4, for two target tables obtained by conversion, the target table field id is a primary key field, and it is assumed that the priority of the source table field corresponding to the target table field name in table 3 configured by the user is 1, and the priority of the source table field corresponding to the target table field name in table 4 is 2. Since the data 1000 contained in the primary key field id appears in both table 3 and table 4, 2 records can be obtained, and the data of the target table field name, address, and chief need to be determined from the 2 records respectively. For the name of the field of the target table, it may be determined that 2 records of the data containing the name exist, and since the target table corresponding to the source table where the field of the source table with the highest priority is located is table 4, the record corresponding to the field of the source table with the highest priority is: table 4-name ═ Hikvision; chief is Hu, and the data of the field name of the target table in the record is Hikvision; for the address of the field of the target table, it can be determined that the record of the data containing the address is 1: table 3-name ═ Hik; address is Hangzhou, so that the data of the address of the target table field can be determined to be Hangzhou; for the target table field chief, since it can be determined that the record of data containing chief is 1: table 4-name ═ Hikvision; since chief is Hu, it is determined that data of the target table field chief is Hu, and thus records of 2 primary key fields whose data is 1000 can be merged into the final target table as shown in table 5.
| id | name | address | chief |
| 1000 | Hik | Hangzhou |
TABLE 3
| id | name | address | chief |
| 1000 | Hikvision | Hu |
TABLE 4
| id | name | address | chief |
| 1000 | Hikvision | Hangzhou | Hu |
TABLE 5
In the embodiment of the present application, for each source table, mapping configuration information (including a mapping relationship between a source table field and a target table field and a conversion rule corresponding to the source table field) corresponding to the source table is determined based on a stored target table field, so as to convert data in the source table to a target table according to the mapping configuration information, and since structures of a target table corresponding to each source table and a final target table are consistent, data of a target table field in each target table obtained by conversion can be fused to the final target table according to a primary key field of a preconfigured target table, so as to implement data fusion.
Based on the above description, it can be seen that, by determining the mapping configuration information corresponding to each source table to implement data fusion, the code amount of data fusion can be reduced, and even if the dimension (i.e. the source table field) of the source table changes, only the mapping configuration information needs to be changed, and the code for implementing data fusion does not need to be modified, so that the flexibility of data fusion can be improved.
Fig. 2A is a flowchart of another data fusion method according to an exemplary embodiment of the present application, and based on the embodiment shown in fig. 1A, this embodiment exemplarily illustrates how to determine mapping configuration information corresponding to each source table for each source table. As shown in fig. 2A, the data fusion method further includes the following steps:
step 201: and acquiring and displaying the source table field and the stored target table field in the source table.
In an embodiment, when the source table field in the source table is acquired, the data type of the source table field may also be determined, and the source table field and the data type are output and displayed. In addition, the data type of the target table field may be displayed along with the stored target table field.
The data type may include a character type, an integer type, and the like. Because the target table field is stored locally, the structure of the corresponding target table can be automatically completed for each source table without editing by a user.
Step 202: and receiving and storing the mapping relation between the externally input source table field and the target table field and the conversion rule corresponding to the source table field.
In an embodiment, after receiving the mapping relationship between the source table field and the target table field and the conversion rule corresponding to the source table field, it may be detected whether the data type of the source table field is consistent with the data type of the target table field, and if not, prompt information indicating that the data types are inconsistent is output, so that a user may modify the data type of the target table field by using the data type output by the conversion rule, or continue to input the conversion rule for data type conversion.
In an exemplary scenario, as shown in fig. 2B, the source table list column contains 2 source tables, and a user may switch between the 2 source tables to configure mapping configuration information of each source table; the source table display column displays the source table field and the data type of the currently selected source table; the target table display column displays the stored target table fields and data types, and after configuration is completed, the background generates a target table corresponding to the currently selected source table for subsequent conversion of data in the source table into the target table. Connecting the source table field NAME with the target table field NAME to trigger the generation of a mapping relation between the source table field NAME and the target table field NAME, and inputting a conversion rule in an edit box of a connecting line: cn2py (NAME), connecting the source table field RN and the target table field AGE to trigger the generation of the mapping relationship between the two, and inputting the conversion rule in the edit box of the connecting line: sfz _ to _ birthday _ convert (RN), which shows the conversion rule for extracting the age from the ID card.
It should be noted that, for each source table, after the mapping configuration information is configured, a mapping test may be performed to verify whether the mapping configuration information meets the actual requirement, and the implementation process may be: randomly acquiring a preset amount of data from a source table, outputting and displaying the data, converting the acquired data into a target table according to mapping configuration information corresponding to the source table when an externally input test instruction is received, and displaying the converted target table for comparison and verification by a user.
In another exemplary scenario, as shown in fig. 2C, the left area of fig. 2C shows data of the source table 1, and after the user triggers the "sample test" button, the data of the target table 1 shown in the right area of fig. 2C is obtained, and it can be verified whether the mapping configuration information of the source table 1 meets the actual requirement by comparing the source table 1 with the target table 1.
By this, the flow of fig. 2A is completed, and the configuration of the mapping configuration information of the source table can be realized by the flow of fig. 2A.
Fig. 3 is a hardware block diagram of an electronic device according to an exemplary embodiment of the present application, where the electronic device includes: acommunication interface 301, aprocessor 302, a machine-readable storage medium 303, and abus 304; wherein thecommunication interface 301, theprocessor 302 and the machine-readable storage medium 103 communicate with each other via abus 304. Theprocessor 302 may execute the data fusion method described above by reading and executing machine executable instructions corresponding to the control logic of the data fusion method in the machinereadable storage medium 303, and the specific content of the method is described in the above embodiments, which will not be described herein again.
The machine-readable storage medium 303 referred to herein may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: volatile memory, non-volatile memory, or similar storage media. In particular, the machine-readable storage medium 303 may be a RAM (random Access Memory), a flash Memory, a storage drive (e.g., a hard drive), any type of storage disk (e.g., an optical disk, a DVD, etc.), or similar storage medium, or a combination thereof.
Fig. 4 is a block diagram of an embodiment of a data fusion device according to an exemplary embodiment of the present application, and as shown in fig. 4, the data fusion device includes:
a determiningmodule 410, configured to determine, for each source table, mapping configuration information corresponding to the source table based on a stored target table field, where the mapping configuration information includes a mapping relationship between the source table field and the target table field and a conversion rule corresponding to the source table field;
aconversion module 420, configured to convert, according to a mapping relationship between the source table field and the target table field, data in the source table field into a target table corresponding to the source table by using a conversion rule corresponding to the source table field;
thefusion module 430 is configured to fuse, according to a primary key field of a preconfigured target table, data of the target table field in each target table obtained through conversion into a final target table;
and the structures of the target table corresponding to each source table and the final target table are consistent.
In an optional implementation manner, the determiningmodule 410 is specifically configured to obtain and display a source table field and a stored target table field in the source table; and receiving and storing the mapping relation between the externally input source table field and the target table field and the conversion rule corresponding to the source table field.
In an alternative implementation, the apparatus further comprises (not shown in fig. 4):
the priority configuration module is used for determining externally selected target table fields when receiving an externally input priority configuration command before the fusion module fuses the data contained in the target table fields in each target table obtained by conversion into a final target table according to the primary key fields of the pre-configured target tables, and acquiring and displaying the source table fields corresponding to the target table fields from the mapping configuration information corresponding to each source table; and receiving and storing externally input priority configuration information aiming at the source table field corresponding to the target table field.
In an optional implementation manner, thefusion module 430 is specifically configured to obtain data included in the primary key field from each target table, and add the obtained data to the final target table; for each data contained in the primary key field, if the data only appears in one target table, acquiring a record corresponding to the data from the target table, and adding the data corresponding to the record to a final target table; and if the data appears in a plurality of target tables, acquiring records corresponding to the data from the plurality of target tables, determining data of each target field except the primary key field from the acquired records, and adding the determined data of each target table field corresponding to the data to a final target table.
In an optional implementation manner, thefusion module 430 is further specifically configured to, in the process of determining data of each target field except for the primary key field from the obtained multiple records, determine, for each target table field except for the primary key field, a record including the data of the target table field from the multiple records; when one record is determined, determining the data of the target table field in the record as the data of the target table field; when a plurality of records are determined, determining priority configuration information of a source table field corresponding to the target table field, selecting the source table field with the highest priority from the priority configuration information, acquiring the record corresponding to the source table field from the determined plurality of records, and determining data of the target table field in the record as data of the target table field; the record corresponding to the source table field is located in the target table corresponding to the source table where the source table field with the highest priority is located.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (11)
1. A method of data fusion, the method comprising:
for each source table, determining mapping configuration information corresponding to the source table based on a stored target table field, wherein the mapping configuration information comprises a mapping relation between the source table field and the target table field and a conversion rule corresponding to the source table field; converting the data of the source table field in the source table into a target table corresponding to the source table by using a conversion rule corresponding to the source table field according to the mapping relation between the source table field and the target table field;
fusing the data of the target table field in each target table obtained by conversion into a final target table according to a main key field of a preset target table;
and the structures of the target table corresponding to each source table and the final target table are consistent.
2. The method of claim 1, wherein determining, for each source table, mapping configuration information corresponding to the source table based on the stored target table field comprises:
acquiring and displaying a source table field and a stored target table field in the source table;
and receiving and storing the mapping relation between the externally input source table field and the target table field and the conversion rule corresponding to the source table field.
3. The method of claim 1, wherein before fusing the data contained in the target table field of each target table obtained by conversion into the final target table according to the primary key field of the pre-configured target table, the method further comprises:
when a priority configuration command input from the outside is received, determining target table fields selected from the outside, and acquiring and displaying source table fields corresponding to the target table fields from mapping configuration information corresponding to each source table;
and receiving and storing externally input priority configuration information aiming at the source table field corresponding to the target table field.
4. The method of claim 3, wherein fusing the data contained in the target table field of each target table obtained by conversion into the final target table according to the primary key field of the pre-configured target table comprises:
acquiring data contained in the primary key field from each target table, and adding the acquired data to a final target table;
for each data contained in the primary key field, if the data only appears in one target table, acquiring a record corresponding to the data from the target table, and adding the data corresponding to the record to a final target table;
and if the data appears in a plurality of target tables, acquiring records corresponding to the data from the plurality of target tables, determining data of each target field except the primary key field from the acquired records, and adding the determined data of each target table field corresponding to the data to a final target table.
5. The method of claim 4, wherein determining data for each target field other than the primary key field from the obtained plurality of records comprises:
for each target table field except the primary key field, determining a record containing the data of the target table field from the plurality of records;
when one record is determined, determining the data of the target table field in the record as the data of the target table field;
when a plurality of records are determined, determining priority configuration information of a source table field corresponding to the target table field, selecting the source table field with the highest priority from the priority configuration information, acquiring the record corresponding to the source table field from the determined plurality of records, and determining data of the target table field in the record as data of the target table field; the record corresponding to the source table field is located in the target table corresponding to the source table where the source table field with the highest priority is located.
6. A data fusion apparatus, the apparatus comprising:
the determining module is used for determining mapping configuration information corresponding to each source table based on a stored target table field, wherein the mapping configuration information comprises a mapping relation between the source table field and the target table field and a conversion rule corresponding to the source table field;
the conversion module is used for converting the data of the source table field in the source table into a target table corresponding to the source table by using a conversion rule corresponding to the source table field according to the mapping relation between the source table field and the target table field;
the fusion module is used for fusing the data of the target table fields in each target table obtained by conversion into a final target table according to the primary key fields of the target tables configured in advance;
and the structures of the target table corresponding to each source table and the final target table are consistent.
7. The apparatus according to claim 6, wherein the determining module is specifically configured to obtain and display a source table field and a stored target table field in the source table; and receiving and storing the mapping relation between the externally input source table field and the target table field and the conversion rule corresponding to the source table field.
8. The apparatus of claim 6, further comprising:
the priority configuration module is used for determining externally selected target table fields when receiving an externally input priority configuration command before the fusion module fuses the data contained in the target table fields in each target table obtained by conversion into a final target table according to the primary key fields of the pre-configured target tables, and acquiring and displaying the source table fields corresponding to the target table fields from the mapping configuration information corresponding to each source table; and receiving and storing externally input priority configuration information aiming at the source table field corresponding to the target table field.
9. The apparatus according to claim 8, wherein the fusion module is specifically configured to obtain data included in the primary key field from each target table, and add the obtained data to the final target table; for each data contained in the primary key field, if the data only appears in one target table, acquiring a record corresponding to the data from the target table, and adding the data corresponding to the record to a final target table; and if the data appears in a plurality of target tables, acquiring records corresponding to the data from the plurality of target tables, determining data of each target field except the primary key field from the acquired records, and adding the determined data of each target table field corresponding to the data to a final target table.
10. The apparatus according to claim 9, wherein the fusion module is further specifically configured to, in the process of determining data of each target field except for the primary key field from the obtained plurality of records, determine, for each target table field except for the primary key field, a record including the data of the target table field from the plurality of records; when one record is determined, determining the data of the target table field in the record as the data of the target table field; when a plurality of records are determined, determining priority configuration information of a source table field corresponding to the target table field, selecting the source table field with the highest priority from the priority configuration information, acquiring the record corresponding to the source table field from the determined plurality of records, and determining data of the target table field in the record as data of the target table field; the record corresponding to the source table field is located in the target table corresponding to the source table where the source table field with the highest priority is located.
11. An electronic device, characterized in that the device comprises a readable storage medium and a processor;
wherein the readable storage medium is configured to store machine executable instructions;
the processor configured to read the machine executable instructions on the readable storage medium and execute the instructions to implement the steps of the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811368415.9ACN111198910B (en) | 2018-11-16 | 2018-11-16 | Data fusion method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811368415.9ACN111198910B (en) | 2018-11-16 | 2018-11-16 | Data fusion method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111198910Atrue CN111198910A (en) | 2020-05-26 |
| CN111198910B CN111198910B (en) | 2023-09-26 |
Family
ID=70744010
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811368415.9AActiveCN111198910B (en) | 2018-11-16 | 2018-11-16 | Data fusion method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111198910B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625545A (en)* | 2020-06-08 | 2020-09-04 | 北京创鑫旅程网络技术有限公司 | Data fusion method, data fusion device and storage medium |
| CN112597168A (en)* | 2020-12-28 | 2021-04-02 | 恩亿科(北京)数据科技有限公司 | Processing method, device and platform of multi-source customer data and storage medium |
| CN114327419A (en)* | 2020-10-10 | 2022-04-12 | 中兴通讯股份有限公司 | Model conversion method, device, electronic device, storage medium |
| CN114547038A (en)* | 2022-02-25 | 2022-05-27 | 杭州迪普科技股份有限公司 | Data processing method and device for priority database |
| CN114911750A (en)* | 2021-02-07 | 2022-08-16 | 北京三快在线科技有限公司 | Data conversion method and device |
| CN116431614A (en)* | 2023-04-26 | 2023-07-14 | 北京计算机技术及应用研究所 | A multi-source data governance method based on unsupervised learning |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080313204A1 (en)* | 2007-06-14 | 2008-12-18 | Colorquick, L.L.C. | Method and apparatus for database mapping |
| CN105069033A (en)* | 2015-07-22 | 2015-11-18 | 北京京东尚科信息技术有限公司 | Method and device for creating database table model |
| CN105589968A (en)* | 2015-12-25 | 2016-05-18 | 中国银联股份有限公司 | Data summarization system and method |
| CN106033436A (en)* | 2015-03-13 | 2016-10-19 | 中国石油化工股份有限公司 | Merging method for database |
| CN106055654A (en)* | 2016-06-01 | 2016-10-26 | 东软集团股份有限公司 | Heterogeneous data integration method and device |
| CN106909689A (en)* | 2017-03-07 | 2017-06-30 | 山东浪潮云服务信息科技有限公司 | A kind of data fusion method and device |
| CN107169094A (en)* | 2017-05-12 | 2017-09-15 | 北京小米移动软件有限公司 | information aggregation method and device |
- 2018
- 2018-11-16CNCN201811368415.9Apatent/CN111198910B/enactiveActive
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080313204A1 (en)* | 2007-06-14 | 2008-12-18 | Colorquick, L.L.C. | Method and apparatus for database mapping |
| CN106033436A (en)* | 2015-03-13 | 2016-10-19 | 中国石油化工股份有限公司 | Merging method for database |
| CN105069033A (en)* | 2015-07-22 | 2015-11-18 | 北京京东尚科信息技术有限公司 | Method and device for creating database table model |
| CN105589968A (en)* | 2015-12-25 | 2016-05-18 | 中国银联股份有限公司 | Data summarization system and method |
| CN106055654A (en)* | 2016-06-01 | 2016-10-26 | 东软集团股份有限公司 | Heterogeneous data integration method and device |
| CN106909689A (en)* | 2017-03-07 | 2017-06-30 | 山东浪潮云服务信息科技有限公司 | A kind of data fusion method and device |
| CN107169094A (en)* | 2017-05-12 | 2017-09-15 | 北京小米移动软件有限公司 | information aggregation method and device |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111625545A (en)* | 2020-06-08 | 2020-09-04 | 北京创鑫旅程网络技术有限公司 | Data fusion method, data fusion device and storage medium |
| CN111625545B (en)* | 2020-06-08 | 2023-09-01 | 北京创鑫旅程网络技术有限公司 | Data fusion method, data fusion device and storage medium |
| CN114327419A (en)* | 2020-10-10 | 2022-04-12 | 中兴通讯股份有限公司 | Model conversion method, device, electronic device, storage medium |
| CN114327419B (en)* | 2020-10-10 | 2025-06-17 | 中兴通讯股份有限公司 | Model conversion method, device, electronic device, and storage medium |
| CN112597168A (en)* | 2020-12-28 | 2021-04-02 | 恩亿科(北京)数据科技有限公司 | Processing method, device and platform of multi-source customer data and storage medium |
| CN114911750A (en)* | 2021-02-07 | 2022-08-16 | 北京三快在线科技有限公司 | Data conversion method and device |
| CN114547038A (en)* | 2022-02-25 | 2022-05-27 | 杭州迪普科技股份有限公司 | Data processing method and device for priority database |
| CN114547038B (en)* | 2022-02-25 | 2024-10-25 | 杭州迪普科技股份有限公司 | Data processing method and device of priority database |
| CN116431614A (en)* | 2023-04-26 | 2023-07-14 | 北京计算机技术及应用研究所 | A multi-source data governance method based on unsupervised learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111198910B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111198910B (en) | Data fusion method and device | |
| CN109614545A (en) | Page loading method, device and equipment | |
| US9767086B2 (en) | System and method for enablement of data masking for web documents | |
| CN110502515B (en) | Data acquisition method, device, equipment and computer readable storage medium | |
| CN106991112B (en) | Information query method and device | |
| US10810364B2 (en) | Data flow view for a spreadsheet | |
| CN113835692B (en) | Dictionary data processing method and device, electronic equipment and computer storage medium | |
| CN106055528A (en) | Automatic generation method and apparatus of documents | |
| JP2021518021A (en) | Data processing methods, equipment and computer readable storage media | |
| CN108846069B (en) | Document execution method and device based on markup language | |
| CN102346925B (en) | Electronic ticket storage equipment, electronic ticket checking system and method | |
| CN113900955A (en) | Automated testing method, device, equipment and storage medium | |
| CN109726295A (en) | Brand knowledge graph display method, device, graph server and storage medium | |
| CN112765197A (en) | Data query method and device, computer equipment and storage medium | |
| CN114064638B (en) | Information processing method, device, equipment and storage medium | |
| CN108304291B (en) | Test input information retrieval apparatus and method | |
| US20080082535A1 (en) | Method and system for automatically generating a communication interface | |
| CN113177389A (en) | Text processing method and device, electronic equipment and storage medium | |
| CN107133204A (en) | A kind of terminal shortcut input method | |
| JP6042229B2 (en) | k-anonymous database control server and control method | |
| KR102323880B1 (en) | System for providing cultural property information using machine learning image analysis method | |
| CN106909570B (en) | Data conversion method and device | |
| KR20180033623A (en) | Method and system of language localization | |
| CN110009173A (en) | Business rule processing method and processing device | |
| CN114282510B (en) | Document generation method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |