Data cleansing and integration intelligent systemTechnical field
The present invention relates to a kind of data cleansing and integrate intelligent system.
Background technology
Large data are sunrise industries, but its utilization is also in initial stage, is that the data volume that industry self accumulates is insufficient because enterprise is not deep enough to the understanding of large data processing on the one hand, in relatively limited data, cannot extract the valuable information to enterprise; Because the comparatively experience of ripe large data analysis processing of nothing on the other hand, because available data analytical technology rests on the aspect of data display substantially, too many increment information and intelligence suggestion can not be provided, enterprise still will paddle one's own canoe to carry out decision-making, from extracting data be worth ability too a little less than.
Enterprise is in the budding stage to the demand of the large data analysis solution of commercialization, and present situation is that enterprise self feels simply helpless to the mass data day by day adding up.Often both not known how to analyze also not know what the target of analyzing is.Under the overall background of national industry upgrading, all kinds of enterprises all, attempting innovation, provide high value-added product and service.How to utilize existing data to help in time, effectively, automatically and the decision-making of science becomes the embodiment of enterprise core competence day by day.Following enterprise will be more and more stronger to the dependence of data analysis, and this place, great market space of data analysis just.
Along with the generation of cloud concept, enterprise has had the ability to create the cloud platform of oneself at present, and the collection of large data has become possibility with storage, how cloud platform is applied to the urgent problem that becomes current research in the self-growth for enterprise.
Summary of the invention
Goal of the invention: the object of the invention is in order to solve the deficiencies in the prior art, adapt to the growth requirement of the data processing of accumulating over a long period, provide a kind of management flexibly, the high and quantity of information data cleansing accurately of efficiency with integrate intelligent system.
Technical scheme: data cleansing of the present invention with integrate intelligent system, its objective is such realization,
A kind of data cleansing and integration intelligent system, comprising:
Database Unit: according to the required structure Database Unit of industry, and set up index;
Cloud storage platform: collection data source and this data source have relevant information the construction logic relation of set membership, contrasting data library unit, revise and mate also the coupling of downward revision one by one with the superiors' information in this set membership, the data that match are carried out to algorithm for encryption storage;
Artificial intelligence data platform: the data that are stored in cloud storage platform are carried out to Data Audit, and in conjunction with terminal calling rule, audit recommendation is proposed, available data specification is become to be applicable to the form of CRM application, set up the CRM database of terminal according to calling rule, rear audit and normalized available data are migrated in the CRM database of terminal, for CRM application provides data basis;
Terminal: for terminal provides most suitable data fetching, guarantee that data integrity is written into CRM database, the data that grab in unit interval are carried out to data cleansing according to specification, guarantee data fit CRM using standard, be integrated into the form of daily form, according to terminal requirements, carry out data pick-up temporarily, according to terminal requirements, provide form as required.
The mainstream data source of ASCII text file, XML file, Excel form document is exported to SQL server, Oracle, Teradata by described cloud storage platform, and be transferred to cloud storage platform by the mode of Sterling File Gateway, FTP/SFTP/HTTPS.
Beneficial effect: data cleansing and integration platform are realized data acquisition and the distribution of sharing data center, provide the data exchange service such as warehouse-in are cleaned, change, loaded to exchanged information, clear up dirty data, the arrangement of complete paired data, guarantees data consistency, integrality and correctness.
Each operation system is carried out exchanges data and is shared by cleaning and integration system and shared data center platform, and each operation system independent operating, is independent of each other, and a certain operation system fault can not cause the impact on other system.
Embodiment
In order to deepen the understanding of the present invention, below in conjunction with embodiment, the invention will be further described, and this embodiment only, for explaining the present invention, does not form limiting the scope of the present invention.
A kind of data cleansing and integration intelligent system, comprising:
Database Unit: according to the required structure Database Unit of industry, and set up index;
Cloud storage platform: collection data source and this data source have relevant information the construction logic relation of set membership, contrasting data library unit, revise and mate also the coupling of downward revision one by one with the superiors' information in this set membership, the data that match are carried out to algorithm for encryption storage;
Artificial intelligence data platform: the data that are stored in cloud storage platform are carried out to Data Audit, and in conjunction with terminal calling rule, audit recommendation is proposed, available data specification is become to be applicable to the form of CRM application, set up the CRM database of terminal according to terminal calling rule, rear audit and normalized available data are migrated in the CRM database of terminal, for CRM application provides data basis;
Terminal: for terminal provides most suitable data fetching, guarantee that data integrity is written into CRM database, the data that grab in unit interval are carried out to data cleansing according to specification, guarantee data fit CRM using standard, be integrated into the form of daily form, according to terminal requirements, carry out data pick-up temporarily, according to terminal requirements, provide form as required.
Referring to Fig. 1, the structure flow process of system of the present invention is as follows:
The first step: building database, the certain industry information of collecting as required, builds Database Unit, and sets up index;
Second step: data analysis, collection data source and this data source have relevant information the construction logic relation of set membership, contrasting data library unit, revise and mate also the coupling of downward revision one by one with the superiors' information in this set membership, the data that match are carried out to algorithm for encryption storage;
The 3rd step: Data Audit, the data that are stored in cloud storage platform are carried out to Data Audit, and propose audit recommendation in conjunction with terminal calling rule, available data specification is become to be applicable to the form of CRM application;
The 4th step: Data Migration, set up the CRM database of terminal according to terminal calling rule, by after audit and normalized available data migrate in the CRM database of terminal, for CRM application provides data basis;
The 5th step: data capture, and for terminal provides most suitable data fetching, guarantee that data integrity is written into CRM database;
The 6th step: data cleansing, the data that grab in the unit interval are carried out to data cleansing according to specification, guarantee data fit CRM using standard, be integrated into the form of daily form;
The 7th step: data pick-up and form, according to terminal requirements, carry out data pick-up temporarily, according to terminal requirements, provide form as required.
The foregoing is only preferred embodiment of the present invention, in order to limit the present invention, within the spirit and principles in the present invention not all, any amendment of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.