CN113094360A - Cross-industry data processing method - Google Patents
Cross-industry data processing methodDownload PDFInfo
- Publication number
- CN113094360A CN113094360ACN202110296258.0ACN202110296258ACN113094360ACN 113094360 ACN113094360 ACN 113094360ACN 202110296258 ACN202110296258 ACN 202110296258ACN 113094360 ACN113094360 ACN 113094360A
- Authority
- CN
- China
- Prior art keywords
- data
- entity
- sub
- business
- main
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2264—Multidimensional index structures
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Description
Technical Field
The invention relates to the technical field of business data processing, in particular to a cross-industry data processing method.
Background
The data management platform integrates the scattered multi-party data into a uniform technical platform, standardizes and subdivides the data, and enables users to push the subdividing results to the existing interactive marketing environment. The current data management platform only supports defining one service data management-contact related service, only supports the data field and the data structure of a user-defined contact, and only supports two types of data sources of a butt joint database type and a form data type.
In the prior art, a set of data management platform cannot be compatible with data management of a plurality of main services, and enterprises need to pay more financial and material resources; different main service data cannot be stored in an isolated mode, so that data redundancy is caused, and data use is seriously influenced; the data is not supported to be cleaned, a series of problems of low accuracy, poor timeliness and the like of the data caused by dirty data occur, and the data value cannot be mined to the maximum extent; when data in the system has problems, the upstream and downstream of the data cannot be checked, and the problems cannot be quickly positioned and the influence range and degree cannot be evaluated.
Disclosure of Invention
Therefore, the cross-industry data processing method provided by the invention can be instantiated according to different application scenes, and solves the problems that data models between industries are large in difference and data management and data analysis cannot be uniformly carried out.
In order to achieve the above purpose, the invention provides the following technical scheme: a cross-industry data processing method abstracts service data into entities for storage, and the entities are divided into main entities, sporocarp, behavior sporocarp and service entities according to different logic application modes and storage schemes; the main entity is a carrier of the business data, and data analysis is carried out through a data object in the main entity; the sub-entity having a logical affiliation with the master entity, the sub-entity comprising affiliation data that exists in association with the master entity; the behavior sub-entity has a logical affiliation with the main entity, the behavior sub-entity inherits to the sub-entity, and the behavior feature information is expanded on the basis of the sub-entity; the business entity serves as a data source of the main entity, the sporocarp and the behavior sporocarp.
As a preferred solution for the cross-industry data processing method, the sub-entities and the behavioral sub-entities exist in logical affiliation from one main entity, and one sub-entity or behavioral sub-entity is affiliated to only one main entity.
As a preferred scheme of the cross-industry data processing method, one-to-many or many-to-one incidence relation exists between main entities of different business data.
As an optimal scheme of the cross-industry data processing method, data structure and field customization are carried out on each service data, and the customized service data are independently stored to realize isolation among the service data.
As a preferred scheme of the cross-industry data processing method, data aggregation is carried out on a plurality of service data which are isolated from each other according to requirements in a pushing and associated configuration mode.
As a preferred scheme of a cross-industry data processing method, performing two-dimensional management of function modularization and data individuation on the service data;
the functional module freely configures whether the functions of label management, grouping management, index management or user portrait are needed or not aiming at each service data;
and the data personalization is used for performing label system, grouping and index counting operation on each service data, and performing data deduplication according to the acquired service data to generate a dedicated user portrait.
As a preferred scheme of the cross-industry data processing method, a relationship between the business data source and the destination entity is defined as a genetic relationship, and objects of the genetic relationship include business entity to main entity, business entity to sub-entity, business entity to behavior sub-entity, or main entity to main entity.
As a preferred scheme of the cross-industry data processing method, the business data are subjected to data circulation display through a data blood margin analysis chart, data problem positioning is performed through the data blood margin analysis chart, and data with problems after positioning are extracted or pushed again through upstream and downstream business data.
As a preferred scheme of the cross-industry data processing method, data cleaning is performed on the acquired business data, wherein the data cleaning comprises value replacement, interception length, UTM value extraction and MD5 aggregation.
The invention has the following advantages: the business data is abstracted into entities for storage, and the entities are divided into main entities, sporocarps, behavior sporocarps and business entities according to different logic application modes and storage schemes; the main entity is a carrier of business data, and data analysis is carried out through a data object in the main entity; the sub-entity has a logical affiliation with the main entity, the sub-entity including affiliation data that exists in association with the main entity; the behavior sub-entity has a logical affiliation with the main entity, the behavior sub-entity inherits to the sub-entity, and the behavior sub-entity expands the behavior characteristic information on the basis of the sub-entity; the business entity serves as a data source of the main entity, the sporocarp and the behavior sporocarp. The invention greatly helps to save enterprise expenses, is convenient and quick and improves human efficiency; the data is stored in an isolated manner, so that the service requirement is met, and the data safety is improved; the diversity of various data source types can be supported to the maximum extent; the realization of multiple services not only meets the requirement of individuation, but also can realize unified management; tracing the whole trace of the data through the blood margin of the data; through data cleaning, improve data quality to promote the degree of accuracy.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It should be apparent that the drawings in the following description are merely exemplary, and that other embodiments can be derived from the drawings provided by those of ordinary skill in the art without inventive effort.
Fig. 1 is a schematic diagram of an entity relationship in a cross-industry data processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of data flow between entities in the cross-industry data processing method according to the embodiment of the present invention.
Detailed Description
The present invention is described in terms of particular embodiments, other advantages and features of the invention will become apparent to those skilled in the art from the following disclosure, and it is to be understood that the described embodiments are merely exemplary of the invention and that it is not intended to limit the invention to the particular embodiments disclosed. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1 and 2, a cross-industry data processing method is provided, in which business data is abstracted into entities for storage, and the entities are divided into main entities, sub-entities, behavior sub-entities and business entities according to different logic application modes and storage schemes; the main entity is a carrier of the business data, and data analysis is carried out through a data object in the main entity; the sub-entity having a logical affiliation with the master entity, the sub-entity comprising affiliation data that exists in association with the master entity; the behavior sub-entity has a logical affiliation with the main entity, the behavior sub-entity inherits to the sub-entity, and the behavior feature information is expanded on the basis of the sub-entity; the business entity serves as a data source of the main entity, the sporocarp and the behavior sporocarp.
Specifically, the main entity is a main carrier for storing data, and is a main object of data analysis, and the application of the data is mainly an application to the main entity. Such as: contact person and enterprise information. The sub-entity is the attached data attached to the main entity and is the data of the main entity with logical attached relation. Such as: educational history, work history, etc. of the contact. The behavior sub-entity is the behavior information generated by the main entity, a logical affiliation relationship exists between the behavior sub-entity and the main entity, the behavior sub-entity is inherited to the sub-entity, and the characteristic information (such as time and behavior type) of some behaviors is expanded on the basis of the sub-entity. Such as: purchase information for the contact. When all the service data enter the data management system, the service entities are generated in the same structure, so that the safety and the availability of the data are ensured, and the service entities are the source entities of other entity data.
In particular, the sub-entities and the behavioral sub-entities exist in logical affiliation from one main entity, and one sub-entity or behavioral sub-entity is affiliated only to one main entity. I.e. sub-entities and behavioral sub-entities can only exist in logical affiliation with a certain main entity and can only be affiliated with one main entity. And one-to-many or many-to-one association relationship exists between the main entities of different business data.
Specifically, data structure and field customization are performed on each service data, and the customized service data is independently stored to realize isolation between the service data. Each service data is self-defined in data structure and field and is stored independently, and one service data is equal to a set of reduced service systems, so that real data isolation is realized.
Specifically, data aggregation is performed on a plurality of service data which are isolated from each other according to requirements in a pushing and associated configuration mode. The data of a plurality of service data can be aggregated and associated according to the requirement through pushing, association configuration and the like, and the data aggregation can also be realized.
Performing functional modularization and data individuation two-dimensional management on the service data in the cross-industry data processing method;
the functional module freely configures whether the functions of label management, grouping management, index management or user portrait are needed or not aiming at each service data;
and the data personalization is used for performing label system, grouping and index counting operation on each service data, and performing data deduplication according to the acquired service data to generate a dedicated user portrait.
For each service data, whether the functions of label management, grouping management, index management, user portrait, and the like are needed or not can be freely configured, and redundancy of functional modules is avoided. Each business data has a set of label system, grouping and statistical indexes, data duplication is removed according to the collected business data, exclusive user figures are automatically generated, and accurate marketing, driving protection and navigation of enterprises are achieved.
In one embodiment of the cross-industry data processing method, the relationship between the business data source and the destination entity is defined as a consanguinity relationship, and the objects of the consanguinity relationship include business entity to main entity, business entity to sub-entity, business entity to behavior sub-entity or main entity to main entity.
With the aid of fig. 2, the original business data forms business entities in the system, and the business entities are pushed to the designated main entity(s), sub-entities and behavioral sub-entities according to the relationship of blood relationship, and establish the affiliation between them. When a plurality of main entities are put in storage at the same time, the association relationship between the main entities can be established. And after the last behavior entity is put in storage, pushing the data to the next main entity according to the relationship of the blood relationship of the main entity to enter the next round of data stream transfer until the end.
Specifically, the business data are subjected to data circulation display through a data blood margin analysis chart, data problem positioning is carried out through the data blood margin analysis chart, and data with problems after positioning are extracted or pushed again through upstream and downstream business data.
The visualized data blood relationship analysis chart clearly shows that data comes from which table, which fields and data volumes are received, how to circulate, and not only can be clear at a glance, but also can quickly locate the problem root, and can perform re-extraction or push on data influenced by the upstream and the downstream so as to thoroughly correct the data problem.
In one embodiment of the cross-industry data processing method, data cleansing is performed on the collected business data, wherein the data cleansing includes value replacement, interception length, extraction of UTM values, and MD5 aggregation. After the business data is collected, the data can be specially processed, the data is normalized or derived into a new field, the data conversion module supports various cleaning gadgets, and the data conversion module supports the expansion of various cleaning gadgets by 'value replacement, interception length, UTM value extraction and MD5 aggregation'.
The cross-industry data processing method is characterized in that business data are abstracted into entities for storage, and the entities are divided into main entities, sporocarps, behavior sporocarps and business entities according to different logic application modes and storage schemes; the main entity is a carrier of business data, and data analysis is carried out through a data object in the main entity; the sub-entity has a logical affiliation with the main entity, the sub-entity including affiliation data that exists in association with the main entity; the behavior sub-entity has a logical affiliation with the main entity, the behavior sub-entity inherits to the sub-entity, and the behavior sub-entity expands the behavior characteristic information on the basis of the sub-entity; the business entity serves as a data source of the main entity, the sporocarp and the behavior sporocarp. The data management system can support the self-definition of a plurality of service data management, each service data can self-define the type (field or data relation) and the functional module (whether a label is needed or not, data rating and the like) of the service data, all main service data are simultaneously managed in one set of data management platform or system after the data are accessed through a data source in various modes, and different main service data are completely stored and isolated, so that the data safety is improved. And the upstream and downstream of all data uploaded to a data management platform or system can be inquired through the data blooding margin, and after a data problem is met, the problem can be quickly positioned, if a serious data problem is met, dirty data can be cleared through one key, and then the data can be re-extracted/re-pushed. The invention greatly helps to save enterprise expenses, is convenient and quick and improves human efficiency; the data is stored in an isolated manner, so that the service requirement is met, and the data safety is improved; the diversity of various data source types can be supported to the maximum extent; the realization of multiple services not only meets the requirement of individuation, but also can realize unified management; tracing the whole trace of the data through the blood margin of the data; through data cleaning, improve data quality to promote the degree of accuracy.
Although the invention has been described in detail above with reference to a general description and specific examples, it will be apparent to one skilled in the art that modifications or improvements may be made thereto based on the invention. Accordingly, such modifications and improvements are intended to be within the scope of the invention as claimed.
Claims (9)
1. A cross-industry data processing method is characterized in that business data is abstracted into entities for storage, and the entities are divided into main entities, sporocarp, behavior sporocarp and business entities according to different logic application modes and storage schemes; the main entity is a carrier of the business data, and data analysis is carried out through a data object in the main entity; the sub-entity having a logical affiliation with the master entity, the sub-entity comprising affiliation data that exists in association with the master entity; the behavior sub-entity has a logical affiliation with the main entity, the behavior sub-entity inherits to the sub-entity, and the behavior feature information is expanded on the basis of the sub-entity; the business entity serves as a data source of the main entity, the sporocarp and the behavior sporocarp.
2. The method of claim 1, wherein the sub-entities and behavioral sub-entities are in logical affiliation with a host entity, and a sub-entity or behavioral sub-entity is affiliated with only a host entity.
3. The cross-industry data processing method according to claim 1, wherein a one-to-many or many-to-one association exists between the main entities of different business data.
4. The cross-industry data processing method of claim 1, wherein data structure and field customization is performed on each service data, and the customized service data is independently stored to achieve isolation between the service data.
5. The cross-industry data processing method according to claim 4, wherein a plurality of mutually isolated service data are aggregated on demand in a push and association configuration mode.
6. The cross-industry data processing method according to claim 1, wherein the business data is managed in two dimensions of functional modularization and data individuation;
the functional module freely configures whether the functions of label management, grouping management, index management or user portrait are needed or not aiming at each service data;
and the data personalization is used for performing label system, grouping and index counting operation on each service data, and performing data deduplication according to the acquired service data to generate a dedicated user portrait.
7. The cross-industry data processing method according to claim 1, wherein a relationship between the business data source and destination entities is defined as a genetic relationship, and the objects of the genetic relationship comprise business entity to main entity, business entity to sub-entity, business entity to behavioral sub-entity or main entity to main entity.
8. The cross-industry data processing method according to claim 7, wherein the business data is displayed by data flow through a data blood margin analysis chart, the data problem is positioned through the data blood margin analysis chart, and the positioned data with problems is extracted or pushed again by using upstream and downstream business data.
9. The cross-industry data processing method according to claim 1, wherein data cleaning is performed on the collected business data, and the data cleaning comprises value replacement, truncation length, extraction of UTM value and MD5 aggregation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110296258.0ACN113094360B (en) | 2021-03-19 | 2021-03-19 | Cross-industry data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110296258.0ACN113094360B (en) | 2021-03-19 | 2021-03-19 | Cross-industry data processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113094360Atrue CN113094360A (en) | 2021-07-09 |
| CN113094360B CN113094360B (en) | 2023-11-10 |
Family
ID=76668480
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110296258.0AActiveCN113094360B (en) | 2021-03-19 | 2021-03-19 | Cross-industry data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113094360B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040139203A1 (en)* | 2003-01-09 | 2004-07-15 | Graham Edward A. | Software business platform with networked, association-based business entity access management |
| US20040177075A1 (en)* | 2003-01-13 | 2004-09-09 | Vasudev Rangadass | Master data management system for centrally managing core reference data associated with an enterprise |
| US20120158757A1 (en)* | 2010-12-15 | 2012-06-21 | Microsoft Corporation | Inherited Entity Storage Model |
| CN102929771A (en)* | 2012-09-28 | 2013-02-13 | 用友软件股份有限公司 | Log recording device and log recording method |
| CN108038222A (en)* | 2017-12-22 | 2018-05-15 | 冶金自动化研究设计院 | System for Information System Modeling and entity-property frame of data access |
| CN109739486A (en)* | 2019-01-03 | 2019-05-10 | 深圳英飞拓科技股份有限公司 | Multi-data source database manipulation implementation method and device based on JdbcTemplate |
| CN110196889A (en)* | 2019-05-30 | 2019-09-03 | 北京字节跳动网络技术有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN111858615A (en)* | 2020-08-04 | 2020-10-30 | 中国工商银行股份有限公司 | Database table generation method, system, computer system and readable storage medium |
| CN111897890A (en)* | 2020-08-21 | 2020-11-06 | 中国工商银行股份有限公司 | Financial business processing method and device |
| CN111897883A (en)* | 2020-07-15 | 2020-11-06 | 中国工商银行股份有限公司 | Entity model construction method and device, electronic equipment and medium |
- 2021
- 2021-03-19CNCN202110296258.0Apatent/CN113094360B/enactiveActive
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040139203A1 (en)* | 2003-01-09 | 2004-07-15 | Graham Edward A. | Software business platform with networked, association-based business entity access management |
| US20040177075A1 (en)* | 2003-01-13 | 2004-09-09 | Vasudev Rangadass | Master data management system for centrally managing core reference data associated with an enterprise |
| US20120158757A1 (en)* | 2010-12-15 | 2012-06-21 | Microsoft Corporation | Inherited Entity Storage Model |
| CN102929771A (en)* | 2012-09-28 | 2013-02-13 | 用友软件股份有限公司 | Log recording device and log recording method |
| CN108038222A (en)* | 2017-12-22 | 2018-05-15 | 冶金自动化研究设计院 | System for Information System Modeling and entity-property frame of data access |
| CN109739486A (en)* | 2019-01-03 | 2019-05-10 | 深圳英飞拓科技股份有限公司 | Multi-data source database manipulation implementation method and device based on JdbcTemplate |
| CN110196889A (en)* | 2019-05-30 | 2019-09-03 | 北京字节跳动网络技术有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN111897883A (en)* | 2020-07-15 | 2020-11-06 | 中国工商银行股份有限公司 | Entity model construction method and device, electronic equipment and medium |
| CN111858615A (en)* | 2020-08-04 | 2020-10-30 | 中国工商银行股份有限公司 | Database table generation method, system, computer system and readable storage medium |
| CN111897890A (en)* | 2020-08-21 | 2020-11-06 | 中国工商银行股份有限公司 | Financial business processing method and device |
Non-Patent Citations (4)
| Title |
|---|
| RUDNICHENKO Y 等: "Safe interaction management of state institutions and business entities based on the concepts of evolutionary economics: modeling and scenario forecasting of processes", TEM JOURNAL, vol. 9, no. 1, pages 233* |
| 朴岩;陈远平;及俊川;: "基于统一搜索的信息服务平台", 计算机系统应用, no. 11, pages 134 - 140* |
| 王红斌;李金绘;沈强;线岩团;毛存礼;: "基于最大熵的泰语句子级实体从属关系抽取", 南京大学学报(自然科学), no. 04, pages 124 - 132* |
| 胥寿春: "基于多层架构的网格开发模式的设计和实现", 中国优秀硕士学位论文全文数据库信息科技辑, no. 12, pages 138 - 330* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113094360B (en) | 2023-11-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tang et al. | Identifying evolving groups in dynamic multimode networks | |
| CN104731896B (en) | A kind of data processing method and system | |
| CN109325200B (en) | Method and device for acquiring data and computer readable storage medium | |
| CN110555172B (en) | User relationship mining method and device, electronic equipment and storage medium | |
| CN114329082A (en) | A hugegraph-based data blood relationship analysis method and system | |
| CN111461666A (en) | Demand tracking matrix display method and system | |
| CN105812175B (en) | Resource management method and resource management equipment | |
| CN115204251B (en) | Data processing method, device, electronic device and storage medium | |
| CN113032252A (en) | Method and device for collecting buried point data, client device and storage medium | |
| CN105678323A (en) | Image-based-on method and system for analysis of users | |
| CN110674231A (en) | Data lake-oriented user ID integration method and system | |
| CN112784113B (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN113761102A (en) | Data processing method, device, server, system and storage medium | |
| CN106131134B (en) | Method and system for merging and deduplicating message content | |
| Becker et al. | Analysing differences between business process similarity measures | |
| CN104765875A (en) | Distributed processing method and system for passenger behavior data | |
| CN114153862A (en) | Service data processing method, device, equipment and storage medium | |
| CN113094360A (en) | Cross-industry data processing method | |
| CN114462885A (en) | Data ranking method and device based on service information, medium and equipment | |
| CN112699107B (en) | Data management platform supporting high definition | |
| CN115840772B (en) | Passenger group data statistics method and device, electronic equipment and storage medium | |
| CN111625655A (en) | Method, device and storage medium for merging and classifying based on knowledge graph | |
| CN116186337A (en) | A business scene data processing method, system and electronic device | |
| CN110321435B (en) | Data source dividing method, device, equipment and storage medium | |
| CN115543753A (en) | Big data end-to-end monitoring method and system for power grid data center |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |