CN112199433A - Data management system for city-level data middling station - Google Patents

Abstract

The invention provides a data management system for a city-level data middlebox, which relates to the technical field of data processing and comprises the following components: the data standard management platform is used for storing and managing at least one preset data management standard and at least one preset quality management standard; the metadata management module is used for managing metadata related to the data resources according to the data management specification; the data quality management module is used for generating a data quality management standard according to the quality management standard and performing quality control on the management process of the metadata according to the data quality management standard; and the public management and portal module is used for providing uniform user authority management support and navigation query for the metadata management module, the data standard management module and the data quality management module. The technical scheme meets the requirements of checking the metadata of the base table, checking the relationship and influence of the blood relationship among the data and analyzing the dependency relationship among the data, and can better serve the urban data middleboxes.

Description

Data management system for city-level data middling station

Technical Field

The invention relates to the field of data processing, in particular to a data management system for a city-level data middling station.

Background

The data center platform is a data application development portal and has the functions of covering the whole process of data application development in a closed loop mode, completely covering off-line calculation, real-time calculation application and the like. The method can meet the requirements of developers on data acquisition, data analysis, data mining, data quality, data maps, data models and data API application in each level. It can be understood that the data center can be used for liberating the productivity of developers, greatly shortening the extraction process of data value and improving the capability of enterprises for refining the data value. However, in the existing data center, the data governance service is not complete enough, and the management function of the metadata in the data governance service function is not perfect enough.

Disclosure of Invention

Aiming at the problems in the prior art, the invention provides a data management system for a city-level data middlebox, which specifically comprises the following steps:

the method is used for carrying out data governance on data resources stored in a data resource center of an urban data center, and comprises the following steps:

the data standard management platform is used for storing and managing at least one preset data management standard and at least one preset quality management standard;

the metadata management module is connected with the data standard management platform and is used for managing metadata related to the data resources according to the data management specification;

the data quality management module is respectively connected with the metadata management module and the data standard management platform and is used for generating a data quality management standard according to the quality management standard and performing quality control on the management process of the metadata according to the data quality management standard;

and the public management and portal module is respectively connected with the metadata management module, the data standard management platform and the data quality management module and is used for providing unified user authority management support and navigation query for the metadata management module, the data standard management platform and the data quality management module.

Preferably, the metadata management module adopts a layered architecture, and the layered architecture comprises, from bottom to top:

the acquisition layer is used for acquiring various metadata and carrying out preliminary data analysis on the metadata to obtain metadata relations among the metadata;

the storage layer is used for carrying out centralized storage on the metadata input by the acquisition layer and storing the metadata relation;

the application layer is used for analyzing and managing the metadata and the metadata relation;

and the service layer is used for providing external services based on the metadata.

Preferably, the collection layer collects the metadata through a pre-configured collection adapter and/or a pre-generated data import module.

Preferably, the metadata includes:

one or more of distributed file system metadata, distributed data warehouse metadata, distributed NoSQL database metadata, data retrieval component metadata, and real-time stream computation component metadata.

Preferably, the application layer includes:

the blood margin analysis unit is used for performing blood margin analysis on each metadata to obtain a data source of each metadata;

and the influence analysis unit is used for analyzing the influence on other systems and data when the metadata is changed.

A data importance analyzing unit for analyzing the importance of each metadata from the used frequency, the accessed frequency and the dark data discrimination result of each metadata;

and an analysis result derivation unit connected to the blood-margin analysis unit, the influence analysis unit, and the data importance analysis unit, respectively, and configured to derive an analysis result of each of the metadata.

Preferably, the application layer further includes a metadata management unit, configured to perform metadata view management, and/or metadata maintenance, and/or metadata query, and/or metadata export, and/or metadata version management, and/or metadata change management on the metadata.

Preferably, the data quality management module includes:

the standard mapping unit is used for generating corresponding constraints or templates according to the quality management specifications so that a user can customize data quality rules, the rating cards and the knowledge base;

the checking management unit is used for the user to configure and schedule the checking task generated by calculation;

the data quality report database is used for generating a customized data quality report according to the checking result corresponding to the checking task;

and the problem management unit is connected with the data quality report database and is used for discovering, alarming and processing the data quality problem according to the data quality report.

Preferably, the public management and portal module includes:

and the public management unit is used for providing services of user management, role management, authority management, log management, database management, safety management and audit management, and providing unified user authority management support for the metadata management module, the data standard management platform and the data quality management module.

Preferably, the public management and portal module further includes:

and the portal unit is used for providing services of query retrieval, navigation, general functions and portal management and realizing navigation query for the metadata management module, the data standard management platform and the data quality management module.

The technical scheme has the following advantages or beneficial effects:

the technical scheme integrates the metadata assets of all links so as to browse and analyze the metadata and also be a source for forming a data resource management portal; meanwhile, the method also meets the requirements of checking the metadata of the base table, checking the relationship and influence of the blood relationship among the data and analyzing the dependency relationship among the data, and can better serve for the middle platforms of the urban data.

Drawings

FIG. 1 is a schematic diagram of a data governance system in accordance with a preferred embodiment of the present invention;

FIG. 2 is a diagram illustrating a structure of a metadata management module according to a preferred embodiment of the present invention.

Detailed Description

The invention is described in detail below with reference to the figures and specific embodiments. The present invention is not limited to the embodiment, and other embodiments may be included in the scope of the present invention as long as the gist of the present invention is satisfied.

In a preferred embodiment of the present invention, based on the above problems in the prior art, there is provided a data management system for a data center of a city-level data center, for performing data management on data resources stored in a data resource center of the city-level data center, as shown in fig. 1, including:

the datastandard management platform 1 is used for storing and managing at least one preset data management standard and at least one preset quality management standard;

themetadata management module 2 is connected with the datastandard management platform 1 and is used for managing metadata related to the data resources according to data management specifications;

the dataquality management module 3 is respectively connected with themetadata management module 2 and the datastandard management platform 1 and is used for generating a data quality management standard according to the quality management standard and performing quality control on the management process of the metadata according to the data quality management standard;

and the public management andportal module 4 is respectively connected with themetadata management module 2, the datastandard management platform 1 and the dataquality management module 3 and is used for providing unified user authority management support and navigation query for themetadata management module 2, the datastandard management platform 1 and the dataquality management module 3.

Specifically, in this embodiment, the data governance system is an important tool for data governance and data management, and can integrate metadata assets of each link of the platform so as to browse and analyze metadata, and is also a source for forming a data resource management portal. The data management system provides a visualmetadata management module 2, and the requirements of checking the metadata of the base table, checking the blood relationship and influence among the data and analyzing the dependency relationship among the data are met.

The data management system comprises ametadata management module 2, a datastandard management platform 1, a dataquality management module 3, a public management andportal module 4 and data cube design work, and is used for supporting data extraction from regional data lakes as required, carrying out data cube modeling and data management according to established data standards, business models and the like to form a basic library and a theme library, and carrying out data resource management work such as metadata management and blood source analysis on data resources stored in all data resource centers.

In this embodiment, the data set of the metadata is analyzed using an online analysis processing technique. The online analysis processing is a multidimensional analysis technology, and helps business personnel to quickly and interactively know and observe data from multiple aspects, so that the information in the data can be deeply mastered. The online analysis process uses the concept of a multidimensional view cube to describe the structure of a data set. Fields in a dataset are divided into two categories, dimensions and measures, according to the role they play for the decision: dimension is a characteristic attribute describing a fact record, corresponding to an axis in the cube, such as time, location, measure is data reflected on the fact record, and value field is a general term, corresponding to location in coordinates, such as sales, production, population. The on-line analysis processing starts from dimension transformation, and provides operations of drilling, slicing, dicing, rotating and the like. Drilling is the analysis of different levels (from high level to low level, and from low level to high level) of different dimensionalities and granularities; the slicing and dicing is to select a specific dimension and perform analysis in the defined dimension; rotation is a transformation of the dimensional direction. Decision-making personnel can extract intuitive and understandable data reflecting government operation conditions from the original data through the operations, thereby providing support for decision-making. The data cube design tool serves a business library in the subject library which needs efficient dimension query. The cube design is carried out through a visual interface, the construction of an M-OLAP scene data mart is mainly met, an optimized storage data table which accords with business characteristics is established based on the existing data table, the query based on the source data table can fall on the optimized storage data table, the query performance is further accelerated, and the query concurrency is improved. Supporting the instantiation of the established cube in a distributed file system; supporting incremental construction and dimension reduction optimization; the snowflake model and the star model can be supported; the data source can support various formats, including a distributed file system and a relational database management system; the method supports the life cycle management of the cube, including the monitoring management of cube establishment, update and deletion; and the import and export of the cube model can be supported.

In this embodiment, the datastandard management platform 1 manages the process and content of making the data standard according to the data standard management specification, and the datastandard management platform 1 mainly has the following functions:

importing an external standard: the method is characterized in that an Excel template mode is provided, external national standards or industry standards are imported, standard import of types such as data items, data dictionaries, indexes and the like is supported, and expansion according to a meta model is supported.

Data standard mapping: and establishing a mapping relation from the database system to the data standard, and ensuring the landing and execution of the data standard.

Data standard cataloging: and supporting a custom data standard directory structure and content, and providing corresponding query, browsing and using pages.

Data resource association: support associating data criteria with arbitrary resources. The method comprises the steps of providing association analysis of data standards with metadata and data resources, analyzing data resource distribution of reference specified standards, and reference statistics of the standards.

Data standard maintenance: and standard maintenance functions including functions of standard addition, change, audit and the like are realized, standard version management and comparison functions are supported, and standard change conditions are recorded.

And (3) standard format configuration: and the storage and display format and type of the custom data standard are supported.

The data quality management is to provide support for the data quality management process according to the data quality management standard, so that the checking rules can be matched according to the data standard, and the checking objects can be automatically matched according to the metadata. The main process of data quality management comprises quality rule configuration and data quality check, and the problems of data integrity, uniqueness, authority, consistency, legality and the like can be solved.

In a preferred embodiment of the present invention, themetadata management module 2 adopts a layered architecture, as shown in fig. 2, the layered architecture comprises, from bottom to top:

theacquisition layer 21 is configured to acquire various metadata and perform preliminary data analysis on the metadata to obtain a metadata relationship between the metadata;

thestorage layer 22 is used for storing all metadata input by theacquisition layer 21 in a centralized manner and storing metadata relationships;

theapplication layer 23 is used for analyzing and managing metadata and metadata relationships;

and a service layer 24 for providing an external service based on the metadata.

Specifically, in this embodiment, the process of theacquisition layer 21 for acquiring metadata includes: managing a collected data source: the technical scheme provides various metadata data source management functions, including access to different types of acquisition sources, acquisition source parameter configuration and the like.

And (3) collecting template management: the technical scheme provides template management for collecting the metadata and provides an auxiliary technical means for technicians to maintain the metadata. The template function is mainly used when the metadata is automatically and manually imported, the function mainly provides a data template for the technical staff to export the metadata to be imported, and the format of the imported data is conveniently provided.

Metadata mapping management: in order to solve the problem that a plurality of metadata exist between different layers in enterprise information construction, but the phenomenon that the same metadata exist is described, and the phenomenon that the plurality of metadata exist between design and implementation are also described, but the phenomenon that the same metadata exist is described, the technical scheme introduces the concept of metadata mapping management, and by configuring the mapping relation among the metadata, the metadata are considered to be the same metadata if the codes of the metadata are the same under a metadata directory with the mapping relation.

The method also comprises collection task management and collection scheduling management.

In a preferred embodiment of the present invention, thecollection layer 21 collects the metadata through a pre-configured collection adapter and/or a pre-generated data import module.

Specifically, in the present embodiment, theacquisition layer 21 further implements the following management for acquisition adapters: according to the technical scheme, metadata information is collected from each system of an enterprise, and collection task configuration and the collected model result information are supported to be checked. The technical scheme supports the collection of metadata information such as various relational databases, urban data middleboxes, storage processes, modeling tools and the like.

In a preferred embodiment of the present invention, the metadata includes:

one or more of distributed file system metadata, distributed data warehouse metadata, distributed NoSQL database metadata, data retrieval component metadata, and real-time stream computation component metadata.

Specifically, in this embodiment, the metadata is a place where each component (distributed file system, analytic data warehouse, distributed NoSQL database, data retrieval component, real-time computation component, etc.) of the data governance system stores related data information, and is used to describe data, including creation information, affiliated space, access authority, type description, etc. The data governance system needs to provide a highly available database to provide a unified management storage of metadata for all components.

The distributed file system metadata comprises file attribute information such as file names, directory names, father directory information, file sizes, creation time, modification time and the like, and also comprises storage related information such as file blocking conditions, copy numbers, nodes where each copy is located and the like; and the data belonging relation is also recorded, the information of the user and the user group to which the user belongs is provided, and the authority of the user and the user group can be marked.

The distributed data warehouse metadata includes:

library level meta-information: including library name, description information, creator, creation time, authority to build table and look up table in library, etc.

Table level meta information: the method comprises table name, description information, creator, creation time, belonged library, table field, table adding and deleting authority, table deleting authority and the like.

Field meta information: including field names, descriptive information, field type, default values, whether empty, user access rights, etc.

The distributed NoSQL database metadata refers to metadata of the NoSQL database mapping table in the distributed bins. Similar to the distributed number warehouse table level and field authority, the system comprises a table name, table description information, a creator, creation time, a belonging library, a field in the table, a table adding and deleting authority, a table deleting authority, a field name, field description information, a field type, a user access authority and the like.

Data retrieval component metadata refers to metadata of a data retrieval engine mapping table in a distributed data warehouse. Similar to the distributed number warehouse table level and field authority, the system comprises a table name, table description information, a creator, creation time, a belonging library, a field in the table, a table adding and deleting authority, a table deleting authority, a field name, field description information, a field type, a user access authority and the like.

Real-time stream computation component metadata real-time stream computation has three core concepts: flows, flow tasks, and flow applications. A stream is a stream of data, a stream task is a task that performs computations on one or more streams of data and writes the results into a table, and a stream application is a collection of one or more stream tasks.

The stream meta information needs to include stream name, description information, creator, creation time, belonging library, in-stream field, in-stream add/delete/modify authority, delete stream authority, etc.

The stream task meta-information needs to include task name, description information, creator, start time, belonging library, task logic, start-stop authority, etc.

The stream application meta-information needs to include application name, description information, creator, creation time, belonging library, application stream task information, and the like.

In a preferred embodiment of the present invention, theapplication layer 23 includes:

a bloodmargin analysis unit 231, configured to perform blood margin analysis on each metadata to obtain a data source of each metadata;

and aninfluence analysis unit 232 for analyzing influences on other systems and data when the metadata is changed.

A dataimportance analyzing unit 233 for analyzing the importance of each metadata from the used frequency, the number of times of access, and the dark data discrimination result of each metadata;

the analysisresult derivation section 234 is connected to the bloodmargin analysis section 231, theinfluence analysis section 232, and the dataimportance analysis section 233, and derives the analysis result of each metadata.

Specifically, in the embodiment, the bloodmargin analysis unit 231 performs blood margin analysis by using the metadata, and traces back the source of the data upward for reference for tracking and tracing the system operation and maintenance and data quality problems.

Theinfluence analysis unit 232 analyzes the influence of a metadata object change on a downstream system in the next direction, and is used for analyzing the influence on other systems and data when the metadata change occurs.

The dataimportance analyzing unit 233 analyzes the importance of data from the frequency with which the data is used, the number of times it is accessed, and whether it is dark data, respectively.

The analysisresult derivation unit 234 derives the result of each metadata analysis.

In a preferred embodiment of the present invention, theapplication layer 23 further comprises ametadata management unit 235 for performing metadata view management, and/or metadata maintenance, and/or metadata query, and/or metadata export, and/or metadata version management, and/or metadata change management on the metadata.

Specifically, in this embodiment, metadata view management refers to that the data governance system provides an enterprise data view according to a service line, so that service personnel can conveniently view data from a service perspective.

Metadata maintenance refers to query modification and deletion operations of metadata basic information, attributes, depended relationships, dependency relationships and combination relationships.

The metadata inquiry refers to inquiring metadata which accords with data access authority according to the search condition.

Metadata export refers to the data governance system providing metadata export functionality.

Metadata version management refers to the data management system providing the lifecycle management of metadata, and the release, deletion and state change have strict processes and provide version management functions.

The metadata change management means that a user can subscribe concerned metadata by himself, when the metadata are changed, the data management system informs the user of the change in a form specified by the user, and the user can further inquire specific content of the change and related influence analysis in the system according to guidance.

In a preferred embodiment of the present invention, the dataquality management module 3 includes:

thestandard mapping unit 31 is used for generating corresponding constraints or templates according to the quality management specifications, so that a user can customize the data quality rules, the rating cards and the knowledge base;

thechecking management unit 32 is used for the user to configure and schedule the checking task generated by calculation;

the dataquality report database 33 is used for generating a customized data quality report according to a checking result corresponding to the checking task;

and theproblem management unit 34 is connected with the dataquality report database 33 and is used for discovering, alarming and processing data quality problems according to the data quality reports.

Specifically, in this embodiment, thestandard mapping unit 31 generates a corresponding constraint or template based on the relevant standard of the standard platform, and provides the user with customization of the data quality rule, the rating card, and the knowledge base; thechecking management unit 32 supports the user to configure, schedule and manage the checking task. The related checking task is calculated and generated in the data quality analysis engine based on the apachespark self-research, and the scheduling management is realized through a calculation scheduling module in the universal component and service layer 24; the dataquality report database 33 generates customized data quality reports according to the checking results, including quality summarization, quality trend analysis and the like, and supports various customized queries; theproblem management unit 34 supports discovery, alerting and handling of relevant data quality problems for data quality reports.

The dataquality management module 3 mainly functions including:

and (4) quality rule configuration, and data quality measurement rule and checking method management are realized. The configuration realizes the management of the checking task, and supports the generation of the quality checking method according to the data standard through the interface maintenance task.

And (4) data quality checking, namely performing compliance checking on checked objects in sequence according to data quality rules, and recording problem data and abnormal records after finding a data quality problem.

And the data quality problem detail report is required to be provided after the checking task is executed, the overall situation of the problem is described, and the detail quality report is provided according to dimensions such as ownership departments, resource classification and the like.

The quality management tool can perform rule setting such as data cleaning conversion and the like through the data migration tool, performs scheduling design on the quality management process, and embeds data standardization work into a business process so as to achieve the purpose of defining the type, definition and rule of data according to a unified data standard and a treatment standard.

And (2) data quality audit, wherein an analytic data warehouse provided by a big data support platform is used for providing a data audit function, dirty data are written into a specified dirty data table according to rules (including but not limited to data misreading, field type mismatching and UDF condition filtering), and after data import is finished, dirty data reasons, record numbers, import interfaces and data quality reports are recorded so as to facilitate judgment and processing of a monitoring program.

In a preferred embodiment of the present invention, the common management andportal module 4 includes:

and thepublic management unit 41 is used for providing services of user management, role management, authority management, log management, database management, security management and audit management, and providing unified user authority management support for themetadata management module 2, the datastandard management platform 1 and the dataquality management module 3.

Specifically, in this embodiment, the user manages: the interface unified identity authentication platform provides user management including modules of metadata, data standards, data quality and the like, including user information, password verification and the like.

And (3) role management: different roles are divided according to different business departments and responsibilities, such as an administrator, a data subject manager, an inquiry user and the like.

And (3) authority management: and managing related rights of data governance, including rights definition, rights allocation and rights removal.

Log management: including management of data standards, data quality, related logs of functional modules such as metadata, running logs of the system itself, and the like.

Knowledge base management: and managing data management related knowledge base, including processing knowledge of data problems, checking rule making knowledge and the like.

Safety management: the page watermark display is supported, and the access authority can be distinguished according to the network segment.

And (4) audit management: and auditing of operations such as addition, deletion, modification, check and the like is supported.

In a preferred embodiment of the present invention, the public management andportal module 4 further includes:

and theportal unit 42 is used for providing services of query retrieval, navigation, general functions and portal management, and is used for providing navigation query for themetadata management module 2, the datastandard management platform 1 and the dataquality management module 3.

Specifically, in this embodiment, query retrieval: the method comprises the steps of inputting keywords to inquire and derive related information of precise and fuzzy search, such as data standard information item inquiry, quality rule inquiry and data dictionary inquiry.

Navigation: and the entrance for guiding the user to find the related functions from different dimensions comprises function navigation, classification navigation, role navigation and the like.

General functions: various convenient functions of the user are provided, including bulletin boards, message notification, subscription services, favorites, backlogs and the like.

And (3) portal management: and managing and retrieving the portal, customizing a browsing view by a user and the like.

While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims (9)

1. A data governance system for a city-level data center, for data governance of data resources stored by a data resource center of the city-level data center, comprising:

the data standard management platform is used for storing and managing at least one preset data management standard and at least one preset quality management standard;

the metadata management module is connected with the data standard management platform and is used for managing metadata related to the data resources according to the data management specification;

the data quality management module is respectively connected with the metadata management module and the data standard management platform and is used for generating a data quality management standard according to the quality management standard and performing quality control on the management process of the metadata according to the data quality management standard;

and the public management and portal module is respectively connected with the metadata management module, the data standard management platform and the data quality management module and is used for providing unified user authority management support and navigation query for the metadata management module, the data standard management platform and the data quality management module.

2. The data governance system of claim 1, wherein the metadata management module employs a hierarchical architecture comprising, from bottom to top:

the acquisition layer is used for acquiring various metadata and carrying out preliminary data analysis on the metadata to obtain metadata relations among the metadata;

the storage layer is used for carrying out centralized storage on the metadata input by the acquisition layer and storing the metadata relation;

the application layer is used for analyzing and managing the metadata and the metadata relation;

and the service layer is used for providing external services based on the metadata.

3. The data governance system of claim 2, wherein the collection layer performs collection of each of the metadata via a pre-configured collection adapter and/or a pre-generated data import module.

4. The data governance system of claim 2, wherein the metadata comprises:

one or more of distributed file system metadata, distributed data warehouse metadata, distributed NoSQL database metadata, data retrieval component metadata, and real-time stream computation component metadata.

5. The data governance system of claim 2, wherein the application layer comprises:

the blood margin analysis unit is used for performing blood margin analysis on each metadata to obtain a data source of each metadata;

and the influence analysis unit is used for analyzing the influence on other systems and data when the metadata is changed.

A data importance analyzing unit for analyzing the importance of each metadata from the used frequency, the accessed frequency and the dark data discrimination result of each metadata;

and an analysis result derivation unit connected to the blood-margin analysis unit, the influence analysis unit, and the data importance analysis unit, respectively, and configured to derive an analysis result of each of the metadata.

6. The data governance system of claim 5, wherein the application layer further comprises a metadata management unit for metadata view management, and/or metadata maintenance, and/or metadata query, and/or metadata export, and/or metadata version management, and/or metadata change management for the metadata.

7. The data governance system of claim 1, wherein the data quality management module comprises:

the standard mapping unit is used for generating corresponding constraints or templates according to the quality management specifications so that a user can customize data quality rules, the rating cards and the knowledge base;

the checking management unit is used for the user to configure and schedule the checking task generated by calculation;

the data quality report database is used for generating a customized data quality report according to the checking result corresponding to the checking task;

and the problem management unit is connected with the data quality report database and is used for discovering, alarming and processing the data quality problem according to the data quality report.

8. The data governance system of claim 1, wherein the common management and portal module comprises:

and the public management unit is used for providing services of user management, role management, authority management, log management, database management, safety management and audit management, and providing unified user authority management support for the metadata management module, the data standard management platform and the data quality management module.

9. The data governance system of claim 8, wherein the common management and portal module further comprises:

and the portal unit is used for providing services of query retrieval, navigation, general functions and portal management and realizing navigation query for the metadata management module, the data standard management platform and the data quality management module.

Images