技术领域technical field
本发明涉及大数据技术领域，尤其涉及一种元数据更新方法、装置、系统、电子设备及存储介质。The present invention relates to the technical field of big data, and in particular, to a method, device, system, electronic device and storage medium for updating metadata.
背景技术Background technique
元数据是关于数据的数据，即对数据的描述信息。IT(Internet Technology，互联网技术)系统作为实际生产系统，在IT系统中，保存有不同的元数据，根据其属性的不同，元数据可分为技术元数据和业务元数据。其中，技术元数据偏重数据结构和数据处理细节方面的技术化描述，主要包括源系统接口规范、数据仓库中模型、表、视图、字段等结构信息的描述、数据集市定义描述以及数据处理过程的描述等信息；业务元数据是对数据处理规则和业务含义的描述，主要包括业务规则、业务术语、指标业务口径、信息分类等。Metadata is the data about the data, that is, the descriptive information about the data. As an actual production system, the IT (Internet Technology, Internet Technology) system stores different metadata in the IT system. According to the different attributes, the metadata can be divided into technical metadata and business metadata. Among them, technical metadata focuses on the technical description of data structure and data processing details, mainly including source system interface specification, description of model, table, view, field and other structural information in data warehouse, data mart definition description and data processing process Business metadata is a description of data processing rules and business meanings, including business rules, business terms, indicators, business calibers, and information classification.
在大数据时代的背景下，数据即资产，元数据管理实现了信息的描述和分类的格式化，它帮助企业更好地对数据资产进行管理，理清数据之间的关系。在以大规模并行处理(MPP)、高性能一体机等技术为基础的传统数据仓库环境下，相对于业务数据，元数据的变更并不频繁，因此元数据的采集一般以批量方式进行。即元数据管理系统定期从生产系统、管理系统中获取各种元数据信息文件，例如SQL(Structured Query Language，结构化查询语言)脚本、ETL(Extract-Transform-Load，数据抽取、转换、装载的过程)脚本、运行日志、业务文档等，批量的进行解析、分析、关联后，实现元数据的采集与更新。In the context of the era of big data, data is an asset, and metadata management realizes the description and formatting of information, which helps enterprises to better manage data assets and clarify the relationship between data. In the traditional data warehouse environment based on technologies such as massively parallel processing (MPP) and high-performance all-in-one machines, compared with business data, metadata changes are not frequent, so metadata collection is generally carried out in batches. That is, the metadata management system regularly obtains various metadata information files from the production system and management system, such as SQL (Structured Query Language) scripts, ETL (Extract-Transform-Load, data extraction, transformation, loading) Process) scripts, operation logs, business documents, etc., after batch parsing, analysis, and association, the collection and update of metadata is realized.
图1为现有数据仓库下元数据的采集更新过程示意图，元数据管理系统将IT系统中采集的SQL脚本、ETL日志和数据库设计等数据，进行解析、分析后得到技术元数据，并获取IT系统中录入的文档、表格等数据，进行解析、分析后得到业务元数据，将技术元数据和业务元数据关联，并对元数据存储库中存储的元数据进行更新。元数据应用通过元数据管理系统提供的API(Application Programming Interface，应用程序编程接口)，采集元数据应用自身需要的元数据，其中元数据应用包括信息浏览、血缘分析、数据质量、资产视图、数据安全等。Figure 1 is a schematic diagram of the metadata collection and update process under the existing data warehouse. The metadata management system analyzes and analyzes the data collected in the IT system, such as SQL scripts, ETL logs and database design, to obtain technical metadata, and obtains IT Documents, tables and other data entered in the system are parsed and analyzed to obtain business metadata, associate technical metadata with business metadata, and update the metadata stored in the metadata repository. The metadata application collects the metadata required by the metadata application itself through the API (Application Programming Interface) provided by the metadata management system. The metadata application includes information browsing, blood relationship analysis, data quality, asset view, data safety etc.
元数据采集以批量方式进行，如在每天24点之前元数据管理系统收集当日元数据变更信息，并进行处理，如果IT系统中的元数据发生变更，则IT系统可以实时获取到变更的元数据，因此当元数据在时间T发生变化，IT系统中的元数据更新过程无延迟即在时间T+0发生更新，而由于元数据管理系统在特定时间批量采集发生变更的元数据，则元数据管理系统中元数据可能在时间T+H发生更新，H为采集变更的元数据的特定时间与IT系统中元数据更新时间的差值，元数据管理系统中存储的元数据与实际生产系统中的数据情况存在一定的时间延迟。Metadata collection is carried out in batches. For example, before 24:00 every day, the metadata management system collects the current yen data change information and processes it. If the metadata in the IT system changes, the IT system can obtain the changed metadata in real time. , so when the metadata changes at time T, the metadata update process in the IT system is updated at time T+0 without delay. Since the metadata management system collects the changed metadata in batches at a specific time, the metadata The metadata in the management system may be updated at time T+H. H is the difference between the specific time of collecting the changed metadata and the metadata update time in the IT system. The metadata stored in the metadata management system is different from that in the actual production system. There is a certain time delay in the data situation.
由于传统数据仓库环境下不支持同步更新，如果需要在元数据管理系统批量更新元数据的时间之前获取到IT系统中已经更新的元数据，经常会出现以下经典应用场景：如数据采集人员在IT系统即生产系统的集群中添加了日期元数据及日期元数据的相关数据，并通知数据处理人员，数据处理人员先加载获取新添加的日期元数据及相关数据，并对相关数据进行清洗；数据分析人员根据日期元数据修改相关表结构，并在相关表中添加日期元数据对应的新数据列及相关数据，从而实现元数据管理系统中日期元数据的添加。也就是当存在元数据更新且未达到批量更新元数据的时间时，数据采集处理分析人员需要人为操作来实现元数据更新，增加了数据采集处理分析人员的工作量，降低数据采集效率，因此现有方式中在特定的时间批量采集更新元数据存在延迟、时效性差、效率低。Since the traditional data warehouse environment does not support synchronous update, if you need to obtain the metadata that has been updated in the IT system before the time when the metadata management system updates the metadata in batches, the following classic application scenarios often occur: The system, that is, the cluster of the production system, adds date metadata and related data of date metadata, and informs the data processing personnel. The data processing personnel first load and obtain the newly added date metadata and related data, and clean the related data; The analyst modifies the related table structure according to the date metadata, and adds new data columns and related data corresponding to the date metadata in the related table, so as to realize the addition of date metadata in the metadata management system. That is to say, when there is a metadata update and the time for batch update metadata is not reached, the data collection, processing and analysis personnel need manual operations to realize the metadata update, which increases the workload of the data collection, processing and analysis personnel and reduces the data collection efficiency. In some methods, there are delays, poor timeliness, and low efficiency in collecting and updating metadata in batches at a specific time.
发明内容SUMMARY OF THE INVENTION
本发明提供一种元数据更新方法、装置、系统、电子设备及存储介质，用以解决现有技术中特定的时间批量采集更新元数据存在延迟、时效性差、效率低问题。The present invention provides a method, device, system, electronic device and storage medium for updating metadata, which are used to solve the problems of delay, poor timeliness and low efficiency in the prior art when collecting and updating metadata in batches at a specific time.
本发明提供了一种元数据更新方法，应用于安装有元数据管理系统的第一电子设备，该方法包括：The present invention provides a method for updating metadata, which is applied to a first electronic device installed with a metadata management system. The method includes:
接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的；Receive metadata change information, wherein the metadata change information includes the change type and the information of the first metadata after the change, and the information of the change type and the first metadata after the change is obtained by the production system by means of bridging of;
根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。According to the change type and the information of the changed first metadata, the corresponding first metadata stored by itself is updated.
进一步地，所述接收元数据变更信息后，在所述根据所述变更类型及变更后的第一元数据的信息，对自身保存的元数据进行更新之前，所述方法还包括：Further, after the metadata change information is received, before the metadata stored by itself is updated according to the change type and the changed first metadata information, the method further includes:
当所述元数据变更信息中包含文件的信息，且所述变更类型及变更后的第一元数据的信息包含在所述文件的信息中时，对所述文件进行解析，获取变更类型及变更后的第一元数据的信息。When the metadata change information includes file information, and the change type and the changed first metadata information are included in the file information, the file is parsed to obtain the change type and the change After the first metadata information.
进一步地，其中所述元数据变更信息中包含所述第一元数据所属的第一应用的信息，所述对所述文件进行解析包括：Further, wherein the metadata change information includes information of the first application to which the first metadata belongs, and the parsing of the file includes:
根据预先保存的应用与解析队列的对应关系，确定所述第一应用对应的目标解析队列；determining the target parsing queue corresponding to the first application according to the pre-saved correspondence between the application and the parsing queue;
通过所述目标解析队列对所述文件进行解析。The file is parsed through the target parsing queue.
进一步地，所述方法还包括：Further, the method also includes:
将所述变更后的第一元数据保存在针对每个应用创建的发布队列中；saving the changed first metadata in a release queue created for each application;
针对每个应用，判断该应用是否订阅了所述第一元数据；如果是，通过该应用对应的发布队列将所述变更后的第一元数据，发送给该应用，并在该应用对应的发布队列中删除所述变更后的第一元数据。For each application, determine whether the application has subscribed to the first metadata; if so, send the changed first metadata to the application through the publication queue corresponding to the application, and send the changed first metadata to the application corresponding to the application. The changed first metadata is deleted from the release queue.
进一步地，所述方法还包括：Further, the method also includes:
接收第二应用发送的数据获取请求；receiving a data acquisition request sent by the second application;
通过所述第二应用对应的发布队列，将所述第二应用的发布队列中变更后的元数据发送给所述第二应用。The changed metadata in the publishing queue of the second application is sent to the second application through the publishing queue corresponding to the second application.
进一步地，所述方法还包括：Further, the method also includes:
在每个发布队列中，判断第二元数据保存的时长是否大于设定的时长阈值；In each release queue, determine whether the storage duration of the second metadata is greater than the set duration threshold;
如果是，删除所述第二元数据。If so, delete the second metadata.
本发明提供了一种元数据更新方法，应用于安装有生产系统的第二电子设备，该方法包括：The present invention provides a method for updating metadata, which is applied to a second electronic device installed with a production system. The method includes:
通过桥接的方式获取变更类型及变更后的第一元数据的信息；Obtain the information of the change type and the changed first metadata by bridging;
将生成的元数据变更信息发送给元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息，使所述元数据管理系统接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对所述元数据管理系统保存的对应第一元数据进行更新。Sending the generated metadata change information to the metadata management system, wherein the metadata change information includes the change type and the information of the first metadata after the change, so that the metadata management system receives the metadata change information , according to the change type and the information of the changed first metadata, update the corresponding first metadata saved by the metadata management system.
进一步地，所述通过桥接的方式获取变更类型及变更后的第一元数据的信息包括以下至少一种：Further, the information about obtaining the change type and the changed first metadata by means of bridging includes at least one of the following:
截取操作生产系统的数据库产生的指令；判断所述指令是否为进行元数据变更的指令；如果是，获取所述指令中携带的变更类型及变更后的第一元数据的信息；Intercepting the instruction generated by the database of the operation production system; judging whether the instruction is an instruction to change metadata; if so, obtaining the information of the change type carried in the instruction and the first metadata after the change;
解析生产系统的数据库的执行日志，判断所述执行日志中是否存在针对元数据的元数据变更；如果是，获取所述执行日志中的变更类型及变更后的第一元数据的信息；和Parse the execution log of the database of the production system, and determine whether there is a metadata change for the metadata in the execution log; if so, obtain the information of the change type in the execution log and the first metadata after the change; and
获取生产系统的数据库的配置文件，判断所述配置文件中是否存在针对元数据的元数据变更；如果是，获取所述配置文件中的变更类型及变更后的第一元数据的信息。Obtain the configuration file of the database of the production system, and determine whether there is a metadata change for the metadata in the configuration file; if so, obtain the change type in the configuration file and the information of the first metadata after the change.
本发明提供了一种元数据更新装置，应用于安装有元数据管理系统的第一电子设备，该装置包括：The present invention provides a metadata update device, which is applied to a first electronic device installed with a metadata management system. The device includes:
接收模块，用于接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的；The receiving module is used to receive metadata change information, wherein the metadata change information includes the change type and the information of the first metadata after the change, and the information of the change type and the first metadata after the change is passed by the production system obtained by bridging;
更新模块，用于根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。The updating module is configured to update the corresponding first metadata saved by itself according to the change type and the information of the changed first metadata.
所述装置还包括：The device also includes:
获取模块，用于所述接收元数据变更信息后，在所述根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新之前，当所述元数据变更信息中包含文件的信息，且所述变更类型及变更后的第一元数据的信息包含在所述文件的信息中时，对所述文件进行解析，获取变更类型及变更后的第一元数据的信息。The acquiring module is configured to, after receiving the metadata change information, before updating the corresponding first metadata stored by itself according to the change type and the information of the changed first metadata, when the metadata is updated When the data change information includes file information, and the change type and the changed first metadata information are included in the file information, the file is parsed to obtain the change type and the changed first metadata. Metadata information.
所述获取模块，具体用于根据预先保存的应用与解析队列的对应关系，确定所述第一应用对应的目标解析队列；通过所述目标解析队列对所述文件进行解析。The acquiring module is specifically configured to determine the target parsing queue corresponding to the first application according to the pre-saved correspondence between the application and the parsing queue; and parse the file through the target parsing queue.
所述装置还包括：The device also includes:
发送模块，用于将所述变更后的第一元数据保存在针对每个应用创建的发布队列中；针对每个应用，判断该应用是否订阅了所述第一元数据；如果是，通过该应用对应的发布队列将所述变更后的第一元数据，发送给该应用，并在该应用对应的发布队列中删除所述变更后的第一元数据。A sending module, configured to save the changed first metadata in a release queue created for each application; for each application, determine whether the application subscribes to the first metadata; if so, pass the The release queue corresponding to the application sends the changed first metadata to the application, and deletes the changed first metadata from the release queue corresponding to the application.
所述发送模块，还用于接收第二应用发送的数据获取请求；通过所述第二应用对应的发布队列，将所述第二应用的发布队列中变更后的元数据发送给所述第二应用。The sending module is further configured to receive a data acquisition request sent by the second application; and send the changed metadata in the publishing queue of the second application to the second application through the publishing queue corresponding to the second application. application.
所述装置还包括：The device also includes:
删除模块，用于在每个发布队列中，判断第二元数据保存的时长是否大于设定的时长阈值；如果是，删除所述第二元数据。A deletion module, configured to determine whether the storage duration of the second metadata is greater than a set duration threshold in each release queue; if so, delete the second metadata.
本发明提供了一种元数据更新装置，应用于安装有生产系统的第二电子设备，该装置包括：The present invention provides a device for updating metadata, which is applied to a second electronic device installed with a production system. The device includes:
获取模块，用于通过桥接的方式获取变更类型及变更后的第一元数据的信息；The acquisition module is used to acquire the information of the change type and the changed first metadata by means of bridging;
发送模块，用于将生成的元数据变更信息发送给元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息，使所述元数据管理系统接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对所述元数据管理系统保存的对应第一元数据进行更新。A sending module, configured to send the generated metadata change information to the metadata management system, wherein the metadata change information includes the change type and the information of the changed first metadata, so that the metadata management system Receive metadata change information, and update the corresponding first metadata stored in the metadata management system according to the change type and the information of the changed first metadata.
所述获取模块，具体用于通过桥接的方式获取变更类型及变更后的第一元数据的信息包括以下至少一种：截取操作生产系统的数据库产生的指令；判断所述指令是否为进行元数据变更的指令；如果是，获取所述指令中携带的变更类型及变更后的第一元数据的信息；解析生产系统的数据库的执行日志，判断所述执行日志中是否存在针对元数据的元数据变更；如果是，获取所述执行日志中的变更类型及变更后的第一元数据的信息；和获取生产系统的数据库的配置文件，判断所述配置文件中是否存在针对元数据的元数据变更；如果是，获取所述配置文件中的变更类型及变更后的第一元数据的信息。The obtaining module is specifically used to obtain the information of the changed type and the changed first metadata by means of bridging, including at least one of the following: intercepting an instruction generated by operating a database of the production system; judging whether the instruction is for performing metadata The instruction to change; if so, obtain the information of the change type and the first metadata after the change carried in the instruction; parse the execution log of the database of the production system, and determine whether there is metadata for the metadata in the execution log change; if so, obtain the information of the change type in the execution log and the first metadata after the change; and obtain the configuration file of the database of the production system, and determine whether there is a metadata change for the metadata in the configuration file ; If yes, obtain the information of the change type and the changed first metadata in the configuration file.
本发明提供了一种电子设备，包括：存储器、处理器和收发机；The present invention provides an electronic device, comprising: a memory, a processor and a transceiver;
所述处理器，用于读取所述存储器中的程序，执行下列过程：控制所述收发机接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的；根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。The processor, configured to read the program in the memory, executes the following process: controlling the transceiver to receive metadata change information, wherein the metadata change information includes the change type and the information of the changed first metadata; information, the information of the change type and the changed first metadata is obtained by the production system through bridging; according to the change type and the information of the changed first metadata, the corresponding first metadata stored by itself Metadata is updated.
进一步地，所述处理器，还用于当所述元数据变更信息中是否包含文件的信息，且所述变更类型及变更后的第一元数据的信息包含在所述文件的信息中时，对所述文件进行解析，获取变更类型及变更后的第一元数据的信息。Further, the processor is further configured to, when the metadata change information includes file information, and the change type and the changed first metadata information are included in the file information, The file is parsed to obtain the information of the change type and the changed first metadata.
进一步地，所述处理器，具体用于所述元数据变更信息中包含所述第一元数据所属的第一应用的信息，根据预先保存的应用与解析队列的对应关系，确定所述第一应用对应的目标解析队列；通过所述目标解析队列对所述文件进行解析。Further, the processor is specifically configured to include the information of the first application to which the first metadata belongs in the metadata change information, and determine the first application according to the pre-saved correspondence between the application and the parsing queue. Apply the corresponding target parsing queue; parse the file through the target parsing queue.
进一步地，所述处理器，还用于将所述变更后的第一元数据保存在针对每个应用创建的发布队列中；针对每个应用，判断该应用是否订阅了所述第一元数据；如果是，控制所述收发机通过该应用对应的发布队列将所述变更后的第一元数据，发送给该应用，并在该应用对应的发布队列中删除所述变更后的第一元数据。Further, the processor is further configured to save the changed first metadata in a release queue created for each application; for each application, determine whether the application has subscribed to the first metadata If yes, control the transceiver to send the first metadata after the change to the application through the release queue corresponding to the application, and delete the first metadata after the change in the release queue corresponding to the application data.
进一步地，所述处理器，还用于控制所述收发机接收第二应用发送的数据获取请求；通过所述第二应用对应的发布队列，将所述第二应用的发布队列中变更后的元数据发送给所述第二应用。Further, the processor is further configured to control the transceiver to receive a data acquisition request sent by the second application; through the release queue corresponding to the second application, the changed content in the release queue of the second application is changed. Metadata is sent to the second application.
进一步地，所述处理器，还用于在每个发布队列中，判断第二元数据保存的时长是否大于设定的时长阈值；如果是，删除所述第二元数据。Further, the processor is further configured to, in each release queue, determine whether the storage duration of the second metadata is greater than a set duration threshold; if so, delete the second metadata.
本发明提供了一种电子设备，包括：存储器、处理器和收发机；The present invention provides an electronic device, comprising: a memory, a processor and a transceiver;
所述处理器，用于读取所述存储器中的程序，执行下列过程：通过桥接的方式获取变更类型及变更后的第一元数据的信息；将生成的元数据变更信息发送给元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息，使所述元数据管理系统接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对所述元数据管理系统保存的对应第一元数据进行更新。The processor is configured to read the program in the memory, and perform the following processes: obtain the information of the change type and the changed first metadata by means of bridging; send the generated metadata change information to the metadata management system, wherein the metadata change information includes the change type and the information of the changed first metadata, so that the metadata management system receives the metadata change information, according to the change type and the changed first metadata The metadata information is used to update the corresponding first metadata stored in the metadata management system.
进一步地，所述处理器，具体用于通过桥接的方式获取变更类型及变更后的第一元数据的信息包括以下至少一种：截取操作生产系统的数据库产生的指令；判断所述指令是否为进行元数据变更的指令；如果是，获取所述指令中携带的变更类型及变更后的第一元数据的信息；解析生产系统的数据库的执行日志，判断所述执行日志中是否存在针对元数据的元数据变更；如果是，获取所述执行日志中的变更类型及变更后的第一元数据的信息；和获取生产系统的数据库的配置文件，判断所述配置文件中是否存在针对元数据的元数据变更；如果是，获取所述配置文件中的变更类型及变更后的第一元数据的信息。Further, the processor, which is specifically used to obtain the information of the change type and the changed first metadata by means of bridging, includes at least one of the following: intercepting an instruction generated by operating a database of the production system; judging whether the instruction is a An instruction to change metadata; if so, obtain the information of the type of change carried in the instruction and the first metadata after the change; parse the execution log of the database of the production system, and determine whether there is any target metadata in the execution log If yes, obtain the information of the change type in the execution log and the first metadata after the change; and obtain the configuration file of the database of the production system, and judge whether there is any metadata specific to the configuration file in the configuration file. Metadata change; if yes, obtain the change type in the configuration file and the information of the changed first metadata.
本发明提供了一种电子设备，包括：处理器、通信接口、存储器和通信总线，其中，处理器，通信接口，存储器通过通信总线完成相互间的通信；The present invention provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
所述存储器中存储有计算机程序，当所述程序被所述处理器执行时，使得所述处理器执行上述任一方法的步骤。A computer program is stored in the memory, and when the program is executed by the processor, the processor causes the processor to perform the steps of any of the above methods.
本发明提供了一种电子设备，包括：处理器、通信接口、存储器和通信总线，其中，处理器，通信接口，存储器通过通信总线完成相互间的通信；The present invention provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
所述存储器中存储有计算机程序，当所述程序被所述处理器执行时，使得所述处理器执行上述任一方法的步骤。A computer program is stored in the memory, and when the program is executed by the processor, the processor causes the processor to perform the steps of any of the above methods.
本发明提供了一种计算机可读存储介质，其存储有可由电子设备执行的计算机程序，当所述程序在所述电子设备上运行时，使得所述电子设备执行上述任一所述方法的步骤。The present invention provides a computer-readable storage medium, which stores a computer program executable by an electronic device, and when the program runs on the electronic device, causes the electronic device to perform the steps of any one of the above-mentioned methods. .
本发明提供了一种计算机可读存储介质，其存储有可由电子设备执行的计算机程序，当所述程序在所述电子设备上运行时，使得所述电子设备执行上述任一所述方法的步骤。The present invention provides a computer-readable storage medium, which stores a computer program executable by an electronic device, and when the program runs on the electronic device, causes the electronic device to perform the steps of any one of the above-mentioned methods. .
本发明提供了一种元数据更新系统，所述元数据更新系统包括：如上述应用于安装有元数据管理系统的第一电子设备，和如上述应用于安装有生产系统的第二电子设备。The present invention provides a metadata update system, which includes: as described above, applied to a first electronic device installed with a metadata management system, and as described above, applied to a second electronic device installed with a production system.
本发明提供了一种元数据更新方法、装置、系统、电子设备及存储介质，应用于安装有元数据管理系统的第一电子设备的元数据更新方法包括：接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的；根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。在本发明实施例中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，元数据管理系统对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。The present invention provides a metadata update method, device, system, electronic device and storage medium. The metadata update method applied to a first electronic device installed with a metadata management system includes: receiving metadata change information, wherein the metadata The metadata change information includes the change type and the information of the first metadata after the change, and the information of the change type and the first metadata after the change is obtained by the production system by means of bridging; according to the change type and the change After the information of the first metadata is updated, the corresponding first metadata stored by itself is updated. In the embodiment of the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by means of bridging, and sends it to the metadata management system, and the metadata management system saves the metadata of its own. To update the data, there is no need to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened, and manual participation is not required, thus improving the timeliness of metadata update. performance and efficiency.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.
图1为现有数据仓库下元数据的采集更新过程示意图；Fig. 1 is the schematic diagram of the collection and update process of metadata under the existing data warehouse;
图2为本发明实例1提供的一种元数据更新过程的示意图；2 is a schematic diagram of a metadata update process provided by Example 1 of the present invention;
图3为本发明实施例6提供的一种元数据采集更新过程的示意图；3 is a schematic diagram of a metadata collection and update process according to Embodiment 6 of the present invention;
图4为本发明实施例7提供的一种元数据更新过程的示意图；4 is a schematic diagram of a metadata update process according to Embodiment 7 of the present invention;
图5为本发明实施例7提供的一种元数据采集更新的示意图；5 is a schematic diagram of a metadata collection and update provided by Embodiment 7 of the present invention;
图6为本发明实施例9提供的一种元数据采集更新系统的结构示意图；6 is a schematic structural diagram of a system for collecting and updating metadata according to Embodiment 9 of the present invention;
图7为本发明实施例10提供的一种电子设备的结构示意图；7 is a schematic structural diagram of an electronic device according to Embodiment 10 of the present invention;
图8为本发明实施例11提供的一种电子设备的结构示意图；8 is a schematic structural diagram of an electronic device according to Embodiment 11 of the present invention;
图9为本发明实施例12提供的一种电子设备的结构示意图；9 is a schematic structural diagram of an electronic device according to Embodiment 12 of the present invention;
图10为本发明实施例13提供的一种电子设备的结构示意图；10 is a schematic structural diagram of an electronic device according to Embodiment 13 of the present invention;
图11为本发明实施例提供的一种元数据更新装置示意图；11 is a schematic diagram of a device for updating metadata according to an embodiment of the present invention;
图12为本发明实施例提供的一种元数据更新装置示意图。FIG. 12 is a schematic diagram of an apparatus for updating metadata according to an embodiment of the present invention.
具体实施方式Detailed ways
为了提高了元数据更新的时效性和效率，本发明实施例提供了一种元数据更新方法、装置、系统、电子设备及存储介质。In order to improve the timeliness and efficiency of metadata update, embodiments of the present invention provide a metadata update method, apparatus, system, electronic device, and storage medium.
为了使本发明的目的、技术方案和优点更加清楚，下面将结合附图本发明作进一步地详细描述，显然，所描述的实施例仅仅是本发明的一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
实施例1：Example 1:
图2为本发明实施例提供的一种元数据更新过程的示意图，该过程包括以下步骤：2 is a schematic diagram of a metadata update process provided by an embodiment of the present invention, and the process includes the following steps:
S201：接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的。S201: Receive metadata change information, wherein the metadata change information includes the change type and the information of the first metadata after the change, and the information of the change type and the first metadata after the change is a way of bridging the production system obtained.
本发明实施例提供的元数据更新方法应用于第一电子设备，该第一电子设备安装有元数据管理系统，该第一电子设备可以为桌面计算机、便携式计算机、智能手机、平板电脑、个人数字助理(Personal Digital Assistant，PDA)、服务器等电子设备。The metadata updating method provided by the embodiment of the present invention is applied to a first electronic device, where a metadata management system is installed on the first electronic device, and the first electronic device may be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital Assistant (Personal Digital Assistant, PDA), server and other electronic equipment.
当生产系统通过桥接的方式获取到变更类型及变更后的第一元数据的信息，会将及变更后的第一元数据的信息包括在元数据变更信息中发送给第一电子设备安装的元数据管理系统。When the production system obtains the information of the change type and the changed first metadata by means of bridging, it will include the information of the changed first metadata in the metadata change information and send it to the metadata installed on the first electronic device. data management system.
生产系统通过桥接的方式获取变更类型及变更后的第一元数据的信息时，可以是实时采集的，也可以是每隔较小的时间间隔定期进行的采集，例如2秒，1分钟、10分钟等等，该较小的时间间隔是相对于现有的更新间隔来说的。When the production system obtains the information of the type of change and the first metadata after the change by bridging, it can be collected in real time, or it can be collected periodically at small time intervals, such as 2 seconds, 1 minute, 10 minutes, etc., the smaller time interval is relative to the existing update interval.
生产系统通过桥接的方式获取变更类型及变更后的第一元数据的信息可以由生产系统中的第一探查器组件完成，相应的则可以在元数据管理系统中部署第二探查器组件，从而实时接收生产系统发送的元数据变更信息。The production system obtains the information of the change type and the changed first metadata by means of bridging, which can be completed by the first profiler component in the production system, and correspondingly, the second profiler component can be deployed in the metadata management system, thereby Receive metadata change information sent by the production system in real time.
元数据的变更类型至少包括：新增元数据、修改元数据和删除元数据。The types of metadata changes include at least: adding metadata, modifying metadata, and deleting metadata.
生产系统发送的元数据变更信息中包括变更类型及变更后的第一元数据的信息。The metadata change information sent by the production system includes the change type and the information of the changed first metadata.
如果生产系统获取到多个变更后的第一元数据，可以是在一个元数据变更信息中包括一个变更后的第一元数据的信息及对应的第一元数据的变更类型，通过多个元数据更新信息发送给元数据管理系统，也可以是一个元数据变更信息中包括多个变更后的第一元数据的信息及对应的每个第一元数据的变更类型。If the production system obtains a plurality of changed first metadata, it may include information of one changed first metadata and the corresponding change type of the first metadata in a piece of metadata change information. The data update information is sent to the metadata management system, or a piece of metadata change information includes information of a plurality of changed first metadata and the corresponding change type of each first metadata.
此外，生产系统发送的元数据变更信息中还可以包括第一元数据的标识信息；另外，为了使更新的信息更详细，该元数据变更信息中还可以包括第一元数据对应的数据类型、元数据变更附加信息如变更人、变更时间等。In addition, the metadata change information sent by the production system may also include identification information of the first metadata; in addition, in order to make the updated information more detailed, the metadata change information may also include the data type corresponding to the first metadata, Metadata changes additional information such as who made the change, the time of the change, etc.
S202：根据变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。S202: Update the corresponding first metadata stored by itself according to the change type and the information of the changed first metadata.
当第一电子设备的元数据管理系统接收到元数据变更信息，元数据管理系统根据变更后的第一元数据的信息及第一元数据的变更类型，对自身保存的元数据进行更新。When the metadata management system of the first electronic device receives the metadata change information, the metadata management system updates the metadata stored by itself according to the changed first metadata information and the change type of the first metadata.
元数据管理系统中的元数据保存在元数据存储库中，因此对自身的元数据进行更新可以理解为是对元数据存储库中的元数据进行更新。The metadata in the metadata management system is stored in the metadata repository, so updating its own metadata can be understood as updating the metadata in the metadata repository.
元数据的变更类型至少包括：新增元数据、修改元数据和删除元数据。The types of metadata changes include at least: adding metadata, modifying metadata, and deleting metadata.
当第一元数据的变更类型为新增加元数据，则在元数据存储库中新增加变更后的第一元数据的信息。When the change type of the first metadata is newly added metadata, the information of the changed first metadata is newly added in the metadata repository.
当第一元数据的变更类型为修改元数据，则将元数据存储库中保存的对应的第一元数据的信息修改为变更后的第一元数据的信息。When the modification type of the first metadata is modification metadata, the information of the corresponding first metadata stored in the metadata repository is modified to the changed information of the first metadata.
当第一元数据的变更类型为删除元数据，则删除元数据存储库中保存的对应的第一元数据的信息。如果第一元数据的变更类型为删除元数据，则变更后的第一元数据的信息为空。When the modification type of the first metadata is to delete the metadata, the information of the corresponding first metadata stored in the metadata repository is deleted. If the change type of the first metadata is to delete metadata, the information of the changed first metadata is empty.
元数据管理系统中还可以保存元数据之间的关联关系，则当第一元数据的变更类型为新增元数据时，元数据变更信息中还可以包含有第一元数据与其他元数据的关联关系，元数据管理系统在保存第一元数据时，还可以保存第一元数据与其他元数据的关联关系。The metadata management system can also save the association relationship between the metadata. When the change type of the first metadata is new metadata, the metadata change information can also include the first metadata and other metadata. Associative relationship, when the metadata management system saves the first metadata, the metadata management system may also store the association relationship between the first metadata and other metadata.
下面以一个具体的实施例进行说明，数据采集人员在生产系统的集群中添加了日期元数据及日期元数据的相关数据，生产系统将通过桥接的方式获取到的日期元数据并发送给元数据管理系统，元数据管理系统接收日期元数据并对元数据存储库进行更新，数据处理人员直接根据元数据管理系统更新后的日期元数据，对日期元数据的相关数据进行清洗，元数据存储库将清洗后的数据直接存储，数据分析人员无需任何操作，即可实现元数据管理系统中日期元数据的添加。A specific example will be described below. The data collector adds date metadata and related data of date metadata to the cluster of the production system, and the production system sends the date metadata obtained by bridging to the metadata. The management system, the metadata management system receives the date metadata and updates the metadata repository. The data processing personnel directly clean the related data of the date metadata according to the updated date metadata of the metadata management system, and the metadata repository By storing the cleaned data directly, data analysts can add date metadata in the metadata management system without any operations.
生产系统识别到存在变更的元数据，通过桥接的方式获取，并向元数据管理系统发送给变更后的元数据，元数据管理系统根据接收到的变更后的元数据进行更新，相比于在特定的时间批量采集更新元数据，本发明实施例中元数据变更时进行采集更新的周期大大缩短，因此可承载大数据平台环境下千万级大规模、高性能的元数据管理，避免了批量采集更新元数据时造成的处理更新瓶颈，提高了元数据采集更新时的可用性。The production system recognizes that there is changed metadata, obtains it by bridging, and sends the changed metadata to the metadata management system. The metadata management system updates the received metadata according to the changed metadata. Collect and update metadata in batches at a specific time. In the embodiment of the present invention, the period for collecting and updating metadata is greatly shortened when metadata is changed. Therefore, it can carry tens of millions of large-scale, high-performance metadata management in a big data platform environment, avoiding batch processing. The bottleneck of processing and updating caused by collecting and updating metadata improves the usability when collecting and updating metadata.
在本发明实施例中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，元数据管理系统对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the embodiment of the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by means of bridging, and sends it to the metadata management system, and the metadata management system saves the metadata of its own. To update the data, there is no need to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened, and manual participation is not required, thus improving the timeliness of metadata update. performance and efficiency.
实施例2：Example 2:
为了获取变更后的元数据的信息，在上述实施例的基础上，本发明实施例中，所述接收元数据变更信息后，在所述根据变更类型及待所述变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新之前，所述方法还包括：In order to obtain the information of the changed metadata, on the basis of the above embodiment, in this embodiment of the present invention, after receiving the metadata change information, the first metadata to be changed according to the type of change and the first metadata to be changed Before updating the corresponding first metadata stored by itself, the method further includes:
当所述元数据变更信息中包含文件的信息，且所述变更类型及变更后的第一元数据的信息包含在所述文件的信息中时，对所述文件进行解析，获取变更类型及变更后的第一元数据的信息。When the metadata change information includes file information, and the change type and the changed first metadata information are included in the file information, the file is parsed to obtain the change type and the change After the first metadata information.
元数据管理系统在接收到元数据变更信息后，消息中可能直接携带变更后的第一元数据的信息，也可能携带的是文件的信息，在该文件中包括变更后的第一元数据的信息，因此为了获取到变更类型及该变更后的第一元数据的信息，在本发明实施例中判断所述元数据变更信息中是否包含文件的信息，如果确定元数据变更信息中包含文件的信息，则需要先对文件解析才能获取到变更类型及变更后的元数据的信息。After the metadata management system receives the metadata change information, the message may directly carry the information of the changed first metadata, or may carry the information of the file, and the file includes the information of the changed first metadata. Therefore, in order to obtain the information of the change type and the first metadata after the change, in this embodiment of the present invention, it is determined whether the metadata change information contains the information of the file, if it is determined that the metadata change information contains the information of the file information, you need to parse the file first to obtain the information of the change type and the changed metadata.
元数据管理系统接收到元数据变更信息后，先判断元数据变更信息中是否包含文件的信息。元数据管理系统中预先保存有文件的格式信息，则判断元数据变更信息中是否包含文件的信息可以是，识别元数据变更信息中是否存在与文件的格式信息匹配的信息，如果是，确定元数据变更信息中包含文件的信息，元数据管理系统对元数据变更信息中包含的文件进行解析，获取变更类型及变更后的第一元数据的信息。After receiving the metadata change information, the metadata management system firstly judges whether the metadata change information includes file information. The format information of the file is pre-stored in the metadata management system, and the information to determine whether the metadata change information contains the file may be to identify whether there is information matching the format information of the file in the metadata change information, and if so, determine the metadata. The data change information includes file information, and the metadata management system parses the files included in the metadata change information to obtain the change type and the information of the first metadata after the change.
如果元数据变更信息中不包含文件的信息，则元数据管理系统可以直接在元数据变更信息中变更类型及获取到变更后的第一元数据的信息。If the metadata change information does not contain the information of the file, the metadata management system can directly change the type in the metadata change information and obtain the information of the changed first metadata.
元数据管理系统根据获取到的变更类型及变更后的第一元数据的信息，对自身保存的元数据进行更新。The metadata management system updates the metadata stored by itself according to the acquired change type and the information of the changed first metadata.
由于本发明实施例中，元数据管理系统在接收到元数据变更信息后，如果确定元数据变更信息中包含文件的信息，则对文件进行解析，以获取到变更类型及变更后的元数据的信息。In this embodiment of the present invention, after receiving the metadata change information, if the metadata management system determines that the metadata change information contains file information, it parses the file to obtain the change type and the metadata of the changed metadata. information.
实施例3：Example 3:
在上述各实施例的基础上，本发明实施例中，其中所述元数据变更信息中包含所述第一元数据所属的第一应用的信息，所述对所述文件进行解析包括：Based on the above embodiments, in the embodiment of the present invention, the metadata change information includes information of the first application to which the first metadata belongs, and the parsing of the file includes:
根据预先保存的应用与解析队列的对应关系，确定所述第一应用对应的目标解析队列；determining the target parsing queue corresponding to the first application according to the pre-saved correspondence between the application and the parsing queue;
通过所述目标解析队列对所述文件进行解析。The file is parsed through the target parsing queue.
不同应用的元数据对应的数据类型可能存在差异，并且不同数据类型的元数据在变更时产生的数据量、解析难度、和其他元数据关联度均不同，因此通过应用对应的解析队列对文件进行解析，可以进一步提高元数据更新效率，并且不同数据类型的元数据在变更时要求的响应时间也有所区别，因此还可以满足不同数据类型的响应时间要求。There may be differences in the data types corresponding to the metadata of different applications, and the amount of data generated when the metadata of different data types is changed, the difficulty of parsing, and the degree of association with other metadata are different. Parsing can further improve the metadata update efficiency, and the response time required for metadata changes of different data types is also different, so it can also meet the response time requirements of different data types.
解析队列可以是在元数据管理系统初始化时创建的，也可以是根据需求实时创建的，解析队列可以理解为更新消息队列组件，更新消息队列组件可以是基于RabbitMQ、Kafka等消息队列中间件实现。The parsing queue can be created when the metadata management system is initialized, or it can be created in real time according to requirements. The parsing queue can be understood as an update message queue component, which can be implemented based on RabbitMQ, Kafka and other message queue middleware.
元数据管理系统中预先保存有应用与解析队列的对应关系，因此根据元数据变更信息中包含的第一元数据所属的第一应用的信息，可以确定第一应用对应的目标解析队列。The metadata management system pre-stores the correspondence between the application and the parsing queue. Therefore, the target parsing queue corresponding to the first application can be determined according to the information of the first application to which the first metadata belongs in the metadata change information.
当确定第一应用对应的目标解析队列后，通过目标解析队列对文件进行解析。通过目标解析队列对文件进行解析的过程可以是通过目标解析队列，基于预先保存的解析脚本对文件进行解析，基于解析脚本对文件进行解析的过程属于现有技术，在本发明实施例中不做赘述。After the target parsing queue corresponding to the first application is determined, the file is parsed through the target parsing queue. The process of parsing the file through the target parsing queue may be parsing the file based on a pre-saved parsing script through the target parsing queue, and the process of parsing the file based on the parsing script belongs to the prior art and is not performed in this embodiment of the present invention. Repeat.
通过目标解析队列对文件进行解析时，可以是先将待解析的文件保存到目标解析队列中再进行解析。通过目标解析队列对文件进行解析，获取到变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新，当确定更新完成后，可以将目标解析队列中的文件删除。When parsing a file through the target parsing queue, the file to be parsed may be saved to the target parsing queue before parsing. The file is parsed through the target parsing queue, the information of the changed type and the changed first metadata is obtained, and the corresponding first metadata saved by itself is updated. When the update is determined to be completed, the file in the target parsing queue can be delete.
将文件保存到目标解析队列中可以是由元数据管理系统的第二探查器添加到目标解析队列的。Saving the file to the target parsing queue may be added to the target parsing queue by a second profiler of the metadata management system.
此外如果解析队列对文件解析后还获取到第一元数据的其他信息，例如所属第一应用的信息、第一应用的元数据对应的数据类型、元数据变更附加信息如变更人、变更时间等，第一应用的信息如应用ID(Identification，唯一标识码)、其他信息，则元数据管理系统根据变更类型、变更后的第一元数据的信息及解析到的这些其他信息，对自身保存的对应第一元数据进行更新。In addition, if the parsing queue also obtains other information of the first metadata after parsing the file, such as the information of the first application, the data type corresponding to the metadata of the first application, and the additional information of metadata change, such as the person who changed it, the time of the change, etc. , the information of the first application is such as application ID (Identification, unique identification code), other information, then the metadata management system according to the change type, the information of the first metadata after the change and the other information parsed, it saves It is updated corresponding to the first metadata.
解析队列解析文件时针对第一元数据对应的数据类型，需要解析的数据量、解析难度、关联范围及响应时间的差异如以下表格所示：When the parsing queue parses the file, for the data type corresponding to the first metadata, the differences in the amount of data to be parsed, the difficulty of parsing, the range of association and the response time are shown in the following table:
如上述表格所述，当第一元数据对应的数据类型为DDL(Data DefinitionLanguage，数据自定义语言)、DML(Data ManipulationL language，数据操纵语言)等数据类型时，这种数据类型在三种数据类型中解析文件获取第一元数据时产生的数据量小，解析难度低，关联范围小即与其他元数据关联度低，要求较快的响应时间；当第一元数据对应的数据类型为ETL作业时，这种数据类型在三种数据类型中解析文件获取第一元数据时产生的数据量中等，解析难度中等，关联范围高即与其他元数据关联度高，要求中等的响应时间；当第一元数据对应的数据类型为非结构化文档时，这种数据类型在三种数据类型中解析文件获取第一元数据时产生的数据量大，解析难度高，关联范围中等即与其他元数据关联度中等，要求较慢的响应时间。As described in the above table, when the data type corresponding to the first metadata is DDL (Data Definition Language), DML (Data ManipulationL language, data manipulation language) and other data types, this data type is divided into three data types. In the type, the amount of data generated when parsing the file to obtain the first metadata is small, the difficulty of parsing is low, the correlation scope is small, that is, the correlation with other metadata is low, and a faster response time is required; when the data type corresponding to the first metadata is ETL During the job, this data type generates a medium amount of data when parsing a file among the three data types to obtain the first metadata, the parsing difficulty is medium, the correlation range is high, that is, the correlation with other metadata is high, and the response time is medium; when When the data type corresponding to the first metadata is an unstructured document, this data type generates a large amount of data when parsing the file among the three data types to obtain the first metadata, the parsing difficulty is high, and the correlation range is medium, that is, it is related to other metadata. The data is moderately correlated and requires slow response times.
由于本发明实施例中通过应用对应的解析队列对文件进行解析，可以进一步提高元数据更新效率，还可以满足不同数据类型的响应时间要求。Since the file is parsed by applying the corresponding parsing queue in the embodiment of the present invention, the metadata update efficiency can be further improved, and the response time requirements of different data types can also be met.
实施例4：Example 4:
在上述各实施例的基础上，本发明实施例中，所述方法还包括：On the basis of the foregoing embodiments, in this embodiment of the present invention, the method further includes:
将所述变更后的第一元数据保存在针对每个应用创建的发布队列中；saving the changed first metadata in a release queue created for each application;
针对每个应用，判断该应用是否订阅了所述第一元数据；如果是，通过该应用对应的发布队列将所述变更后的第一元数据，发送给该应用，并在该应用对应的发布队列中删除所述变更后的第一元数据。For each application, determine whether the application has subscribed to the first metadata; if so, send the changed first metadata to the application through the publication queue corresponding to the application, and send the changed first metadata to the application corresponding to the application. The changed first metadata is deleted from the release queue.
对于元数据的实时性要求较高的应用可以预先订阅第一元数据，元数据管理系统可以向订阅了第一元数据的应用，主动推送变更后的第一元数据，从而实时高效地更新元数据应用中的元数据。Applications that require high real-time metadata can subscribe to the first metadata in advance, and the metadata management system can actively push the changed first metadata to the applications that have subscribed to the first metadata, so as to update the metadata in real time and efficiently. Metadata in data applications.
元数据管理系统中预先创建有发布队列，发布队列可以是在元数据管理系统初始化时创建的，也可以是根据需求实时创建的，发布队列可以理解为发布消息队列组件，发布消息队列组件可以是基于RabbitMQ、Kafka等消息队列中间件实现。A release queue is pre-created in the metadata management system. The release queue can be created when the metadata management system is initialized, or it can be created in real time according to requirements. The release queue can be understood as the release message queue component, and the release message queue component can be It is implemented based on message queue middleware such as RabbitMQ and Kafka.
元数据管理系统针对第一应用创建对应的发布队列。因此元数据管理系统可以将变更后的第一元数据保存在每个应用对应的发布队列中。The metadata management system creates a corresponding release queue for the first application. Therefore, the metadata management system can store the changed first metadata in the release queue corresponding to each application.
如果某个或某些应用对第一元数据的实时性要求较高，则这个或这些应用可以订阅第一元数据，元数据管理系统保存订阅信息，其中订阅信息中包括哪个应用订阅了哪个元数据。当第一元数据发生了变更，元数据管理系统针对订阅信息中订阅了第一元数据的每个应用，通过该应用对应的发布队列将变更后的第一元数据发送给该应用。当在确定将变更后的第一元数据发送给了该应用后，则在该应用对应的发布队列中删除变更后的第一元数据。If one or some applications have high real-time requirements for the first metadata, this or these applications can subscribe to the first metadata, and the metadata management system saves the subscription information, where the subscription information includes which application subscribes to which metadata data. When the first metadata is changed, the metadata management system, for each application in the subscription information that subscribes to the first metadata, sends the changed first metadata to the application through the publication queue corresponding to the application. After it is determined that the changed first metadata is sent to the application, the changed first metadata is deleted from the release queue corresponding to the application.
此外如果解析队列对文件解析后还获取到第一元数据的其他信息，例如元数据ID、元数据变更附加信息如变更人、变更时间等其他信息。则元数据管理系统通过订阅了该第一元数据的应用对应的发布队列，将变更后的第一元数据信息、第一元数据的变更类型及这些其他信息发送给订阅了该第一元数据的应用。In addition, if the parsing queue also obtains other information of the first metadata after parsing the file, for example, metadata ID, metadata change additional information such as change person, change time and other information. Then, the metadata management system sends the changed first metadata information, the change type of the first metadata and these other information to the subscribed first metadata through the release queue corresponding to the application subscribed to the first metadata. Applications.
针对每个应用创建对应的发布队列，还可以避免多个应用同时访问同一个发布队列造成资源抢占、性能下降等问题。Creating a corresponding release queue for each application can also avoid problems such as resource preemption and performance degradation caused by multiple applications accessing the same release queue at the same time.
当然，在实际应用中，如果某些应用的元数据需求较少，则为了节省内存空间，可以为这些应用创建一个发布队列，通过该一个发布队列，向这些应用发送更新后的元数据。Of course, in practical applications, if some applications have less metadata requirements, in order to save memory space, a release queue can be created for these applications, and updated metadata can be sent to these applications through the release queue.
下面以一个图3所示的元数据采集更新过程对上述各实施例进行说明，用户操作生产系统的数据库，如执行将两张数据表加工成一张汇总数据的脚本为例，该脚本共包括3条SQL语句，4个元数据变更，则生产系统的第一探查器通过桥接的方式获取到3条SQL语句对应的变更信息，分别为import(导入)变更信息、create_table(建表)变更信息、insert_into(插入)变更信息，将这些变更信息依次进行封装、打包发送给元数据管理系统的第二探查器，第二探查器将这些变更信息发送给解析队列，解析队列经过解析处理，在这些变更信息中识别出4项元数据存在变更，包括变更元数据1、变更元数据2、变更元数据3和变更元数据4，其中变更元数据1、变更元数据2、变更元数据3和变更元数据4均为变更后的元数据，并根据变更后的每项变更元数据和变更类型，对自身保存的对应元数据进行更新。并且将该4项变更后的元数据推送至发布队列，如元数据应用1订阅了变更元数据1和变更元数据3，元数据应用2订阅了变更元数据2和变更元数据4，则通过发布队列将变更元数据1和变更元数据3发送给元数据应用1，通过发布队列将变更元数据2和变更元数据4发送给元数据应用2。The above embodiments will be described below with a metadata collection and update process shown in FIG. 3 . The user operates the database of the production system, such as executing a script for processing two data tables into one summary data as an example. The script includes 3 If there are four SQL statements and four metadata changes, the first profiler of the production system obtains the change information corresponding to the three SQL statements through bridging, namely import (import) change information, create_table (create table) change information, insert_into (insert) change information, encapsulate and package the change information in turn and send it to the second profiler of the metadata management system. The second profiler sends the change information to the parsing queue, and the parsing queue undergoes parsing processing. There are 4 metadata changes identified in the information, including Change Metadata 1, Change Metadata 2, Change Metadata 3 and Change Metadata 4, among which Change Metadata 1, Change Metadata 2, Change Metadata 3 and Change Metadata Data 4 is all metadata after the change, and the corresponding metadata stored by itself is updated according to each changed metadata and the type of change after the change. And push the 4 items of changed metadata to the release queue, if metadata application 1 subscribes to change metadata 1 and change metadata 3, and metadata application 2 subscribes to change metadata 2 and change metadata 4, then pass The release queue sends change metadata 1 and change metadata 3 to metadata application 1, and sends change metadata 2 and change metadata 4 to metadata application 2 through the release queue.
由于本发明实施例中对于元数据的实时性要求较高的应用预先订阅第一元数据，元数据管理系统向订阅了该第一元数据的应用，主动推送变更后的元数据，从而实时高效地更新元数据应用中的元数据。Since the applications that have high real-time requirements for metadata in the embodiment of the present invention subscribe to the first metadata in advance, the metadata management system actively pushes the changed metadata to the applications that have subscribed to the first metadata, thereby achieving real-time and high-efficiency to update the metadata in the metadata application.
实施例5：Example 5:
在上述各实施例的基础上，本发明实施例中，所述方法还包括：On the basis of the foregoing embodiments, in this embodiment of the present invention, the method further includes:
接收第二应用发送的数据获取请求；receiving a data acquisition request sent by the second application;
通过所述第二应用对应的发布队列，将所述第三应用的发布队列中变更后的元数据发送给所述第二应用。The changed metadata in the publishing queue of the third application is sent to the second application through the publishing queue corresponding to the second application.
应用需要获取元数据时，可以发送数据获取请求，元数据管理系统通过应用对应的发布队列，发送将变更后的元数据。When an application needs to obtain metadata, it can send a data acquisition request, and the metadata management system sends the changed metadata through the release queue corresponding to the application.
元数据管理系统中针对第二应用创建有对应的发布队列，因此元数据管理系统在获取到变更后的元数据后，可以将变更后的元数据保存在第二应用的发布队列中。A corresponding release queue is created for the second application in the metadata management system, so after acquiring the changed metadata, the metadata management system can save the changed metadata in the release queue of the second application.
如果第二应用对元数据的实时性要求不高，则第二应用无需订阅元数据。当第二应用存在获取元数据的需求时，向元数据管理系统发送数据获取请求。因为元数据管理系统中创建有针对第二应用的发布队列，则元数据管理系统还可以通过第二应用的发布队列接收第二应用发送的数据获取请求。If the second application does not require high real-time metadata, the second application does not need to subscribe to the metadata. When the second application needs to acquire metadata, it sends a data acquisition request to the metadata management system. Because a release queue for the second application is created in the metadata management system, the metadata management system can also receive a data acquisition request sent by the second application through the release queue of the second application.
元数据管理系统通过第二应用对应的发布队列，将第二应用的发布队列中变更后的元数据发送给第二应用。当确定将变更后的元数据发送给了该第二应用，则在该第二应用对应的发布队列中删除变更后的元数据。The metadata management system sends the changed metadata in the release queue of the second application to the second application through the release queue corresponding to the second application. When it is determined that the changed metadata is sent to the second application, the changed metadata is deleted from the release queue corresponding to the second application.
本发明实施例中，应用获取元数据时，可以发送数据获取请求，元数据管理系统通过应用对应的发布队列，发送将变更后的元数据，实现了元数据应用中的元数据更新。In this embodiment of the present invention, when an application acquires metadata, a data acquisition request may be sent, and the metadata management system transmits the changed metadata through a release queue corresponding to the application, thereby realizing metadata update in the metadata application.
实施例6：Example 6:
在上述各实施例的基础上，本发明实施例中，所述方法还包括：On the basis of the foregoing embodiments, in this embodiment of the present invention, the method further includes:
在每个发布队列中，判断第二元数据保存的时长是否大于设定的时长阈值；In each release queue, determine whether the storage duration of the second metadata is greater than the set duration threshold;
如果是，删除所述第二元数据。If so, delete the second metadata.
将每个发布队列中保存的时长大于时长阈值的第二元数据删除，可以节约内存资源。Deleting the second metadata that is stored in each release queue and whose duration is greater than the duration threshold can save memory resources.
时长阈值预先保存在第一电子设备中，时长阈值可以为6小时、1天或7天等。The duration threshold is pre-stored in the first electronic device, and the duration threshold may be 6 hours, 1 day, or 7 days.
当通过发布队列将变更后的元数据发送给对应的应用后，元数据管理系统会将发布队列中保存的变更后的元数据进行删除，以节约内存资源。而对于没有被发送的变更后的元数据，依然会保存在应用的发布队列中，如果这些变更后的元数据的数量很大，则会消耗设备大量的内存资源。After the changed metadata is sent to the corresponding application through the release queue, the metadata management system will delete the changed metadata saved in the release queue to save memory resources. For the changed metadata that has not been sent, it will still be stored in the application's release queue. If the number of these changed metadata is large, it will consume a lot of memory resources of the device.
每个变更后的元数据保存到应用的发布队列中后，元数据管理系统会记录这些元数据的保存时长。元数据管理系统中保存有元数据保存时长的时长阈值，针对不同数据类型的元数据，时长阈值可以相同，可以不同。当第二元数据保存的时长大于时长阈值时，可以认为目前没有应用对该第二元数据存在获取的需求，因此为了节约内存资源，可以将发布队列中保存的时长大于时长阈值的第二元数据删除。After each changed metadata is saved in the application's release queue, the metadata management system will record the storage duration of these metadata. The metadata management system stores a duration threshold for the duration of metadata storage. For metadata of different data types, the duration threshold may be the same or different. When the storage duration of the second metadata is greater than the duration threshold, it can be considered that there is currently no application need to acquire the second metadata. Therefore, in order to save memory resources, the second metadata stored in the release queue for a duration greater than the duration threshold can be stored. Data deletion.
由于本发明实施例中将每个发布队列中保存的时长大于时长阈值的第二元数据删除，可以节约内存资源。Since in the embodiment of the present invention, the second metadata whose duration is greater than the duration threshold stored in each publishing queue is deleted, memory resources can be saved.
实施例7：Example 7:
图4为本发明实施例提供的一种元数据更新过程示意图，该过程包括以下步骤：4 is a schematic diagram of a metadata update process provided by an embodiment of the present invention, and the process includes the following steps:
S401：通过桥接的方式变更类型及获取变更后的第一元数据的信息。S401: Change the type and obtain the information of the changed first metadata by means of bridging.
本发明实施例提供的元数据更新方法应用于第二电子设备，该第二电子设备安装有生产系统，该第二电子设备可以为桌面计算机、便携式计算机、智能手机、平板电脑、个人数字助理(Personal Digital Assistant，PDA)、服务器等电子设备。The metadata update method provided in the embodiment of the present invention is applied to a second electronic device, where a production system is installed on the second electronic device, and the second electronic device may be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant ( Personal Digital Assistant, PDA), servers and other electronic equipment.
生产系统包括传统数据库系统、MPP(Massively Parallel Processing，大规模并行处理)系统、ETL作业、HDFS(Hadoop Distributed File System，Hadoop分布式文件系统)、Hbase(Hadoop Database，Hadoop数据库)、Hive(Hadoop数据仓库基础工具)系统、Storm(大数据框架)系统，Sqoop(数据库间数据传输的工具)系统等。Production systems include traditional database systems, MPP (Massively Parallel Processing) systems, ETL jobs, HDFS (Hadoop Distributed File System, Hadoop Distributed File System), Hbase (Hadoop Database, Hadoop database), Hive (Hadoop data) Warehouse basic tools) system, Storm (big data framework) system, Sqoop (tool for data transfer between databases) system, etc.
生产系统可以通过桥接的方式获取到变更类型及变更后的第一元数据的信息，生产系统通过桥接的方式获取变更类型及变更后的第一元数据的信息时，可以是实时采集的，也可以是每隔较小的时间间隔定期进行的采集，例如2秒，1分钟、10分钟等等，该较小的时间间隔是相对于现有的更新间隔来说的。The production system can obtain the information of the change type and the changed first metadata through bridging. When the production system obtains the information of the changed type and the changed first metadata through the bridge, it can be collected in real time or The collection may be periodically performed at small time intervals, such as 2 seconds, 1 minute, 10 minutes, etc., and the small time interval is relative to the existing update interval.
生产系统在通过桥接的方式获取变更类型及变更后的第一元数据的信息时，可以由生产系统中的第一探查器组件完成。When the production system obtains the information of the change type and the changed first metadata in a bridging manner, the first profiler component in the production system can complete the information.
元数据的变更类型至少包括：新增元数据、修改元数据和删除元数据。The types of metadata changes include at least: adding metadata, modifying metadata, and deleting metadata.
如果生产系统获取到多个变更后的第一元数据，可以是在一个元数据变更信息中包括一个变更后的第一元数据的信息及对应的第一元数据的变更类型，通过多个元数据更新信息发送给元数据管理系统，也可以是一个元数据变更信息中包括多个变更后的第一元数据的信息及对应的每个第一元数据的变更类型。If the production system obtains a plurality of changed first metadata, it may include information of one changed first metadata and the corresponding change type of the first metadata in a piece of metadata change information. The data update information is sent to the metadata management system, or a piece of metadata change information includes information of a plurality of changed first metadata and the corresponding change type of each first metadata.
S402：将生成的元数据变更信息发送给元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息，使所述元数据管理系统接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对所述元数据管理系统保存的对应第一元数据进行更新。S402: Send the generated metadata change information to a metadata management system, wherein the metadata change information includes the change type and information of the changed first metadata, so that the metadata management system receives the metadata The change information is to update the corresponding first metadata stored in the metadata management system according to the change type and the information of the changed first metadata.
生产系统根据变更类型及变更后的第一元数据的信息，生成元数据变更信息。The production system generates metadata change information according to the change type and the information of the changed first metadata.
生成的元数据变更信息中包括变更类型及变更后的第一元数据的信息。The generated metadata change information includes the change type and the information of the changed first metadata.
此外，元数据变更信息中还可以包括第一元数据的标识信息；另外，为了使更新的信息更详细，该元数据变更信息中还可以包括第一元数据对应的数据类型、元数据变更附加信息如变更人、变更时间等。In addition, the metadata change information may also include identification information of the first metadata; in addition, in order to make the updated information more detailed, the metadata change information may also include the data type corresponding to the first metadata, the metadata change additional Information such as change person, change time, etc.
如果生产系统中的第一探查器组件获取变更类型及变更后的第一元数据的信息，相应的则可以在元数据管理系统中部署第二探查器组件，从而实时接收生产系统发送的元数据变更信息。If the first profiler component in the production system obtains the information of the change type and the changed first metadata, correspondingly, a second profiler component can be deployed in the metadata management system to receive the metadata sent by the production system in real time Change information.
元数据管理系统接收元数据变更信息，根据变更类型及变更后的第一元数据的信息，对元数据管理系统自身保存的对应第一元数据进行更新。The metadata management system receives the metadata change information, and updates the corresponding first metadata stored by the metadata management system itself according to the change type and the information of the changed first metadata.
下面以一个具体的实例进行说明，如图5所示，生产系统包括传统数据库、MPP架构、ETL文件、HDFS、Hbase、Hive、Storm，Sqoop等，当生产系统中的第一探查器通过桥接的方式捕获到了元数据变更命令，并在元数据变更命令中获取到了变更的第一元数据，生成包括变更类型及变更后的第一元数据的信息的元数据变更信息，将元数据变更信息发送给元数据管理系统中的第二探查器，第二探查器将元数据变更信息发送给解析队列进行解析处理关联，元数据管理系统对自身保存的元数据进行更新，将变更后的第一元数据保存到发布队列中，针对订阅了第一元数据的元数据应用，通过元数据应用对应的发布消息队列将变更后的第一元数据推送给元数据应用，元数据应用包括信息浏览应用、血缘分析应用、数据质量应用、资产视图应用、数据安全应用。因为第一探查器实时获取变更的第一元数据，因此元数据管理系统可以实时对变更后的第一元数据进行更新。A specific example is described below. As shown in Figure 5, the production system includes traditional databases, MPP architecture, ETL files, HDFS, Hbase, Hive, Storm, Sqoop, etc. When the first profiler in the production system passes the bridged The metadata change command is captured by the method, and the changed first metadata is obtained in the metadata change command, the metadata change information including the change type and the information of the changed first metadata is generated, and the metadata change information is sent. To the second profiler in the metadata management system, the second profiler sends the metadata change information to the parsing queue for parsing processing and association, and the metadata management system updates the metadata stored by itself, and changes the changed first metadata. The data is stored in the publishing queue. For the metadata application that has subscribed to the first metadata, the changed first metadata is pushed to the metadata application through the publishing message queue corresponding to the metadata application. The metadata application includes the information browsing application, Bloodline analysis application, data quality application, asset view application, data security application. Because the first profiler acquires the changed first metadata in real time, the metadata management system can update the changed first metadata in real time.
在本发明实施例中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，元数据管理系统对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the embodiment of the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by means of bridging, and sends it to the metadata management system, and the metadata management system saves the metadata of its own. To update the data, there is no need to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened, and manual participation is not required, thus improving the timeliness of metadata update. performance and efficiency.
实施例8：Example 8:
在上述实施例的基础上，本发明实施例中，所述通过桥接的方式获取变更类型及变更后的第一元数据的信息包括以下至少一种：On the basis of the above-mentioned embodiment, in the embodiment of the present invention, the information about obtaining the change type and the changed first metadata by means of bridging includes at least one of the following:
截取操作生产系统的数据库产生的指令；判断所述指令是否为进行元数据变更的指令；如果是，获取所述指令中携带的变更类型及变更后的第一元数据的信息；Intercepting the instruction generated by the database of the operation production system; judging whether the instruction is an instruction to change metadata; if so, obtaining the information of the change type carried in the instruction and the first metadata after the change;
解析生产系统的数据库的执行日志，判断所述执行日志中是否存在针对元数据的元数据变更；如果是，获取所述执行日志中的变更类型及变更后的第一元数据的信息；和Parse the execution log of the database of the production system, and determine whether there is a metadata change for the metadata in the execution log; if so, obtain the information of the change type in the execution log and the first metadata after the change; and
获取生产系统的数据库的配置文件，判断所述配置文件中是否存在针对所述第一元数据的元数据变更；如果是，获取所述配置文件中的变更类型及变更后的第一元数据的信息。Obtain the configuration file of the database of the production system, and determine whether there is a metadata change for the first metadata in the configuration file; if so, obtain the change type in the configuration file and the changed first metadata. information.
生产系统可以通过至少一种桥接的方式获取变更类型及变更后的第一元数据的信息。The production system may acquire the information of the change type and the changed first metadata through at least one bridging manner.
第一种、命令拦截：当用户直接操作生产系统的数据库时，会产生操作数据库的指令，生产系统截取到指令后，判断该指令是否为进行元数据变更的指令，如果是，则认为该指令会产生元数据变更，则获取该指令中携带的变更类型及变更后的第一元数据的信息，如果否，则认为该指令不会产生元数据变更，则对该指令不做处理，因为生产系统会对操作数据库的每条指令进行拦截，因此该方式实时性最高。The first one, command interception: when the user directly operates the database of the production system, an instruction to operate the database will be generated. After the production system intercepts the instruction, it determines whether the instruction is an instruction to change metadata. If so, it is considered that the instruction is If metadata changes will occur, obtain the information of the change type carried in the instruction and the first metadata after the change. If not, it is considered that the instruction will not generate metadata changes, and the instruction will not be processed, because the production The system intercepts every instruction that operates the database, so this method has the highest real-time performance.
拦截命令的语言是基于生产系统的数据库应用的数据定义语言DDL、与生产系统的数据库应用的数据操控语言DML等，其中DDL包括create table、alter table、droptable等、DML包括load、import、insert into等。该方式主要适用于生产系统为NoSQL数据库，以及在Hadoop平台各组件中的联机即席查询例如Spark、Strom等。The language of intercepting commands is based on the data definition language DDL of the database application of the production system, and the data manipulation language DML of the database application of the production system, etc. DDL includes create table, alter table, droptable, etc., DML includes load, import, insert into Wait. This method is mainly suitable for NoSQL databases in production systems, and online ad hoc queries in various components of the Hadoop platform, such as Spark and Strom.
命令拦截主要在客户端层面实施，客户端包括命令行(CLI)、命令交互界面等，当用户登录生产系统的客户端后，生产系统的探查器即被激活，用户操作客户端时，操作客户端的指令会立即被拦截。Command interception is mainly implemented at the client level. The client side includes command line (CLI), command interaction interface, etc. When the user logs in to the client of the production system, the profiler of the production system is activated. The command from the terminal will be intercepted immediately.
第二种、日志解析：对于有些生产系统的数据库系统无法进行命令拦截，可以通过解析生产系统的数据库的执行日志来确定是否存在变更的元数据，具体地判断执行日志中是否存在针对元数据的元数据变更；如果是，则获取该执行日志中的变更类型及变更后的第一元数据的信息，如果否，则对该执行日志不做处理。Second, log parsing: For some database systems of production systems that cannot intercept commands, you can determine whether there is changed metadata by parsing the execution log of the database of the production system, and specifically determine whether there is any metadata specific to the execution log. Metadata change; if yes, obtain the change type in the execution log and the information of the first metadata after the change, if not, do not process the execution log.
该种方式主要适用于生产系统为传统关系型数据库、MPP数据库，以及定时运行的批量作业，如ETL作业、MapReduce脚本等。根据日志存放的位置，日志解析在客户端层或服务器层均可实施。对于生产系统为ETL、MapReduce脚本的情况，由于业务逻辑复杂，产生的日志量巨大，可约定日志格式的规范，例如增加标签的方式对日志进行初步的过滤，缩减数据量的同时，也统一了日志格式。由于日志的产生具备一定的规律性，因此可根据时效性要求配置对日志文件的获取频率，时效性要求越高的内容其获取频率也越高，以减少因日志扫描对目标系统资源造成浪费。This method is mainly suitable for traditional relational databases, MPP databases, and batch jobs that run regularly, such as ETL jobs and MapReduce scripts. Depending on where the logs are stored, log parsing can be implemented at either the client layer or the server layer. For the case where the production system is ETL and MapReduce scripts, due to the complex business logic, the amount of logs generated is huge, and the specification of the log format can be agreed, such as adding tags to preliminarily filter the logs, reducing the amount of data, but also unified. log format. Since logs are generated with certain regularity, you can configure the acquisition frequency of log files according to the timeliness requirements. The higher the timeliness requirements are, the higher the acquisition frequency is to reduce the waste of target system resources due to log scanning.
此外，在进行日志的解析时，对于逻辑简单、产生日志量较少的可配置为在生产系统端进行解析，具体可以直接由探查器在生产系统端进行解析；对于处理逻辑复杂、产生日志量较大的，可采取将日志发送至元数据管理系统进行解析，进一步减轻生产系统的压力。In addition, when parsing logs, it can be configured to perform parsing on the production system side for simple logic and less log volume. Specifically, the profiler can perform parsing on the production system side; for complex processing logic and less log volume If it is larger, the log can be sent to the metadata management system for analysis to further reduce the pressure on the production system.
第三种、文件同步：对半结构化数据及非结构化数据可以选择该种方式，获取生产系统的数据库的配置文件，解析配置文件的内容获取变更的元数据，判断配置文件中是否存在针对元数据的元数据变更，如果是，获取该配置文件中的变更类型及变更后的第一元数据的信息，如果否对该配置文件不做处理。The third type, file synchronization: you can choose this method for semi-structured data and unstructured data to obtain the configuration file of the database of the production system, parse the content of the configuration file to obtain the changed metadata, and determine whether there is any target in the configuration file. Metadata change of metadata, if yes, obtain the change type in the configuration file and the information of the first metadata after the change, if not, do not process the configuration file.
主要适用于word、excel等半结构化数据，以及爬虫、文件流等非结构化数据。例如业务部门定期发布的指标规范说明文档。此种方式下的实施策略包括：对于时效性要求较高的情况，可由生产系统通过探查器开放的接口主动通知探查器进行文件同步操作；对于时效性要求较低的情况，可配置探查器定期进行目录扫描，例如每小时扫描一次。It is mainly suitable for semi-structured data such as word and excel, as well as unstructured data such as crawlers and file streams. For example, the specification specification document of indicators released by the business department on a regular basis. The implementation strategy in this way includes: for high timeliness requirements, the production system can actively notify the profiler to perform file synchronization operations through the open interface of the profiler; for low timeliness requirements, the profiler can be configured to periodically Do a directory scan, for example, every hour.
由于本发明实施例中，生产系统通过至少一种桥接的方式获取变更后的第一元数据的信息及第一元数据的变更类型，发送给元数据管理系统，元数据管理系统对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。Because in the embodiment of the present invention, the production system obtains the information of the changed first metadata and the change type of the first metadata through at least one bridging method, and sends them to the metadata management system. To update the metadata, it is not necessary to collect the updated metadata in batches at a specific time, so the time required to update the changed metadata in the metadata management system is greatly shortened, and no manual participation is required, thus improving the metadata update. Timeliness and efficiency.
实施例9：Example 9:
图6为本发明实施例9提供的一种元数据采集更新系统，所述元数据更新系统包括：应用于安装有元数据管理系统的第一电子设备61，和应用于安装有生产系统的第二电子设备62。FIG. 6 is a metadata collection and update system provided in Embodiment 9 of the present invention. The metadata update system includes: a first electronic device 61 applied to a metadata management system installed, and a first electronic device applied to a production system installed Two electronic devices 62 .
为了提高了元数据更新的时效性和效率，本发明实施例提供了一种元数据更新方法系统，该元数据更新系统包括安装有元数据管理系统的第一电子设备，和安装有生产系统的第二电子设备。In order to improve the timeliness and efficiency of metadata update, an embodiment of the present invention provides a metadata update method system. The metadata update system includes a first electronic device with a metadata management system installed, and a production system installed with a first electronic device. second electronic device.
第二电子设备，用于通过桥接的方式获取变更类型及变更后的第一元数据的信息；将生成的元数据变更信息发送给第一电子设备中的元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息。The second electronic device is configured to acquire the information of the change type and the changed first metadata by means of bridging; send the generated metadata change information to the metadata management system in the first electronic device, wherein the metadata The change information includes the change type and information of the changed first metadata.
第一电子设备，用于接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。The first electronic device is configured to receive metadata change information, and update the corresponding first metadata stored by itself according to the change type and the information of the changed first metadata.
所述第一电子设备，还用于所述接收元数据变更信息后，在所述根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新之前，当所述元数据变更信息中包含文件的信息，且所述变更类型及变更后的第一元数据的信息包含在所述文件的信息中时，对所述文件进行解析，获取变更类型及变更后的第一元数据的信息。The first electronic device is further configured to, after receiving the metadata change information, before updating the corresponding first metadata stored by itself according to the change type and the information of the changed first metadata , when the metadata change information includes file information, and the change type and the changed first metadata information are included in the file information, parse the file to obtain the change type and Information of the changed first metadata.
所述第一电子设备，具体用于根据预先保存的应用与解析队列的对应关系，确定所述第一应用对应的目标解析队列；通过所述目标解析队列对所述文件进行解析。The first electronic device is specifically configured to determine a target parsing queue corresponding to the first application according to a pre-saved correspondence between the application and the parsing queue; and parse the file through the target parsing queue.
所述第一电子设备，还用于将所述变更后的第一元数据保存在针对每个应用创建的发布队列中；针对每个应用，判断该应用是否订阅了所述第一元数据；如果是，通过该应用对应的发布队列将所述变更后的第一元数据，发送给该应用，并在该应用对应的发布队列中删除所述变更后的第一元数据。The first electronic device is further configured to save the changed first metadata in a release queue created for each application; for each application, determine whether the application has subscribed to the first metadata; If yes, send the changed first metadata to the application through the release queue corresponding to the application, and delete the changed first metadata from the release queue corresponding to the application.
所述第一电子设备，还用于接收第二应用发送的数据获取请求；通过所述第二应用对应的发布队列，将所述第二应用的发布队列中变更后的元数据发送给所述第二应用。The first electronic device is further configured to receive a data acquisition request sent by a second application; and send the changed metadata in the release queue of the second application to the second application through a release queue corresponding to the second application. Second application.
所述第一电子设备，还用于在每个发布队列中，判断第二元数据保存的时长是否大于设定的时长阈值；如果是，删除所述第二元数据。The first electronic device is further configured to, in each release queue, determine whether the storage duration of the second metadata is greater than a set duration threshold; if so, delete the second metadata.
第二电子设备，具体用于通过桥接的方式获取变更类型及变更后的第一元数据的信息包括以下至少一种：截取操作生产系统的数据库产生的指令；判断所述指令是否为进行元数据变更的指令；如果是，获取所述指令中携带的变更类型及变更后的第一元数据的信息；解析生产系统的数据库的执行日志，判断所述执行日志中是否存在针对元数据的元数据变更；如果是，获取所述执行日志中的变更类型及变更后的第一元数据的信息；和获取生产系统的数据库的配置文件，判断所述配置文件中是否存在针对元数据的元数据变更；如果是，获取所述配置文件中的变更类型及变更后的第一元数据的信息。The second electronic device, which is specifically used to obtain the information of the change type and the changed first metadata by means of bridging, includes at least one of the following: intercepting an instruction generated by operating a database of the production system; judging whether the instruction is metadata The instruction to change; if so, obtain the information of the change type and the first metadata after the change carried in the instruction; parse the execution log of the database of the production system, and determine whether there is metadata for the metadata in the execution log change; if so, obtain the information of the change type in the execution log and the first metadata after the change; and obtain the configuration file of the database of the production system, and determine whether there is a metadata change for the metadata in the configuration file ; If yes, obtain the information of the change type and the changed first metadata in the configuration file.
在本发明实施例中，当第二电子设备中的生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给第一电子设备中的元数据管理系统，元数据管理系统对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the embodiment of the present invention, when there is a metadata change in the production system in the second electronic device, the production system obtains the information of the changed first metadata by bridging, and sends it to the metadata in the first electronic device The management system, the metadata management system updates the metadata stored by itself, and does not need to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened. No human involvement is required, thus improving the timeliness and efficiency of metadata updates.
实施例10：Example 10:
在上述各实施例的基础上，本发明实施例还提供了一种电子设备，如图7所示，包括：处理器701、存储器702和收发机703；On the basis of the foregoing embodiments, an embodiment of the present invention further provides an electronic device, as shown in FIG. 7 , including: a processor 701, a memory 702, and a transceiver 703;
所述处理器701，用于执行读取存储器702中的程序，执行下列过程：控制所述收发机703接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的；根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。The processor 701 is configured to execute the program in the read memory 702, and perform the following process: control the transceiver 703 to receive metadata change information, wherein the metadata change information includes the change type and the changed first metadata. The information of the data, the information of the change type and the changed first metadata is obtained by the production system through bridging; according to the change type and the information of the changed first metadata, the corresponding data stored by itself is The first metadata is updated.
基于同一发明构思，本发明实施例中还提供了一种电子设备，由于上述电子设备解决问题的原理与元数据更新方法相似，因此上述电子设备的实施可以参见方法的实施，重复之处不再赘述。Based on the same inventive concept, the embodiment of the present invention also provides an electronic device. Since the principle of solving the problem of the electronic device is similar to that of the metadata update method, the implementation of the electronic device can refer to the implementation of the method, and the repetition is not repeated. Repeat.
在图7中，总线架构可以包括任意数量的互联的总线和桥，具体由处理器701代表的一个或多个处理器和存储器702代表的存储器的各种电路链接在一起。总线架构还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路链接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。收发机703可以是多个元件，即包括发送机和接收机，提供用于在传输介质上与各种其他装置通信的单元。处理器701负责管理总线架构和通常的处理，存储器702可以存储处理器701在执行操作时所使用的数据。In FIG. 7 , the bus architecture may include any number of interconnected buses and bridges, in particular one or more processors represented by processor 701 and various circuits of memory represented by memory 702 linked together. The bus architecture may also link together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and, therefore, will not be described further herein. Transceiver 703 may be a number of elements, ie, including a transmitter and a receiver, providing a means for communicating with various other devices over a transmission medium. The processor 701 is responsible for managing the bus architecture and general processing, and the memory 702 may store data used by the processor 701 in performing operations.
可选的，处理器701可以是CPU(中央处埋器)、ASIC(Application SpecificIntegrated Circuit，专用集成电路)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)或CPLD(Complex Programmable Logic Device，复杂可编程逻辑器件)。Optionally, the processor 701 may be a CPU (central processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field-Programmable Gate Array, field programmable gate array) or a CPLD (Complex Programmable Logic Device, complex programmable logic devices).
所述处理器701，还用于所述接收元数据变更信息后，在所述根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新之前，当所述元数据变更信息中包含文件的信息，且所述变更类型及变更后的第一元数据的信息包含在所述文件的信息中时，对所述文件进行解析，获取变更类型及变更后的第一元数据的信息。The processor 701 is further configured to, after receiving the metadata change information, before updating the corresponding first metadata stored by itself according to the change type and the information of the changed first metadata, When the metadata change information includes file information, and the change type and the changed first metadata information are included in the file information, the file is parsed to obtain the change type and the change After the first metadata information.
所述处理器701，具体用于根据预先保存的应用与解析队列的对应关系，确定所述第一应用对应的目标解析队列；通过所述目标解析队列对所述文件进行解析。The processor 701 is specifically configured to determine the target parsing queue corresponding to the first application according to the pre-saved correspondence between the application and the parsing queue; and parse the file through the target parsing queue.
所述处理器701，还用于将所述变更后的第一元数据保存在针对每个应用创建的发布队列中；针对每个应用，判断该应用是否订阅了所述第一元数据；如果是，通过该应用对应的发布队列将所述变更后的第一元数据，发送给该应用，并在该应用对应的发布队列中删除所述变更后的第一元数据。The processor 701 is further configured to save the changed first metadata in a release queue created for each application; for each application, determine whether the application subscribes to the first metadata; if Yes, the changed first metadata is sent to the application through the release queue corresponding to the application, and the changed first metadata is deleted from the release queue corresponding to the application.
所述处理器701，还用于接收第二应用发送的数据获取请求；通过所述第二应用对应的发布队列，将所述第二应用的发布队列中变更后的元数据发送给所述第二应用。The processor 701 is further configured to receive a data acquisition request sent by the second application; and send the changed metadata in the release queue of the second application to the first application through the release queue corresponding to the second application. Second application.
所述处理器701，还用于在每个发布队列中，判断第二元数据保存的时长是否大于设定的时长阈值；如果是，删除所述第二元数据。The processor 701 is further configured to, in each release queue, determine whether the storage duration of the second metadata is greater than a set duration threshold; if so, delete the second metadata.
在本发明实施例中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，处理器执行存储器上所存放的程序时，实现对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the embodiment of the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by bridging, and sends it to the metadata management system, and the processor executes the program stored in the memory When updating the metadata stored by itself, it is not necessary to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened, and no manual participation is required. Therefore, the timeliness and efficiency of metadata updating are improved.
实施例11：Example 11:
在上述各实施例的基础上，本发明实施例还提供了一种电子设备，如图8所示，包括：处理器801、存储器802和收发机803；On the basis of the foregoing embodiments, an embodiment of the present invention further provides an electronic device, as shown in FIG. 8 , including: a processor 801, a memory 802, and a transceiver 803;
所述处理器801，用于执行读取存储器802中的程序，执行下列过程：通过桥接的方式获取变更类型及变更后的第一元数据的信息；控制所述收发机803将生成的元数据变更信息发送给元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息，使所述元数据管理系统接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对所述元数据管理系统保存的对应第一元数据进行更新。The processor 801 is configured to execute the program in the read memory 802, and perform the following processes: obtain the information of the changed type and the changed first metadata by means of bridging; control the metadata to be generated by the transceiver 803 The change information is sent to the metadata management system, wherein the metadata change information includes the change type and the information of the first metadata after the change, so that the metadata management system receives the metadata change information, and according to the change The type and the information of the changed first metadata are updated to the corresponding first metadata saved by the metadata management system.
基于同一发明构思，本发明实施例中还提供了一种电子设备，由于上述电子设备解决问题的原理与元数据更新方法相似，因此上述电子设备的实施可以参见方法的实施，重复之处不再赘述。Based on the same inventive concept, the embodiment of the present invention also provides an electronic device. Since the principle of solving the problem of the electronic device is similar to that of the metadata update method, the implementation of the electronic device can refer to the implementation of the method, and the repetition is not repeated. Repeat.
在图8中，总线架构可以包括任意数量的互联的总线和桥，具体由处理器801代表的一个或多个处理器和存储器802代表的存储器的各种电路链接在一起。总线架构还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路链接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。收发机803可以是多个元件，即包括发送机和接收机，提供用于在传输介质上与各种其他装置通信的单元。处理器801负责管理总线架构和通常的处理，存储器802可以存储处理器801在执行操作时所使用的数据。In FIG. 8 , the bus architecture may include any number of interconnected buses and bridges, in particular one or more processors represented by processor 801 and various circuits of memory represented by memory 802 linked together. The bus architecture may also link together various other circuits, such as peripherals, voltage regulators, and power management circuits, which are well known in the art and, therefore, will not be described further herein. Transceiver 803 may be a number of elements, including a transmitter and a receiver, providing a means for communicating with various other devices over a transmission medium. The processor 801 is responsible for managing the bus architecture and general processing, and the memory 802 may store data used by the processor 801 in performing operations.
可选的，处理器801可以是CPU(中央处埋器)、ASIC(Application SpecificIntegrated Circuit，专用集成电路)、FPGA(Field－Programmable Gate Array，现场可编程门阵列)或CPLD(Complex Programmable Logic Device，复杂可编程逻辑器件)。Optionally, the processor 801 may be a CPU (central processor), an ASIC (Application Specific Integrated Circuit, application specific integrated circuit), an FPGA (Field-Programmable Gate Array, field programmable gate array) or a CPLD (Complex Programmable Logic Device, complex programmable logic devices).
所述处理器801，具体用于通过桥接的方式获取变更类型及变更后的第一元数据的信息包括以下至少一种：截取操作生产系统的数据库产生的指令；判断所述指令是否为进行元数据变更的指令；如果是，获取所述指令中携带的变更类型及变更后的第一元数据的信息；解析生产系统的数据库的执行日志，判断所述执行日志中是否存在针对元数据的元数据变更；如果是，获取所述执行日志中的变更类型及变更后的第一元数据的信息；和获取生产系统的数据库的配置文件，判断所述配置文件中是否存在针对元数据的元数据变更；如果是，获取所述配置文件中的变更类型及变更后的第一元数据的信息。The processor 801 is specifically configured to obtain the information of the change type and the changed first metadata by means of bridging, including at least one of the following: intercepting an instruction generated by operating the database of the production system; judging whether the instruction is a process element; The instruction for data change; if so, obtain the information of the change type and the changed first metadata carried in the instruction; parse the execution log of the database of the production system, and determine whether there is a meta-data for the metadata in the execution log data change; if so, obtain the information of the change type in the execution log and the first metadata after the change; and obtain the configuration file of the database of the production system, and determine whether there is metadata for the metadata in the configuration file Change; if yes, obtain the information of the change type and the changed first metadata in the configuration file.
在本发明实施例中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，处理器执行存储器上所存放的程序时，实现对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the embodiment of the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by bridging, and sends it to the metadata management system, and the processor executes the program stored in the memory When updating the metadata stored by itself, it is not necessary to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened, and no manual participation is required. Therefore, the timeliness and efficiency of metadata updating are improved.
实施例12：Example 12:
在上述各实施例的基础上，本发明实施例还提供了一种电子设备，如图9所示，包括：处理器901、通信接口902、存储器903和通信总线904，其中，处理器901，通信接口902，存储器903通过通信总线904完成相互间的通信；On the basis of the foregoing embodiments, an embodiment of the present invention further provides an electronic device, as shown in FIG. 9 , including: a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, The communication interface 902, the memory 903 completes the mutual communication through the communication bus 904;
所述存储器903中存储有计算机程序，当所述程序被所述处理器901执行时，使得所述处理器901执行如下步骤：A computer program is stored in the memory 903, and when the program is executed by the processor 901, the processor 901 is caused to perform the following steps:
接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的；Receive metadata change information, wherein the metadata change information includes the change type and the information of the first metadata after the change, and the information of the change type and the first metadata after the change is obtained by the production system by means of bridging of;
根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。According to the change type and the information of the changed first metadata, the corresponding first metadata stored by itself is updated.
本发明实施例提供的电子设备可以为桌面计算机、便携式计算机、智能手机、平板电脑、个人数字助理(Personal Digital Assistant，PDA)、服务器等电子设备。The electronic device provided by the embodiment of the present invention may be an electronic device such as a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), and a server.
上述电子设备提到的通信总线可以是外设部件互连标准(Peripheral ComponentInterconnect，PCI)总线或扩展工业标准结构(Extended Industry StandardArchitecture，EISA)总线等。该通信总线可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。The communication bus mentioned in the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
通信接口902用于上述电子设备与其他设备之间的通信。The communication interface 902 is used for communication between the above-mentioned electronic device and other devices.
存储器可以包括随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器(Non-Volatile Memory，NVM)，例如至少一个磁盘存储器。可选地，存储器还可以是至少一个位于远离前述处理器的存储装置。The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
上述处理器可以是通用处理器，包括中央处理器、网络处理器(NetworkProcessor，NP)等；还可以是数字信号处理器(Digital Signal Processing，DSP)、专用集成电路、现场可编程门陈列或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。The above-mentioned processor may be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array, or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
在本发明实施例中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，处理器执行存储器上所存放的程序时，实现对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the embodiment of the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by bridging, and sends it to the metadata management system, and the processor executes the program stored in the memory When updating the metadata stored by itself, it is not necessary to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened, and no manual participation is required. Therefore, the timeliness and efficiency of metadata updating are improved.
实施例13：Example 13:
在上述各实施例的基础上，本发明实施例还提供了一种电子设备，如图10所示，包括：处理器1001、通信接口1002、存储器1003和通信总线1004，其中，处理器1001，通信接口1002，存储器1003通过通信总线1004完成相互间的通信；On the basis of the foregoing embodiments, an embodiment of the present invention further provides an electronic device, as shown in FIG. 10 , including: a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, The communication interface 1002 and the memory 1003 complete the communication with each other through the communication bus 1004;
所述存储器1003中存储有计算机程序，当所述程序被所述处理器1001执行时，使得所述处理器1001执行如下步骤：A computer program is stored in the memory 1003, and when the program is executed by the processor 1001, the processor 1001 is caused to perform the following steps:
通过桥接的方式获取变更类型及变更后的第一元数据的信息；Obtain the information of the change type and the changed first metadata by bridging;
将生成的元数据变更信息发送给元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息，使所述元数据管理系统接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对所述元数据管理系统保存的对应第一元数据进行更新。Sending the generated metadata change information to the metadata management system, wherein the metadata change information includes the change type and the information of the first metadata after the change, so that the metadata management system receives the metadata change information , according to the change type and the information of the changed first metadata, update the corresponding first metadata saved by the metadata management system.
本发明实施例提供的电子设备可以为桌面计算机、便携式计算机、智能手机、平板电脑、个人数字助理(Personal Digital Assistant，PDA)、服务器等电子设备。The electronic device provided by the embodiment of the present invention may be an electronic device such as a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (Personal Digital Assistant, PDA), and a server.
上述电子设备提到的通信总线可以是外设部件互连标准(Peripheral ComponentInterconnect，PCI)总线或扩展工业标准结构(Extended Industry StandardArchitecture，EISA)总线等。该通信总线可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。The communication bus mentioned in the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
通信接口1002用于上述电子设备与其他设备之间的通信。The communication interface 1002 is used for communication between the above-mentioned electronic device and other devices.
存储器可以包括随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器(Non-Volatile Memory，NVM)，例如至少一个磁盘存储器。可选地，存储器还可以是至少一个位于远离前述处理器的存储装置。The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
上述处理器可以是通用处理器，包括中央处理器、网络处理器(NetworkProcessor，NP)等；还可以是数字信号处理器(Digital Signal Processing，DSP)、专用集成电路、现场可编程门陈列或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。The above-mentioned processor may be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array, or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
在本发明实施例中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，处理器执行存储器上所存放的程序时，实现对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the embodiment of the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by bridging, and sends it to the metadata management system, and the processor executes the program stored in the memory When updating the metadata stored by itself, it is not necessary to collect the updated metadata in batches at a specific time. Therefore, the time required to update the changed metadata in the metadata management system is greatly shortened, and no manual participation is required. Therefore, the timeliness and efficiency of metadata updating are improved.
图11为本发明实施例提供的一种元数据更新装置示意图，应用于安装有元数据管理系统的第一电子设备，该装置包括：11 is a schematic diagram of an apparatus for updating metadata according to an embodiment of the present invention, which is applied to a first electronic device installed with a metadata management system, and the apparatus includes:
接收模块1101，用于接收元数据变更信息，其中所述元数据变更信息包括变更类型及变更后的第一元数据的信息，所述变更类型及变更后的第一元数据的信息为生产系统通过桥接的方式获取到的；The receiving module 1101 is used to receive metadata change information, wherein the metadata change information includes the change type and the information of the first metadata after the change, and the information of the change type and the first metadata after the change is the production system obtained by bridging;
更新模块1102，用于根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新。The updating module 1102 is configured to update the corresponding first metadata saved by itself according to the change type and the information of the changed first metadata.
所述装置还包括：The device also includes:
获取模块，用于所述接收元数据变更信息后，在所述根据所述变更类型及变更后的第一元数据的信息，对自身保存的对应第一元数据进行更新之前，当所述元数据变更信息中包含文件的信息，且所述变更类型及变更后的第一元数据的信息包含在所述文件的信息中时，对所述文件进行解析，获取变更类型及变更后的第一元数据的信息。The acquiring module is configured to, after receiving the metadata change information, before updating the corresponding first metadata stored by itself according to the change type and the information of the changed first metadata, when the metadata is updated When the data change information includes file information, and the change type and the changed first metadata information are included in the file information, the file is parsed to obtain the change type and the changed first metadata. Metadata information.
所述获取模块，具体用于根据预先保存的应用与解析队列的对应关系，确定所述第一应用对应的目标解析队列；通过所述目标解析队列对所述文件进行解析。The acquiring module is specifically configured to determine the target parsing queue corresponding to the first application according to the pre-saved correspondence between the application and the parsing queue; and parse the file through the target parsing queue.
所述装置还包括：The device also includes:
发送模块，用于将所述变更后的第一元数据保存在针对每个应用创建的发布队列中；针对每个应用，判断该应用是否订阅了所述第一元数据；如果是，通过该应用对应的发布队列将所述变更后的第一元数据，发送给该应用，并在该应用对应的发布队列中删除所述变更后的第一元数据。A sending module, configured to save the changed first metadata in a release queue created for each application; for each application, determine whether the application subscribes to the first metadata; if so, pass the The release queue corresponding to the application sends the changed first metadata to the application, and deletes the changed first metadata from the release queue corresponding to the application.
所述发送模块，还用于接收第二应用发送的数据获取请求；通过所述第二应用对应的发布队列，将所述第二应用的发布队列中变更后的元数据发送给所述第二应用。The sending module is further configured to receive a data acquisition request sent by the second application; and send the changed metadata in the publishing queue of the second application to the second application through the publishing queue corresponding to the second application. application.
所述装置还包括：The device also includes:
删除模块，用于在每个发布队列中，判断第二元数据保存的时长是否大于设定的时长阈值；如果是，删除所述第二元数据。A deletion module, configured to determine whether the storage duration of the second metadata is greater than a set duration threshold in each release queue; if so, delete the second metadata.
本发明提供的一种元数据更新方法、装置、系统、电子设备及存储介质中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，元数据管理系统对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the metadata update method, device, system, electronic device and storage medium provided by the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by means of bridging, and sends it to the production system. For the metadata management system, the metadata management system updates the metadata stored by itself, and does not need to collect the updated metadata in batches at a specific time. Therefore, the metadata management system takes a long time to update the changed metadata. It is shortened and requires no human involvement, thus improving the timeliness and efficiency of metadata updates.
图12为本发明实施例提供的一种元数据更新装置示意图，应用于安装有生产系统的第二电子设备，该装置包括：12 is a schematic diagram of an apparatus for updating metadata provided by an embodiment of the present invention, which is applied to a second electronic device installed with a production system, and the apparatus includes:
获取模块1201，用于通过桥接的方式获取变更类型及变更后的第一元数据的信息；an acquisition module 1201, configured to acquire the information of the change type and the changed first metadata by means of bridging;
发送模块1202，用于将生成的元数据变更信息发送给元数据管理系统，其中所述元数据变更信息中包括所述变更类型及变更后的第一元数据的信息，使所述元数据管理系统接收元数据变更信息，根据所述变更类型及变更后的第一元数据的信息，对所述元数据管理系统保存的对应第一元数据进行更新。The sending module 1202 is configured to send the generated metadata change information to the metadata management system, wherein the metadata change information includes the change type and the information of the changed first metadata, so that the metadata management system The system receives the metadata change information, and updates the corresponding first metadata stored in the metadata management system according to the change type and the information of the changed first metadata.
所述获取模块，具体用于通过桥接的方式获取变更类型及变更后的第一元数据的信息包括以下至少一种：截取操作生产系统的数据库产生的指令；判断所述指令是否为进行元数据变更的指令；如果是，获取所述指令中携带的变更类型及变更后的第一元数据的信息；解析生产系统的数据库的执行日志，判断所述执行日志中是否存在针对元数据的元数据变更；如果是，获取所述执行日志中的变更类型及变更后的第一元数据的信息；和获取生产系统的数据库的配置文件，判断所述配置文件中是否存在针对元数据的元数据变更；如果是，获取所述配置文件中的变更类型及变更后的第一元数据的信息。The obtaining module is specifically used to obtain the information of the changed type and the changed first metadata by means of bridging, including at least one of the following: intercepting an instruction generated by operating a database of the production system; judging whether the instruction is for performing metadata The instruction to change; if so, obtain the information of the change type and the first metadata after the change carried in the instruction; parse the execution log of the database of the production system, and determine whether there is metadata for the metadata in the execution log change; if so, obtain the information of the change type in the execution log and the first metadata after the change; and obtain the configuration file of the database of the production system, and determine whether there is a metadata change for the metadata in the configuration file ; If yes, obtain the information of the change type and the changed first metadata in the configuration file.
本发明提供的一种元数据更新方法、装置、系统、电子设备及存储介质中，当生产系统中存在元数据变更时，生产系统通过桥接的方式获取变更后的第一元数据的信息，发送给元数据管理系统，元数据管理系统对自身保存的元数据进行更新，无需在特定的时间批量采集更新的元数据，因此元数据管理系统中对变更后的元数据进行更新所需的时间大大缩短，也无需人工参与，因此提高了元数据更新的时效性和效率。In the metadata update method, device, system, electronic device and storage medium provided by the present invention, when there is a metadata change in the production system, the production system obtains the information of the changed first metadata by means of bridging, and sends it to the production system. For the metadata management system, the metadata management system updates the metadata saved by itself, and does not need to collect the updated metadata in batches at a specific time. Therefore, the metadata management system takes a long time to update the changed metadata. It is shortened and requires no human involvement, thus improving the timeliness and efficiency of metadata updates.
对于系统/装置实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。For the system/apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.
尽管已描述了本申请的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例做出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本申请范围的所有变更和修改。While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包含这些改动和变型在内。It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711167311.7ACN110019267A (en) | 2017-11-21 | 2017-11-21 | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711167311.7ACN110019267A (en) | 2017-11-21 | 2017-11-21 | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium |
| Publication Number | Publication Date |
|---|---|
| CN110019267Atrue CN110019267A (en) | 2019-07-16 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711167311.7APendingCN110019267A (en) | 2017-11-21 | 2017-11-21 | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium |
| Country | Link |
|---|---|
| CN (1) | CN110019267A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110471927A (en)* | 2019-08-20 | 2019-11-19 | 浙江大搜车软件技术有限公司 | Metadata acquisition method, apparatus, computer equipment and storage medium |
| CN110968592A (en)* | 2019-12-06 | 2020-04-07 | 深圳前海环融联易信息科技服务有限公司 | Metadata acquisition method and device, computer equipment and computer-readable storage medium |
| CN111723082A (en)* | 2020-05-25 | 2020-09-29 | 贵州华泰智远大数据服务有限公司 | Data quality monitoring system based on traceability analysis technology |
| CN112347124A (en)* | 2020-11-12 | 2021-02-09 | 浙江百应科技有限公司 | Metadata management platform and metadata management method |
| CN112465630A (en)* | 2020-12-11 | 2021-03-09 | 天冕信息技术(深圳)有限公司 | Index data processing method, device, equipment and storage medium |
| CN112527783A (en)* | 2020-11-27 | 2021-03-19 | 中科曙光南京研究院有限公司 | Data quality probing system based on Hadoop |
| CN112578996A (en)* | 2019-09-30 | 2021-03-30 | 华为技术有限公司 | Method for sending metadata of storage system and storage system |
| CN112612255A (en)* | 2020-12-24 | 2021-04-06 | 特劢丝软件科技(上海)有限公司 | Data acquisition method and device, electronic equipment and storage medium |
| CN112948405A (en)* | 2021-02-26 | 2021-06-11 | 携程商旅信息服务（上海）有限公司 | Obt data update method, system, device and medium for organizing architecture system |
| CN113138973A (en)* | 2021-04-20 | 2021-07-20 | 建信金融科技有限责任公司 | Data management system and working method |
| CN113449196A (en)* | 2021-07-16 | 2021-09-28 | 北京天眼查科技有限公司 | Information generation method and device, electronic equipment and readable storage medium |
| CN113868253A (en)* | 2021-09-28 | 2021-12-31 | 中通服创立信息科技有限责任公司 | Data relationship capturing and big data relationship tree construction method |
| CN114281835A (en)* | 2021-12-28 | 2022-04-05 | 中国工商银行股份有限公司 | Monitoring method and device for batch version changes |
| CN115098522A (en)* | 2022-06-28 | 2022-09-23 | 平安银行股份有限公司 | Data publishing method and device, electronic equipment and storage medium |
| CN115470217A (en)* | 2022-11-14 | 2022-12-13 | 云筑信息科技(成都)有限公司 | Method for solving change response problem of data bin model in real time |
| CN115905114A (en)* | 2023-03-09 | 2023-04-04 | 浪潮电子信息产业股份有限公司 | Metadata batch update method, system, electronic device and readable storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515290A (en)* | 2009-03-25 | 2009-08-26 | 中国工商银行股份有限公司 | Metadata management system with bidirectional interactive characteristics and implementation method thereof |
| CN102467529A (en)* | 2010-11-11 | 2012-05-23 | 金蝶软件（中国）有限公司 | Metadata synchronization method and system |
| CN106599106A (en)* | 2016-11-28 | 2017-04-26 | 深圳中兴网信科技有限公司 | Distributed file system-based data synchronization method and apparatus |
| US20170207985A1 (en)* | 2014-09-24 | 2017-07-20 | Oracle International Corporation | Managing change events for devices in an enterprise system |
| CN107180116A (en)* | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107229755A (en)* | 2017-06-30 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of distributed system optimization method and equipment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101515290A (en)* | 2009-03-25 | 2009-08-26 | 中国工商银行股份有限公司 | Metadata management system with bidirectional interactive characteristics and implementation method thereof |
| CN102467529A (en)* | 2010-11-11 | 2012-05-23 | 金蝶软件（中国）有限公司 | Metadata synchronization method and system |
| US20170207985A1 (en)* | 2014-09-24 | 2017-07-20 | Oracle International Corporation | Managing change events for devices in an enterprise system |
| CN106599106A (en)* | 2016-11-28 | 2017-04-26 | 深圳中兴网信科技有限公司 | Distributed file system-based data synchronization method and apparatus |
| CN107180116A (en)* | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107229755A (en)* | 2017-06-30 | 2017-10-03 | 郑州云海信息技术有限公司 | A kind of distributed system optimization method and equipment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110471927A (en)* | 2019-08-20 | 2019-11-19 | 浙江大搜车软件技术有限公司 | Metadata acquisition method, apparatus, computer equipment and storage medium |
| CN112578996A (en)* | 2019-09-30 | 2021-03-30 | 华为技术有限公司 | Method for sending metadata of storage system and storage system |
| CN112578996B (en)* | 2019-09-30 | 2024-06-04 | 华为云计算技术有限公司 | Metadata sending method of storage system and storage system |
| CN110968592B (en)* | 2019-12-06 | 2023-11-21 | 深圳前海环融联易信息科技服务有限公司 | Metadata acquisition method, metadata acquisition device, computer equipment and computer readable storage medium |
| CN110968592A (en)* | 2019-12-06 | 2020-04-07 | 深圳前海环融联易信息科技服务有限公司 | Metadata acquisition method and device, computer equipment and computer-readable storage medium |
| CN111723082A (en)* | 2020-05-25 | 2020-09-29 | 贵州华泰智远大数据服务有限公司 | Data quality monitoring system based on traceability analysis technology |
| CN112347124A (en)* | 2020-11-12 | 2021-02-09 | 浙江百应科技有限公司 | Metadata management platform and metadata management method |
| CN112527783A (en)* | 2020-11-27 | 2021-03-19 | 中科曙光南京研究院有限公司 | Data quality probing system based on Hadoop |
| CN112527783B (en)* | 2020-11-27 | 2024-05-24 | 中科曙光南京研究院有限公司 | Hadoop-based data quality exploration system |
| CN112465630A (en)* | 2020-12-11 | 2021-03-09 | 天冕信息技术(深圳)有限公司 | Index data processing method, device, equipment and storage medium |
| CN112465630B (en)* | 2020-12-11 | 2024-03-26 | 天冕信息技术(深圳)有限公司 | Index data processing method, device, equipment and storage medium |
| CN112612255A (en)* | 2020-12-24 | 2021-04-06 | 特劢丝软件科技(上海)有限公司 | Data acquisition method and device, electronic equipment and storage medium |
| CN112612255B (en)* | 2020-12-24 | 2021-10-12 | 上海赛美特软件科技有限公司 | Data acquisition method and device, electronic equipment and storage medium |
| CN112948405A (en)* | 2021-02-26 | 2021-06-11 | 携程商旅信息服务（上海）有限公司 | Obt data update method, system, device and medium for organizing architecture system |
| CN113138973A (en)* | 2021-04-20 | 2021-07-20 | 建信金融科技有限责任公司 | Data management system and working method |
| CN113138973B (en)* | 2021-04-20 | 2022-12-16 | 建信金融科技有限责任公司 | Data management system and working method |
| CN113449196B (en)* | 2021-07-16 | 2024-04-19 | 北京金堤科技有限公司 | Information generation method and device, electronic equipment and readable storage medium |
| CN113449196A (en)* | 2021-07-16 | 2021-09-28 | 北京天眼查科技有限公司 | Information generation method and device, electronic equipment and readable storage medium |
| CN113868253B (en)* | 2021-09-28 | 2024-04-23 | 中通服创立信息科技有限责任公司 | Data relationship capturing and big data relationship tree construction method |
| CN113868253A (en)* | 2021-09-28 | 2021-12-31 | 中通服创立信息科技有限责任公司 | Data relationship capturing and big data relationship tree construction method |
| CN114281835A (en)* | 2021-12-28 | 2022-04-05 | 中国工商银行股份有限公司 | Monitoring method and device for batch version changes |
| CN115098522A (en)* | 2022-06-28 | 2022-09-23 | 平安银行股份有限公司 | Data publishing method and device, electronic equipment and storage medium |
| CN115470217A (en)* | 2022-11-14 | 2022-12-13 | 云筑信息科技(成都)有限公司 | Method for solving change response problem of data bin model in real time |
| CN115905114A (en)* | 2023-03-09 | 2023-04-04 | 浪潮电子信息产业股份有限公司 | Metadata batch update method, system, electronic device and readable storage medium |
| Publication | Publication Date | Title |
|---|---|---|
| CN110019267A (en) | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium | |
| US11941017B2 (en) | Event driven extract, transform, load (ETL) processing | |
| CN109189835B (en) | Method and device for generating data wide table in real time | |
| CN109716320B (en) | Method, system, medium and application processing engine for graph generation of event processing | |
| US10545947B2 (en) | Message personalization over multiple internet messaging campaigns | |
| JP5990192B2 (en) | Filtering query data in the data store | |
| WO2022143045A1 (en) | Method and apparatus for determining data blood relationship, and storage medium and electronic apparatus | |
| CN104298771B (en) | A massive web log data query and analysis method | |
| CN110196888A (en) | Data-updating method, device, system and medium based on Hadoop | |
| JP2017157229A (en) | Scalable analysis platform for semi-structured data | |
| CN103646073A (en) | Condition query optimizing method based on HBase table | |
| CN110390739A (en) | Vehicle data processing method and vehicle data processing system | |
| Narkhede et al. | HMR log analyzer: Analyze web application logs over Hadoop MapReduce | |
| CN113656469B (en) | Big data processing method and device | |
| CN110858197A (en) | A method and apparatus for synchronizing data | |
| US10812322B2 (en) | Systems and methods for real time streaming | |
| US10838931B1 (en) | Use of stream-oriented log data structure for full-text search oriented inverted index metadata | |
| CN107181729A (en) | Data encryption in multi-tenant cloud environment | |
| US9229969B2 (en) | Management of searches in a database system | |
| CN111723063A (en) | A method and device for offline log data processing | |
| Taori et al. | Big Data Management | |
| CN114691672A (en) | Method and apparatus for data processing | |
| Gueidi et al. | A nosql-based approach for real-time managing of embedded data bases | |
| CN109492004A (en) | A kind of number fishery isomeric data storage method, system and device | |
| Johnson et al. | Big data processing using Hadoop MapReduce programming model |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20190716 | |
| RJ01 | Rejection of invention patent application after publication |