技术领域Technical Field
本申请涉及数据处理技术领域，尤其涉及一种用于数据的异构存储方法、系统、设备及介质。The present application relates to the field of data processing technology, and in particular to a method, system, device and medium for heterogeneous storage of data.
背景技术Background technique
随着大数据时代的到来，金融业务系统的数据采集面临着巨大的挑战，特别是随着数据监管对于数据质量、数据安全等方面有越来越高的要求的现状下，金融企业不断加强了自身数据资产管理的力度。由此，在金融业务实时数据同步场景下数据输出到异构存储的流程中需要根据金融行业等级保护要求对流转过程中的数据进行加密处理等中间操作。With the advent of the big data era, data collection in financial business systems faces huge challenges. In particular, as data supervision has increasingly higher requirements for data quality and data security, financial enterprises have continuously strengthened their data asset management. Therefore, in the process of data output to heterogeneous storage in the real-time data synchronization scenario of financial business, it is necessary to perform intermediate operations such as encryption processing on the data in the flow process according to the financial industry's level protection requirements.
然而，在大量的复杂业务场景下，还需要根据业务需要嵌入匹配逻辑等诸多需求，尽管存在大量的数据同步工具可以简化流程的复杂度，但是对于存在大量定制化需求的业务环境仍然会出现流程复杂、开发重复和对于业务变更支持不灵活等问题。同时在不同的异构存储端的数据构建过程中不可避免的需要满足相应的存储要求，这也加大了在实现业务数据输出过程中需要进行大量的业务适配和开发工作的复杂度。因此，现有技术中对大量业务数据的处理和存储的业务流程较为复杂，定制化开发的难度较大。However, in a large number of complex business scenarios, it is also necessary to embed matching logic and other requirements according to business needs. Although there are a large number of data synchronization tools that can simplify the complexity of the process, for business environments with a large number of customized requirements, there will still be problems such as complex processes, repeated development, and inflexible support for business changes. At the same time, in the data construction process of different heterogeneous storage terminals, it is inevitable to meet the corresponding storage requirements, which also increases the complexity of a large amount of business adaptation and development work required in the process of realizing business data output. Therefore, the business process for processing and storing a large amount of business data in the existing technology is relatively complex, and customized development is difficult.
发明内容Summary of the invention
为了克服现有技术中对大量业务数据的处理和存储的业务流程较为复杂，定制化开发的难度较大的问题，本申请提供了一种用于数据的异构存储方法、系统、设备及介质。In order to overcome the problems in the prior art of processing and storing large amounts of business data, such as complex business processes and difficulty in customized development, the present application provides a heterogeneous storage method, system, device and medium for data.
第一方面，为了解决上述技术问题，本申请提供了一种用于数据的异构存储方法，包括：In a first aspect, in order to solve the above technical problems, the present application provides a method for heterogeneous storage of data, including:
获取目标策略任务和待存储数据流；Obtain the target policy tasks and the data stream to be stored;
根据目标策略任务，确定目标策略适配器，目标策略适配器包括上下文策略逻辑；According to the target policy task, determine the target policy adapter, the target policy adapter includes context policy logic;
根据上下文策略逻辑，将待存储数据流进行异构存储；其中，异构存储表征将不同类型的待存储数据流存储到对应适配的异构数据存储器中。According to the contextual policy logic, the data stream to be stored is stored heterogeneously; wherein the heterogeneous storage represents storing different types of data streams to be stored in corresponding adapted heterogeneous data storage devices.
第二方面，本申请还提供了一种用于数据的异构存储系统，包括：In a second aspect, the present application also provides a heterogeneous storage system for data, including:
获取模块，用于获取目标策略任务和待存储数据流；An acquisition module is used to acquire target policy tasks and data streams to be stored;
确定模块，用于根据目标策略任务，确定目标策略适配器，目标策略适配器包括上下文策略逻辑；A determination module, used for determining a target policy adapter according to a target policy task, wherein the target policy adapter includes a context policy logic;
存储模块，用于根据上下文策略逻辑，将待存储数据流进行异构存储；其中，异构存储表征将不同类型的待存储数据流存储到对应适配的异构数据存储器中。The storage module is used to perform heterogeneous storage on the data stream to be stored according to the context policy logic; wherein the heterogeneous storage represents storing different types of data streams to be stored in corresponding adapted heterogeneous data storage devices.
第三方面，本申请还提供了一种计算设备，包括存储器、处理器及存储在存储器上并在处理器上运行的程序，处理器执行程序时实现如上述的一种用于数据的异构存储方法的步骤。In a third aspect, the present application also provides a computing device, including a memory, a processor, and a program stored in the memory and running on the processor, and when the processor executes the program, the steps of a heterogeneous storage method for data as described above are implemented.
第四方面，本申请还提供一种计算机可读存储介质，计算机可读存储介质中存储有指令，当指令在终端设备上运行时，使得终端设备执行一种用于数据的异构存储方法的步骤。In a fourth aspect, the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the steps of a heterogeneous storage method for data.
本申请的有益效果是：通过获取预先设定的目标策略任务，使得能够根据实际的业务需求提前将存储任务标准化，能够减少策略任务的重复开发工作。将对待存储数据流的异构存储过程制定为：根据目标策略任务确定出目标策略任务，并根据目标策略任务中的上下文策略逻辑存储待存储数据流，能够使得待存储数据流的异构存储实现标准化、流程化和自动化，使得待存储数据流的异构存储变得简单，同时减少了待存储数据流的异构存储流程的重复开发工作，从而能够减小对待存储数据流的异构存储流程进行定制化开发的难度。The beneficial effect of the present application is that by obtaining the pre-set target policy tasks, the storage tasks can be standardized in advance according to the actual business needs, and the repetitive development of policy tasks can be reduced. The heterogeneous storage process for the data stream to be stored is formulated as follows: the target policy task is determined according to the target policy task, and the data stream to be stored is stored according to the context policy logic in the target policy task, which can standardize, streamline and automate the heterogeneous storage of the data stream to be stored, making the heterogeneous storage of the data stream to be stored simple, while reducing the repetitive development of the heterogeneous storage process of the data stream to be stored, thereby reducing the difficulty of customized development of the heterogeneous storage process of the data stream to be stored.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1为本申请一种用于数据的异构存储方法的流程示意图；FIG1 is a schematic diagram of a process flow of a heterogeneous data storage method of the present application;
图2为本申请一种用于数据的异构存储方法的另一流程示意图；FIG2 is another schematic diagram of a process for heterogeneous data storage in the present application;
图3为图2所示实施例的具体示例；FIG3 is a specific example of the embodiment shown in FIG2 ;
图4为本申请一种用于数据的异构存储方法的架构图；FIG4 is an architecture diagram of a heterogeneous data storage method of the present application;
图5为本申请一种用于数据的异构存储系统的结构示意图。FIG5 is a schematic diagram of the structure of a heterogeneous storage system for data in the present application.
具体实施方式Detailed ways
下列实施例是对本申请的进一步解释和补充，对本申请不构成任何限制。The following examples are provided to further explain and supplement the present application and do not constitute any limitation to the present application.
以下结合附图描述本申请实施例的一种用于数据的异构存储方法、系统、设备及介质。The following describes a heterogeneous storage method, system, device and medium for data according to an embodiment of the present application in conjunction with the accompanying drawings.
本申请实施例的一种用于数据的异构存储方法，该方法应用于终端设备，本申请方案中以终端设备为执行主体，对本申请方案进行说明，终端设备用于执行一种用于数据的异构存储方法的步骤。An embodiment of the present application provides a heterogeneous storage method for data, which is applied to a terminal device. In the present application scheme, the terminal device is used as the execution subject to illustrate the present application scheme, and the terminal device is used to execute the steps of a heterogeneous storage method for data.
如图1所示，本申请提供了一种用于数据的异构存储方法，包括：As shown in FIG1 , the present application provides a method for heterogeneous storage of data, including:
步骤S1，获取目标策略任务和待存储数据流；Step S1, obtaining the target policy task and the data flow to be stored;
步骤S2，根据目标策略任务，确定目标策略适配器，目标策略适配器包括上下文策略逻辑；Step S2, determining a target policy adapter according to the target policy task, where the target policy adapter includes contextual policy logic;
步骤S3，根据上下文策略逻辑，将待存储数据流进行异构存储；其中，异构存储表征将不同类型的待存储数据流存储到对应适配的异构数据存储器中。Step S3, according to the context policy logic, the data stream to be stored is stored heterogeneously; wherein the heterogeneous storage represents storing different types of data streams to be stored in corresponding adapted heterogeneous data storage devices.
本实施例的一种用于数据的异构存储方法，通过获取预先设定的目标策略任务，使得能够根据实际的业务需求提前将存储任务标准化，能够减少策略任务的重复开发工作。将对待存储数据流的异构存储过程制定为：根据目标策略任务确定出目标策略任务，并根据目标策略任务中的上下文策略逻辑存储待存储数据流，能够使得待存储数据流的异构存储实现标准化、流程化和自动化，使得待存储数据流的异构存储变得简单，同时减少了待存储数据流的异构存储流程的重复开发工作，从而能够减小对待存储数据流的异构存储流程进行定制化开发的难度。A heterogeneous storage method for data in this embodiment obtains a pre-set target policy task, so that the storage task can be standardized in advance according to the actual business needs, and the repetitive development work of the policy task can be reduced. The heterogeneous storage process for the data stream to be stored is formulated as follows: the target policy task is determined according to the target policy task, and the data stream to be stored is stored according to the context policy logic in the target policy task, which can standardize, streamline and automate the heterogeneous storage of the data stream to be stored, making the heterogeneous storage of the data stream to be stored simple, while reducing the repetitive development work of the heterogeneous storage process of the data stream to be stored, thereby reducing the difficulty of customized development of the heterogeneous storage process of the data stream to be stored.
本实施例的方法通过Apache Flink引擎实现。Apache Flink是一个框架和分布式处理引擎，用于在无边界和有边界数据流上进行有状态的计算。Flink引擎能在所有常见集群环境中运行，并能以较快的内存速度和任意规模进行计算。The method of this embodiment is implemented by the Apache Flink engine. Apache Flink is a framework and distributed processing engine for performing stateful computing on unbounded and bounded data streams. The Flink engine can run in all common cluster environments and can perform computing at a faster memory speed and at any scale.
在一些实施例中，如图2所示，获取到策略任务(目标策略任务)后，将当前所有需要进行异构存储的数据流进行合并，即合并N流，合并成一个数据流(待存储数据流)，根据策略任务中的上下文策略逻辑，将待存储数据流进行异构存储操作，输出存储至对应适配的异构数据存储器中，实现待存储数据流的异构存储。In some embodiments, as shown in Figure 2, after obtaining the policy task (target policy task), all current data streams that need to be heterogeneously stored are merged, that is, N streams are merged into one data stream (data stream to be stored), and according to the contextual policy logic in the policy task, the data stream to be stored is subjected to heterogeneous storage operations, and the output is stored in the corresponding adapted heterogeneous data storage device, thereby realizing heterogeneous storage of the data stream to be stored.
可选地，上下文策略逻辑包含有数据存储要求，根据上下文策略逻辑，将待存储数据流进行异构存储，包括：Optionally, the context policy logic includes data storage requirements. According to the context policy logic, the data stream to be stored is stored heterogeneously, including:
根据数据存储要求，确定待存储数据流的类型，数据存储要求表征将不同类型的待存储数据流存储到对应适配的异构数据存储器中的要求；Determine the type of data stream to be stored according to data storage requirements, where the data storage requirements represent the requirements for storing different types of data streams to be stored in corresponding adapted heterogeneous data storage devices;
根据待存储数据流的类型，从预设的存储介质库中确定出对应的目标异构数据存储器；According to the type of the data stream to be stored, a corresponding target heterogeneous data storage is determined from a preset storage medium library;
按照数据存储要求，将待存储数据流存储至目标异构数据存储器中。According to the data storage requirements, the data stream to be stored is stored in the target heterogeneous data storage.
本实施例中，由于上下文策略逻辑包含有将不同类型的待存储数据流存储到对应适配的异构数据存储器中的数据存储要求，说明数据存储要求中包含有待存储数据流的数据流类型，该数据存储要求为异构数据存储器对可存储数据流的存储要求。这样，根据该数据存储要求中的数据流类型，能够直接从预设的存储介质库中确定出对应的目标异构数据存储器，并按照数据存储要求，将待存储数据流存储至对应适配的目标异构数据存储器中，进而能够使得待存储数据流的异构存储实现标准化、流程化和自动化。In this embodiment, since the context policy logic includes a data storage requirement for storing different types of data streams to be stored in a correspondingly adapted heterogeneous data storage, it means that the data storage requirement includes the data stream type of the data stream to be stored, and the data storage requirement is the storage requirement of the heterogeneous data storage for the storable data stream. In this way, according to the data stream type in the data storage requirement, the corresponding target heterogeneous data storage can be directly determined from the preset storage medium library, and according to the data storage requirement, the data stream to be stored is stored in the correspondingly adapted target heterogeneous data storage, thereby enabling the heterogeneous storage of the data stream to be stored to be standardized, streamlined and automated.
利用Apache Flink引擎中的实时同步模块，根据策略上下文中配置的数据存储要求，将待存储数据流按照数据存储要求实时写入到目标异构数据存储器中。图3为图2所示实施例的具体示例，展示了从Apache Flink引擎中的实时同步模块中的目标策略任务对应的业务流程说明。如图3所示，对策略配置任务1(目标策略任务1)的场景进行配置，则策略配置任务1为：将N个数据流合并成的1个待存储数据流存储至目标异构数据存储器“Mongo数据库”中，N个数据流中数据流的数据流类型为事实表。将订单表流和订单详情表流合并为订单详情表(合并表)流，由于订单表流和订单详情表流的数据流类型均为事实表，则根据该策略配置任务1，将该订单详情表(合并表)流存储至Mongo数据库中。对策略配置任务2(目标策略任务2)的场景进行配置，则策略配置任务2为：将N个数据流合并成的1个待存储数据流存储至目标异构数据存储器“HBase数据库”中，N个数据流中数据流的数据流类型包括事实表和维度表。将订单表流、订单详情表流、代理商表流和大区表流合并为订单汇总表(合并表)流，由于订单表流和订单详情表流的数据流类型均为事实表，代理商表流和大区表流的数据流类型均为维度表，则根据该策略配置任务2，将该订单汇总表(合并表)流存储至HBase数据库中。这样，根据该数据存储要求中的数据流类型，能够直接从预设的存储介质库中确定出对应的目标异构数据存储器，并按照数据存储要求，将待存储数据流存储至对应适配的目标异构数据存储器中，进而能够使得待存储数据流的异构存储实现标准化、流程化和自动化。Using the real-time synchronization module in the Apache Flink engine, the data stream to be stored is written to the target heterogeneous data storage in real time according to the data storage requirements configured in the policy context. Figure 3 is a specific example of the embodiment shown in Figure 2, showing the business process description corresponding to the target policy task in the real-time synchronization module in the Apache Flink engine. As shown in Figure 3, the scenario of policy configuration task 1 (target policy task 1) is configured, and then policy configuration task 1 is: store the 1 data stream to be stored merged from N data streams into the target heterogeneous data storage "Mongo database", and the data stream type of the data stream in the N data streams is a fact table. The order table stream and the order details table stream are merged into the order details table (merged table) stream. Since the data stream types of the order table stream and the order details table stream are both fact tables, according to the policy configuration task 1, the order details table (merged table) stream is stored in the Mongo database. Configure the scenario of policy configuration task 2 (target policy task 2), then policy configuration task 2 is: merge N data streams into one data stream to be stored in the target heterogeneous data storage "HBase database", and the data stream types of the data streams in the N data streams include fact tables and dimension tables. Merge the order table stream, order detail table stream, agent table stream and region table stream into an order summary table (merged table) stream. Since the data stream types of the order table stream and the order detail table stream are both fact tables, and the data stream types of the agent table stream and the region table stream are both dimension tables, according to the policy configuration task 2, the order summary table (merged table) stream is stored in the HBase database. In this way, according to the data stream type in the data storage requirement, the corresponding target heterogeneous data storage can be directly determined from the preset storage medium library, and according to the data storage requirement, the data stream to be stored is stored in the corresponding adapted target heterogeneous data storage, thereby enabling the heterogeneous storage of the data stream to be stored to be standardized, streamlined and automated.
可选地，获取目标策略任务，包括：Optionally, obtain the target policy task, including:
利用预设的内置策略适配器生成策略模版；Generate policy templates using the preset built-in policy adapter;
制定策略模版对应的策略调度，策略调度包括策略规则和调度资源；Formulate policy scheduling corresponding to policy templates. Policy scheduling includes policy rules and scheduling resources.
根据策略规则，生成对应的待执行策略任务；Generate corresponding policy tasks to be executed according to policy rules;
对策略规则和调度资源进行校验，获得校验结果；Verify the policy rules and scheduling resources to obtain verification results;
在校验结果表征校验通过时，将待执行策略任务作为目标策略任务。When the verification result indicates that the verification is passed, the strategy task to be executed is used as the target strategy task.
本实施例中，通过预设的内置策略适配器生成策略模版，该策略模版具有通用性，能够使需要进行异构存储的数据流的存储要求实现模版化，从而减少重复开发工作。制定策略模版对应的策略调度，并生成策略调度对应的待执行策略任务，使得该待执行策略任务能够满足待存储数据流进行异构存储的要求。同时，只有在策略调度中的策略规则和调度资源对应的校验结果为校验通过时，才将待执行策略任务作为目标策略任务进行执行，能够提高目标策略任务的安全性，从而提高待存储数据流进行异构存储的安全性。In this embodiment, a policy template is generated by a preset built-in policy adapter. The policy template is universal and can realize template-based storage requirements for data streams that need to be stored heterogeneously, thereby reducing repeated development work. A policy schedule corresponding to the policy template is formulated, and a policy task to be executed corresponding to the policy schedule is generated, so that the policy task to be executed can meet the requirements for heterogeneous storage of the data stream to be stored. At the same time, only when the verification results corresponding to the policy rules and scheduling resources in the policy schedule are verified to pass, the policy task to be executed is executed as the target policy task, which can improve the security of the target policy task, thereby improving the security of heterogeneous storage of the data stream to be stored.
本实施例的内置策略适配器设置在策略中心中，策略中心用于管理策略，并利用内置策略适配器生成待存储数据流进行异构存储的策略模版，能够使需要进行异构存储的数据流的存储要求实现模版化，从而能够生成待存储数据流进行异构存储所需的目标策略任务。其中，内置策略适配器映射生成的策略模版中的策略规则主要结构包含数据源端、数据存储端、数据源逻辑段、数据存储逻辑段和特性定义段等规则结构。The built-in policy adapter of this embodiment is set in the policy center, which is used to manage policies and use the built-in policy adapter to generate a policy template for heterogeneous storage of the data stream to be stored, so that the storage requirements of the data stream that needs to be stored heterogeneously can be templated, thereby generating the target policy task required for heterogeneous storage of the data stream to be stored. Among them, the main structure of the policy rules in the policy template generated by the built-in policy adapter mapping includes rule structures such as the data source end, the data storage end, the data source logic segment, the data storage logic segment and the feature definition segment.
由于待存储数据流的生成场景和数据流类型存在着差异，因此待存储数据流的进行异构存储的业务要求也不同。则需根据待存储数据流的业务要求来制定策略模版对应的策略调度，策略模版根据业务要求定义策略调度中kafka源端(消费者源端)的配置规则段和数据存储段。配置规则段和数据存储段即为策略规则。其中，配置规则段包括生产者配置规则段、消费者配置规则段和服务端配置规则段。同时需要根据分配的Apache Flink引擎的实时集群的资源队列，配置提交策略调度对应的资源配置申请，从而完成策略调度中调度资源的配置。资源配置申请包括申请的内存、core数量(核心数量)和并行度等配置。策略调度中，策略规则的配置和调度资源的配置均会根据数据源端和目标存储端的选择由系统自动生成默认选项，其中调度资源的配置默认项为：系统根据全部数据流的资源情况自动生成默认调度资源的配置项，仅特定情况自定义当前默认调度资源的配置项。策略调度支持Apache Flink引擎的集群环境中调度策略，调度策略包括yarn的FIFO调度、容量调度、公平调度等。Since the generation scenarios and data stream types of the data streams to be stored are different, the business requirements for heterogeneous storage of the data streams to be stored are also different. Therefore, it is necessary to formulate the policy scheduling corresponding to the policy template according to the business requirements of the data streams to be stored. The policy template defines the configuration rule segment and data storage segment of the Kafka source (consumer source) in the policy scheduling according to the business requirements. The configuration rule segment and the data storage segment are the policy rules. Among them, the configuration rule segment includes the producer configuration rule segment, the consumer configuration rule segment and the server configuration rule segment. At the same time, it is necessary to configure and submit the resource configuration application corresponding to the policy scheduling according to the resource queue of the real-time cluster of the allocated Apache Flink engine, so as to complete the configuration of the scheduling resources in the policy scheduling. The resource configuration application includes the configuration of the memory, the number of cores (the number of cores) and the degree of parallelism applied. In the policy scheduling, the configuration of the policy rules and the configuration of the scheduling resources will be automatically generated by the system according to the selection of the data source and the target storage end. The default configuration items of the scheduling resources are: the system automatically generates the configuration items of the default scheduling resources according to the resource conditions of all data streams, and only customizes the configuration items of the current default scheduling resources in specific situations. Policy scheduling supports scheduling strategies in the cluster environment of the Apache Flink engine, including YARN's FIFO scheduling, capacity scheduling, and fair scheduling.
可选地，对策略规则和调度资源进行校验，包括：Optionally, verify the policy rules and scheduling resources, including:
确定策略规则对应的规则约束和对应的内置配置器是否合法；Determine whether the rule constraints corresponding to the policy rules and the corresponding built-in configurators are legal;
确定调度资源是否属于预设的资源范围；Determine whether the scheduled resources fall within the preset resource range;
确定调度资源是否合法。Determine whether the scheduling resources are legal.
本实施例中，由于策略规则由对应的内置配置器来进行配置和输出，而策略规则在执行时需要受到集群环境的约束，只有在策略规则对应的内置配置器合法，即策略规则对应的内置配置器的约束要求符合集群环境的配置约束要求，且策略规则对应的规则约束合法，即策略规则对应的规则约束符合集群环境的规则约束时，该策略规则才能在集群环境中安全执行，从而提高策略规则的合法性。同时，预设的资源范围为Apache Flink引擎的集群环境中能够允许存在的全部用户数据的资源情况，只有调度资源中的资源属于该资源情况中数据的资源范围内，且该调度资源为合法资源时，才能够根据该调度资源对对应的待存储数据源资源进行调度和存储，从而提高调度资源的合法性。这样，对策略规则和调度资源进行校验，能够提高策略调度的合法性。In this embodiment, since the policy rules are configured and output by the corresponding built-in configurator, and the policy rules need to be constrained by the cluster environment when they are executed, only when the built-in configurator corresponding to the policy rules is legal, that is, the constraint requirements of the built-in configurator corresponding to the policy rules meet the configuration constraint requirements of the cluster environment, and the rule constraints corresponding to the policy rules are legal, that is, the rule constraints corresponding to the policy rules meet the rule constraints of the cluster environment, the policy rules can be safely executed in the cluster environment, thereby improving the legitimacy of the policy rules. At the same time, the preset resource range is the resource situation of all user data that can be allowed to exist in the cluster environment of the Apache Flink engine. Only when the resources in the scheduling resources belong to the resource range of the data in the resource situation and the scheduling resources are legal resources, can the corresponding data source resources to be stored be scheduled and stored according to the scheduling resources, thereby improving the legitimacy of the scheduling resources. In this way, verifying the policy rules and scheduling resources can improve the legitimacy of policy scheduling.
可选地，在校验结果表征校验失败时，制定策略模版对应的新的调度策略，新的策略调度包括新的策略规则和新的调度资源；Optionally, when the verification result indicates a verification failure, a new scheduling policy corresponding to the policy template is formulated, and the new policy scheduling includes new policy rules and new scheduling resources;
对新的策略规则和新的调度资源进行校验，确定此次的校验结果，直至校验结果表征校验通过。The new policy rules and new scheduling resources are verified to determine the verification result until the verification result indicates that the verification has passed.
本实施例中，在校验结果表征校验失败时，说明当前的调度策略不合法，在强制执行，会降低待存储数据流进行异构存储过程中的安全性，因此需要重新制定策略模版对应的新的调度策略，并继续对新的调度资源进行校验，直至校验通过后，该校验对应的调度策略才能够用于生成对应的策略任务并执行，从而能够提高待存储数据流进行异构存储过程中的安全性。In this embodiment, when the verification result indicates that the verification has failed, it means that the current scheduling policy is illegal. If it is enforced, the security of the data stream to be stored in the heterogeneous storage process will be reduced. Therefore, it is necessary to re-formulate a new scheduling policy corresponding to the policy template and continue to verify the new scheduling resources until the verification passes. The scheduling policy corresponding to the verification can be used to generate and execute the corresponding policy task, thereby improving the security of the data stream to be stored in the heterogeneous storage process.
可选地，待存储数据流中包括至少一条变动数据，获取待存储数据流，包括：Optionally, the data stream to be stored includes at least one piece of changed data, and obtaining the data stream to be stored includes:
实时采集多条用户数据；Collect multiple user data in real time;
在预设数据库中对每条用户数据执行查找操作，确定每条用户数据对应的匹配结果；Perform a search operation on each piece of user data in a preset database to determine a matching result corresponding to each piece of user data;
对于每条用户数据，在匹配结果表征未查找到匹配数据时，将用户数据作为变动数据；For each piece of user data, when the matching result indicates that no matching data is found, the user data is regarded as changed data;
将全部变动数据作为待存储数据流。All changed data is taken as the data stream to be stored.
本实施例中，由于实时采集的用户数据中可能存在部分数据已经存储过，则只将该实时采集的用户数据中没有存储在历史采集的预设数据库中的数据，作为变动数据，并将全部变动数据作为待存储数据流进行异构存储，能够减少实时采集的用户数据中重复数据的无效存储，还能加快实时采集的用户数据中有效数据的存储速度。In this embodiment, since some of the user data collected in real time may have been stored, only the data in the user data collected in real time that is not stored in the preset database of historical collection is used as the changed data, and all the changed data is stored heterogeneously as the data stream to be stored, which can reduce the invalid storage of duplicate data in the user data collected in real time and speed up the storage speed of valid data in the user data collected in real time.
可选地，根据目标策略任务，确定目标策略适配器，包括：Optionally, according to the target policy task, a target policy adapter is determined, including:
对目标策略任务进行解析，获得解析结果；Analyze the target strategy task and obtain the analysis result;
在预设的策略适配器库中对解析结果进行映射匹配，确定出对应的目标策略适配器。The parsing results are mapped and matched in the preset policy adapter library to determine the corresponding target policy adapter.
本实施例中，目标策略任务中包含有对待存储数据流进行异构存储的执行适配要求，对目标策略任务进行解析后，获得的解析结果能够了解到对应的执行适配要求，从而能够根据该执行适配要求从预设的策略适配器库中匹配出相同执行适配要求的目标策略适配器，该目标策略适配器中的上下文策略逻辑包含了目标策略任务中对待存储数据流进行异构存储的执行步骤，使得能够根据该目标策略适配器中的上下文策略逻辑，将待存储数据流进行异构存储，从而使得待存储数据流的异构存储实现标准化、流程化和自动化。In this embodiment, the target policy task includes execution adaptation requirements for heterogeneous storage of data streams to be stored. After parsing the target policy task, the parsing result obtained can understand the corresponding execution adaptation requirements, so that according to the execution adaptation requirements, a target policy adapter with the same execution adaptation requirements can be matched from a preset policy adapter library. The context policy logic in the target policy adapter includes the execution steps for heterogeneous storage of data streams to be stored in the target policy task, so that the data streams to be stored can be heterogeneously stored according to the context policy logic in the target policy adapter, thereby making the heterogeneous storage of the data streams to be stored standardized, streamlined and automated.
在一些实施例中，目标策略任务通过网关接口将策略调度提交到Apache Flink集群，Apache Flink的实时调度任务模块根据目标策略任务进行启动、下线、编辑和删除等管理操作。其中调度任务启动任务或在线任务在启动提交或运行过程中出现异常，调度内置重试机制将会进行重试，在重试策略范围内发起异常预警及异常任务重试后成功等预警提示。重试策略范围外发起主动预警机制通知已经配置的预警目标消息通道。In some embodiments, the target policy task submits the policy scheduling to the Apache Flink cluster through the gateway interface, and the real-time scheduling task module of Apache Flink performs management operations such as startup, offline, editing, and deletion according to the target policy task. If an exception occurs during the startup submission or operation of the scheduling task startup task or online task, the built-in retry mechanism of the scheduling will retry, and issue an abnormal warning within the scope of the retry strategy and an abnormal task will be successfully retried. An active warning mechanism is initiated outside the scope of the retry strategy to notify the configured warning target message channel.
预设的策略适配器库存储在Apache Flink引擎的集群中，预设的策略适配器库中包含有多个预设策略适配器，如表1所示。The preset policy adapter library is stored in the cluster of the Apache Flink engine. The preset policy adapter library contains multiple preset policy adapters, as shown in Table 1.
表1Table 1
预设策略适配器为数据存储适配器、类型适配器、安全适配器或数据格式适配器等。其中，数据存储适配器主要针对数据存储等异构数据存储器的适配功能，该异构数据存储器包含且不限于mongo数据库、HBase数据库、ElsticSearch(以下简称为es)数据库、mysql数据库、oracle数据库等，该数据存储适配器包含mongodb适配器、hbase适配器、es适配器、mysql适配器等。类型适配器主要完成了对异构的数据源和数据存储源两端的类型相关的映射和转换等适配功能，该类型适配器包含mysql-mongo适配器、mysql-hbase适配器、mysql-es适配器、oracle-mongo适配器等。安全适配器主要完成了对通用安全、业务自定义安全和其他安全需求等安全策略的适配功能，该安全适配器包含加密适配器、策略合法性适配器、黑白名单适配器、自定义安全适配器等。数据格式适配器主要完成了对时间、特定格式校验等适配功能，该数据格式适配器包含时间格式适配器、json格式适配器、业务特定格式适配器、自定义格式适配器等适配组件。涉及到源端的数据通过消息队列获取，并按照策略适配器的上下文策略逻辑对数据流进行实时数据同步任务逻辑处理，即按照策略适配器的上下文策略逻辑对待存储数据流进行异构存储。The preset policy adapter is a data storage adapter, a type adapter, a security adapter or a data format adapter, etc. Among them, the data storage adapter mainly targets the adaptation function of heterogeneous data storage such as data storage, and the heterogeneous data storage includes but is not limited to mongo database, HBase database, ElsticSearch (hereinafter referred to as es) database, mysql database, oracle database, etc. The data storage adapter includes mongodb adapter, hbase adapter, es adapter, mysql adapter, etc. The type adapter mainly completes the adaptation functions such as mapping and conversion related to the types of heterogeneous data sources and data storage sources at both ends, and the type adapter includes mysql-mongo adapter, mysql-hbase adapter, mysql-es adapter, oracle-mongo adapter, etc. The security adapter mainly completes the adaptation function of security policies such as general security, business customized security and other security requirements, and the security adapter includes encryption adapter, policy legitimacy adapter, black and white list adapter, custom security adapter, etc. The data format adapter mainly completes the adaptation functions such as time and specific format verification, and the data format adapter includes time format adapter, json format adapter, business specific format adapter, custom format adapter and other adaptation components. The data related to the source end is obtained through the message queue, and the real-time data synchronization task logic is processed on the data stream according to the context policy logic of the policy adapter, that is, the storage data stream is heterogeneously stored according to the context policy logic of the policy adapter.
本申请提供一种用于数据的异构存储方法，能够解决目前金融业务实时数据同步过程中因存在大量定制化需求而产生的流程复杂、重复开发和对于业务变更支持不灵活等问题。图4为本申请一种用于数据的异构存储方法的架构图，如图4所示，包括用于生成目标策略任务的策略中心、用于收集待存储数据流的消息系统、用于确定目标策略适配器并对待存储数据流进行异构存储的Apache Flink实时同步模块和用于储存待存储数据流的异构存储模块。The present application provides a heterogeneous storage method for data, which can solve the problems of complex processes, repeated development, and inflexible support for business changes caused by a large number of customized requirements in the current real-time data synchronization process of financial services. Figure 4 is an architectural diagram of a heterogeneous storage method for data in the present application. As shown in Figure 4, it includes a policy center for generating target policy tasks, a message system for collecting data streams to be stored, an Apache Flink real-time synchronization module for determining target policy adapters and performing heterogeneous storage on data streams to be stored, and a heterogeneous storage module for storing data streams to be stored.
通过实时数据采集系统实时采集到Actor(用户)数据，并将采集到的数据在数据库(该数据库中的数据为历史采集和当前采集到的全部数量的数据)中进行匹配，确定出变动数据，形成待存储数据流同时形成数据库的增量变动日志。将待存储数据流发送至消息系统，同时消息系统将该待存储数据流发送至Apache Flink实时同步模块。The real-time data collection system collects Actor (user) data in real time, matches the collected data in the database (the data in the database is the total amount of historical and currently collected data), determines the changed data, forms the data stream to be stored, and forms the incremental change log of the database. The data stream to be stored is sent to the message system, and the message system sends the data stream to be stored to the Apache Flink real-time synchronization module.
策略中心包括内置策略适配器和策略管理器，通过内置策略适配器生成多个策略模版，多个策略模版形成一个模版列表，策略管理器根据待存储数据流的存储任务要求选择对应的策略模版，并制定策略模版对应的策略调度，策略调度包括策略规则和调度资源，以及生成该策略规则对应的目标策略任务。将目标策略任务发送至Apache Flink实时同步模块。The policy center includes a built-in policy adapter and a policy manager. Multiple policy templates are generated through the built-in policy adapter. Multiple policy templates form a template list. The policy manager selects the corresponding policy template according to the storage task requirements of the data stream to be stored, and formulates the policy scheduling corresponding to the policy template. The policy scheduling includes policy rules and scheduling resources, as well as generating the target policy task corresponding to the policy rule. The target policy task is sent to the Apache Flink real-time synchronization module.
Apache Flink实时同步模块执行同步任务：对策略调度进行策略解析和策略映射，确定出对应的目标策略适配器，目标策略适配器包括上下文策略逻辑。根据上下文策略逻辑，将待存储数据流进行异构存储，存储至对应的目标异构数据存储器中。其中，目标异构数据存储器包括Mongo数据库、HBase数据库、ElsticSearch数据库等。Apache Flink real-time synchronization module performs synchronization tasks: performs policy parsing and policy mapping on policy scheduling, determines the corresponding target policy adapter, and the target policy adapter includes context policy logic. According to the context policy logic, the data stream to be stored is stored heterogeneously and stored in the corresponding target heterogeneous data storage. Among them, the target heterogeneous data storage includes Mongo database, HBase database, ElsticSearch database, etc.
本申请通过策略中心对金融数据实时同步过程中大量需要定制的复杂业务开发进行了有效的抽象及自动化，避免了大量的重复开发工作，同时通过灵活的策略规则配置极大提升了应对业务需求变更的灵活性。This application uses the policy center to effectively abstract and automate a large number of complex business developments that require customization during the real-time synchronization of financial data, avoiding a large amount of repetitive development work. At the same time, it greatly improves the flexibility of responding to changes in business needs through flexible policy rule configuration.
进一步地，策略中心通过策略管理模块进行管理和创建策略调度，策略调度可以支持多种格式的策略模版，支持多项适配各类业务策略的内置适配器，并根据内置策略的定义进行差错检验。策略调度创建后并启动调度任务后自动发起实时同步任务。Furthermore, the policy center manages and creates policy scheduling through the policy management module. Policy scheduling can support policy templates in multiple formats, multiple built-in adapters that adapt to various business policies, and perform error checks based on the definition of built-in policies. After the policy scheduling is created and the scheduling task is started, the real-time synchronization task is automatically initiated.
本申请通过多服务器组成的数据采集集群、策略中心、消息系统和Apache Flink实时同步模块的功能整合，提高了对于金融数据同步开发中存在大量定制化场景的适配能力。通过策略中心页面载入由内置策略适配器中心生成的策略模版，根据同步业务需求配置完成策略规则项配置。策略调度选择生成的策略规则数据创建目标策略任务，并将目标策略任务发送给Apache Flink实时同步集群进行同步任务。Apache Flink实时同步模块对目标策略任务进行解析，并通过策略映射模块找到对应的目标策略适配器，策略适配器适配策略规则的数据将消息系统中的数据库变动数据(待存储数据流)进行处理并将数据实时同步至异构目标数据库。通过上述过程，本申请通过配置中心将复杂的业务变更需求以可配置化的策略规则操作进行简化，交由内置策略适配器去自动化调度任务去执行同步功能，有效解决了目前对于存在大量定制化需求的业务环境仍然会出现流程复杂、开发重复和对于业务变更支持不灵活等问题。This application improves the adaptability to a large number of customized scenarios in the development of financial data synchronization through the functional integration of a data collection cluster, a policy center, a message system, and an Apache Flink real-time synchronization module composed of multiple servers. Load the policy template generated by the built-in policy adapter center through the policy center page, and complete the configuration of the policy rule items according to the synchronization business requirements. The policy scheduler selects the generated policy rule data to create a target policy task, and sends the target policy task to the Apache Flink real-time synchronization cluster for synchronization tasks. The Apache Flink real-time synchronization module parses the target policy task, and finds the corresponding target policy adapter through the policy mapping module. The policy adapter adapts the data of the policy rules to process the database change data (data stream to be stored) in the message system and synchronizes the data to the heterogeneous target database in real time. Through the above process, this application simplifies the complex business change requirements with configurable policy rule operations through the configuration center, and hands it over to the built-in policy adapter to automatically schedule tasks to perform synchronization functions, effectively solving the current business environment with a large number of customized requirements. There are still problems such as complex processes, repeated development, and inflexible support for business changes.
如图5所示，本申请提供了一种用于数据的异构存储系统，包括：As shown in FIG5 , the present application provides a heterogeneous storage system for data, including:
获取模块，用于获取目标策略任务和待存储数据流；An acquisition module is used to acquire target policy tasks and data streams to be stored;
确定模块，用于根据目标策略任务，确定目标策略适配器，目标策略适配器包括上下文策略逻辑；A determination module, used for determining a target policy adapter according to a target policy task, wherein the target policy adapter includes a context policy logic;
存储模块，用于根据上下文策略逻辑，将待存储数据流进行异构存储；其中，异构存储表征将不同类型的待存储数据流存储到对应适配的异构数据存储器中。The storage module is used to perform heterogeneous storage on the data stream to be stored according to the context policy logic; wherein the heterogeneous storage represents storing different types of data streams to be stored in corresponding adapted heterogeneous data storage devices.
可选地，存储模块，具体用于：Optionally, the storage module is specifically used to:
根据数据存储要求，确定待存储数据流的类型；Determine the type of data stream to be stored based on data storage requirements;
根据待存储数据流的类型，从预设的存储介质库中确定出对应的目标异构数据存储器；According to the type of the data stream to be stored, a corresponding target heterogeneous data storage is determined from a preset storage medium library;
按照数据存储要求，将待存储数据流存储至目标异构数据存储器中。According to the data storage requirements, the data stream to be stored is stored in the target heterogeneous data storage.
可选地，获取模块，具体用于：Optionally, obtain a module, specifically for:
利用预设的内置策略适配器生成策略模版；Generate policy templates using the preset built-in policy adapter;
制定策略模版对应的策略调度，策略调度包括策略规则和调度资源；Formulate policy scheduling corresponding to policy templates. Policy scheduling includes policy rules and scheduling resources.
根据策略规则，生成对应的待执行策略任务；Generate corresponding policy tasks to be executed according to policy rules;
对策略规则和调度资源进行校验，获得校验结果；Verify the policy rules and scheduling resources to obtain verification results;
在校验结果表征校验通过时，将待执行策略任务作为目标策略任务。When the verification result indicates that the verification is passed, the strategy task to be executed is used as the target strategy task.
可选地，获取模块，具体用于：Optionally, obtain a module, specifically for:
确定策略规则对应的规则约束和对应的内置配置器是否合法；Determine whether the rule constraints corresponding to the policy rules and the corresponding built-in configurators are legal;
确定调度资源是否属于预设的资源范围；Determine whether the scheduled resources fall within the preset resource range;
确定调度资源是否合法。Determine whether the scheduling resources are legal.
可选地，获取模块，具体用于：Optionally, obtain a module, specifically for:
在校验结果表征校验失败时，制定策略模版对应的新的调度策略，新的策略调度包括新的策略规则和新的调度资源；When the verification result indicates that the verification has failed, a new scheduling policy corresponding to the policy template is formulated, and the new policy scheduling includes new policy rules and new scheduling resources;
对新的策略规则和新的调度资源进行校验，确定此次的校验结果，直至校验结果表征校验通过。The new policy rules and new scheduling resources are verified to determine the verification result until the verification result indicates that the verification has passed.
可选地，获取模块，具体用于：Optionally, obtain a module, specifically for:
实时采集多条用户数据；Collect multiple user data in real time;
在预设数据库中对每条用户数据执行查找操作，确定每条用户数据对应的匹配结果；Perform a search operation on each piece of user data in a preset database to determine a matching result corresponding to each piece of user data;
对于每条用户数据，在匹配结果表征未查找到匹配数据时，将用户数据作为变动数据；For each piece of user data, when the matching result indicates that no matching data is found, the user data is regarded as changed data;
将全部变动数据作为待存储数据流。All changed data is taken as the data stream to be stored.
可选地，确定模块，具体用于：Optionally, a module is determined, specifically for:
对目标策略任务进行解析，获得解析结果；Analyze the target strategy task and obtain the analysis result;
在预设的策略适配器库中对解析结果进行映射匹配，确定出对应的目标策略适配器。The parsing results are mapped and matched in the preset policy adapter library to determine the corresponding target policy adapter.
本申请实施例的一种计算设备，包括存储器、处理器及存储在存储器上并在处理器上运行的程序，处理器执行程序时实现上述一种用于数据的异构存储方法的部分或全部步骤。A computing device according to an embodiment of the present application includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, some or all steps of the above-mentioned heterogeneous storage method for data are implemented.
其中，计算设备可以选用电脑，相对应地，其程序为电脑软件，且上述关于本申请的一种计算设备中的各参数和步骤，可参考上文中一种用于数据的异构存储方法的实施例中的各参数和步骤，在此不做赘述。Among them, the computing device can be a computer, and correspondingly, its program is computer software. The above-mentioned parameters and steps in a computing device of the present application can refer to the parameters and steps in an embodiment of a heterogeneous storage method for data mentioned above, and will not be repeated here.
本申请实施例中一种计算机可读存储介质，计算机可读存储介质中存储有指令，当指令在运行时，执行上述的一种用于数据的异构存储方法的步骤。In an embodiment of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed, the steps of the above-mentioned heterogeneous storage method for data are executed.
其中，计算机可读存储介质可以是暂态计算机可读存储介质，也可以是非暂态计算机可读存储介质。The computer-readable storage medium may be a transient computer-readable storage medium or a non-transient computer-readable storage medium.
本公开实施例的技术方案可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括一个或多个指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开实施例方法的全部或部分步骤。而前述的计算机可读存储介质可以是非暂态计算机可读存储介质，包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，RandomAccess Memory)、磁碟或者光盘等多种可以存储程序代码的介质，也可以是暂态计算机可读存储介质。The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of the embodiment of the present disclosure. The aforementioned computer-readable storage medium may be a non-transitory computer-readable storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, and other media that can store program codes, or a transient computer-readable storage medium.
附图中的流程图和框图，图示了按照本申请各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。其中，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
所属技术领域的技术人员知道，本申请可以实现为系统、方法或计算机程序产品。因此，本公开可以具体实现为以下形式，即：可以是完全的硬件、也可以是完全的软件(包括固件、驻留软件、微代码等)，还可以是硬件和软件结合的形式，本文一般称为“电路”、“模块”或“系统”。此外，在一些实施例中，本申请还可以实现为在一个或多个计算机可读介质中的计算机程序产品的形式，该计算机可读介质中包含计算机可读的程序代码。计算机可读存储介质例如可以是但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。Those skilled in the art know that the present application can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present application can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above.
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
尽管上面已经示出和描述了本申请的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本申请的限制，本领域的普通技术人员在本申请的范围内可以对上述实施例进行变化、修改、替换和变型。Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410357738.7ACN118296066B (en) | 2024-03-27 | 2024-03-27 | Heterogeneous storage method, heterogeneous storage system, heterogeneous storage equipment and heterogeneous storage medium for data |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410357738.7ACN118296066B (en) | 2024-03-27 | 2024-03-27 | Heterogeneous storage method, heterogeneous storage system, heterogeneous storage equipment and heterogeneous storage medium for data |
| Publication Number | Publication Date |
|---|---|
| CN118296066Atrue CN118296066A (en) | 2024-07-05 |
| CN118296066B CN118296066B (en) | 2025-09-09 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410357738.7AActiveCN118296066B (en) | 2024-03-27 | 2024-03-27 | Heterogeneous storage method, heterogeneous storage system, heterogeneous storage equipment and heterogeneous storage medium for data |
| Country | Link |
|---|---|
| CN (1) | CN118296066B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120185911A1 (en)* | 2010-09-30 | 2012-07-19 | Khandys Polite | Mlweb: a multilevel web application framework |
| US20140207928A1 (en)* | 2011-12-29 | 2014-07-24 | Intel Corporation | Providing Different Levels of Service Over a Storage Transport |
| CN108959398A (en)* | 2018-06-04 | 2018-12-07 | 北京至信普林科技有限公司 | Isomery storage expansion system and method |
| CN111858466A (en)* | 2020-06-30 | 2020-10-30 | 苏州浪潮智能科技有限公司 | A data storage method, device, device and storage medium |
| CN113391890A (en)* | 2021-04-16 | 2021-09-14 | 北京沃东天骏信息技术有限公司 | Task processing method, device and equipment and computer storage medium |
| CN113806332A (en)* | 2021-08-23 | 2021-12-17 | 北京金蝶云基科技有限公司 | Heterogeneous system integrated data processing method and device and computer equipment |
| CN115757587A (en)* | 2022-11-30 | 2023-03-07 | 上海浦东发展银行股份有限公司 | Heterogeneous data source integration method and device, electronic equipment and storage medium |
| CN116739336A (en)* | 2023-05-09 | 2023-09-12 | 广西电网有限责任公司 | Power grid disaster early warning method and system based on multi-source heterogeneous data fusion model |
| CN116795778A (en)* | 2023-06-30 | 2023-09-22 | 南京维拓科技股份有限公司 | File storage management method based on strategy mode |
| CN117033952A (en)* | 2023-10-10 | 2023-11-10 | 厦门惟尔拓科技有限公司 | Distributed multi-data source acquisition implementation method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120185911A1 (en)* | 2010-09-30 | 2012-07-19 | Khandys Polite | Mlweb: a multilevel web application framework |
| US20140207928A1 (en)* | 2011-12-29 | 2014-07-24 | Intel Corporation | Providing Different Levels of Service Over a Storage Transport |
| CN108959398A (en)* | 2018-06-04 | 2018-12-07 | 北京至信普林科技有限公司 | Isomery storage expansion system and method |
| CN111858466A (en)* | 2020-06-30 | 2020-10-30 | 苏州浪潮智能科技有限公司 | A data storage method, device, device and storage medium |
| CN113391890A (en)* | 2021-04-16 | 2021-09-14 | 北京沃东天骏信息技术有限公司 | Task processing method, device and equipment and computer storage medium |
| CN113806332A (en)* | 2021-08-23 | 2021-12-17 | 北京金蝶云基科技有限公司 | Heterogeneous system integrated data processing method and device and computer equipment |
| CN115757587A (en)* | 2022-11-30 | 2023-03-07 | 上海浦东发展银行股份有限公司 | Heterogeneous data source integration method and device, electronic equipment and storage medium |
| CN116739336A (en)* | 2023-05-09 | 2023-09-12 | 广西电网有限责任公司 | Power grid disaster early warning method and system based on multi-source heterogeneous data fusion model |
| CN116795778A (en)* | 2023-06-30 | 2023-09-22 | 南京维拓科技股份有限公司 | File storage management method based on strategy mode |
| CN117033952A (en)* | 2023-10-10 | 2023-11-10 | 厦门惟尔拓科技有限公司 | Distributed multi-data source acquisition implementation method |
| Title |
|---|
| 吴念祖: "上海空港", 31 October 2010, 上海科学技术出版社, pages: 120 - 124* |
| 苑希民: "重大防汛应急决策三维电子沙盘关键技术及应用", 31 December 2019, 中国水利水电出版社, pages: 144 - 147* |
| Publication number | Publication date |
|---|---|
| CN118296066B (en) | 2025-09-09 |
| Publication | Publication Date | Title |
|---|---|---|
| CN110704518B (en) | Business data processing method and device based on Flink engine | |
| CN111190753B (en) | Distributed task processing method and device, storage medium and computer equipment | |
| CN113052696B (en) | Financial business task processing method, device, computer equipment and storage medium | |
| CN111177113A (en) | Data migration method and device, computer equipment and storage medium | |
| CN112989171A (en) | Data query method, device, equipment and medium | |
| CN106886535A (en) | A kind of data pick-up method and apparatus for being adapted to multiple data sources | |
| CN108763080A (en) | Test data method for uploading, device, computer equipment and storage medium | |
| CN112905323A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112990850A (en) | Flow implementation method and system based on rule engine | |
| CN106708965A (en) | Data processing method and apparatus | |
| CN111522729A (en) | Method, device and system for determining rule release | |
| CN113360300B (en) | Interface call link generation method, device, equipment and readable storage medium | |
| CN111475494A (en) | Mass data processing method, system, terminal and storage medium | |
| CN114461955A (en) | Method for automatically generating http interface based on web page configuration | |
| WO2024021498A1 (en) | Data processing method and apparatus for virtual resource, storage medium, and processor | |
| CN114756611B (en) | Artificial intelligence platform sample library management method and system | |
| CN118861296A (en) | Reasoning process determination method, device, equipment and medium based on large language model | |
| CN119002925A (en) | Link relation identification method, device, equipment, medium and program product | |
| CN117311675B (en) | Service configuration method and device related to distributed system and electronic equipment | |
| CN118296066A (en) | A heterogeneous storage method, system, device and medium for data | |
| CN116258469B (en) | Service data processing method, system, equipment and medium based on communication software | |
| CN113742036A (en) | Index processing method and device and electronic equipment | |
| CN111984744B (en) | Information processing method based on remote communication and artificial intelligence and cloud service platform | |
| CN111563250A (en) | Rights management method, apparatus, computer equipment and storage medium | |
| US12204943B1 (en) | System and method for dynamically allocating computer resources to a data processing pipeline |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |