技术领域Technical Field
本发明涉及数据处理技术领域，尤其涉及一种数据存储优化方法、装置、电子设备及存储介质。The present invention relates to the field of data processing technology, and in particular to a data storage optimization method, device, electronic device and storage medium.
背景技术Background technique
随着信息化时代的到来，出现了越来越多的数据，尤其对于页面数据进行处理的过程中，也需要数据的查询和数据的存储。如何合理化地进行数据存储，并且节省存储空间和提高数据查询速率的技术至关重要。With the advent of the information age, more and more data has appeared. Especially in the process of processing page data, data query and data storage are also needed. How to reasonably store data, save storage space and improve data query speed is crucial.
发明人在实现本发明的过程中，发现现有技术存在如下缺陷：目前，数据库中存储的数据大部分是原始数据，并在原始数据上做聚合再次存储，双重存储导致数据量太多，查询速度慢。另外的，对于数据库中存储的历史数据，一般来说很少使用，但是却占用极大存储成本。In the process of implementing the present invention, the inventors found that the prior art has the following defects: At present, most of the data stored in the database is raw data, and the raw data is aggregated and stored again. The double storage leads to too much data and slow query speed. In addition, the historical data stored in the database is generally rarely used, but it occupies a huge storage cost.
发明内容Summary of the invention
本发明提供了一种数据存储优化方法、装置、电子设备及存储介质，以实现减少了数据存储成本，提高了快速聚合数据和加快数据查询的效率。The present invention provides a data storage optimization method, device, electronic device and storage medium, so as to reduce the data storage cost and improve the efficiency of fast data aggregation and accelerated data query.
根据本发明的一方面，提供了一种数据存储优化方法，其中，包括：According to one aspect of the present invention, a data storage optimization method is provided, which includes:
实时获取数据采集器中的多条采集数据，并指示时序数据库将各所述采集数据进行存储；Acquire multiple pieces of collected data from the data collector in real time, and instruct the time series database to store each of the collected data;
指示聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；Instructing the aggregation calculation engine to obtain each of the collected data from the time series database, and to perform an aggregation operation on the collected data according to a preset joint aggregation rule, so as to obtain data aggregation results of different granularities;
指示所述时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；Instruct the time series database to obtain each data aggregation result, and store each data aggregation result in each pre-constructed time granularity table;
指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。The time series database is instructed to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.
根据本发明的另一方面，提供了一种数据存储优化装置，其中，包括：According to another aspect of the present invention, there is provided a data storage optimization device, comprising:
采集数据获取模块，用于实时获取数据采集器中的多条采集数据，并指示时序数据库将各所述采集数据进行存储；The acquisition data acquisition module is used to acquire multiple acquisition data in the data collector in real time and instruct the time series database to store each of the acquisition data;
数据聚合结果确定模块，用于指示聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；A data aggregation result determination module is used to instruct the aggregation calculation engine to obtain each of the collected data from the time series database, and perform an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities;
数据聚合结果存储模块，用于指示所述时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；A data aggregation result storage module is used to instruct the time series database to obtain each data aggregation result, and store each data aggregation result in each pre-constructed time granularity table;
数据聚合存储结果确定模块，用于指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。The data aggregation storage result determination module is used to instruct the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.
根据本发明的另一方面，提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，其中，所述处理器执行所述计算机程序时实现本发明任一实施例所述的数据存储优化方法。According to another aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data storage optimization method described in any embodiment of the present invention when executing the computer program.
根据本发明的另一方面，提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使处理器执行时实现本发明任一实施例所述的数据存储优化方法。According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data storage optimization method described in any embodiment of the present invention when executed.
本发明实施例的技术方案，通过实时获取数据采集器中的多条采集数据，并指示预设的时序数据库将各所述采集数据进行存储；指示预设的聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；指示时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；指示时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。解决了对采集到的数据直接存储原始数据而导致的数据量大和查询不方便的问题，减少了数据存储成本，提高了快速聚合数据和加快数据查询的效率。The technical solution of the embodiment of the present invention obtains multiple collected data in the data collector in real time, and instructs the preset time series database to store each of the collected data; instructs the preset aggregation calculation engine to obtain each of the collected data from the time series database, and performs the collection data aggregation operation according to the preset joint aggregation rules to obtain data aggregation results of different granularities; instructs the time series database to obtain each data aggregation result, and stores each data aggregation result in each pre-built time granularity table; instructs the time series database to perform data compression processing on each time granularity table to obtain data aggregation storage results, so as to complete the storage optimization operation of the collected data according to the data aggregation storage results. The problem of large data volume and inconvenient query caused by directly storing the original data of the collected data is solved, the data storage cost is reduced, and the efficiency of fast data aggregation and accelerated data query is improved.
应当理解，本部分所描述的内容并非旨在标识本发明的实施例的关键或重要特征，也不用于限制本发明的范围。本发明的其它特征将通过以下的说明书而变得容易理解。It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
图1是根据本发明实施例一提供的一种数据存储优化方法的流程图；FIG1 is a flow chart of a data storage optimization method provided according to an embodiment of the present invention;
图2是根据本发明实施例二提供的一种数据存储优化装置的结构示意图；FIG2 is a schematic diagram of the structure of a data storage optimization device provided according to a second embodiment of the present invention;
图3是根据本发明实施例三提供的一种电子设备的结构示意图。FIG3 is a schematic diagram of the structure of an electronic device provided according to Embodiment 3 of the present invention.
具体实施方式Detailed ways
为了使本技术领域的人员更好地理解本发明方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分的实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本发明保护的范围。In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
需要说明的是，本发明的说明书和权利要求书及上述附图中的术语“目标”、“当前”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本发明的实施例能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
实施例一Embodiment 1
图1为本发明实施例一提供了一种数据存储优化方法的流程图，本实施例可适用于对采集到的数据进行存储优化和查询的情况，该方法可以由数据存储优化装置来执行，该数据存储优化装置可以采用硬件和/或软件的形式实现。Figure 1 is a flowchart of a data storage optimization method provided in accordance with a first embodiment of the present invention. This embodiment is applicable to situations where storage optimization and query of collected data are performed. The method can be executed by a data storage optimization device, which can be implemented in the form of hardware and/or software.
相应的，如图1所示，该方法包括：Accordingly, as shown in FIG1 , the method includes:
S110、实时获取数据采集器中的多条采集数据，并指示时序数据库将各所述采集数据进行存储。S110, acquiring multiple pieces of collected data from the data collector in real time, and instructing the time series database to store each piece of the collected data.
其中，数据采集器可以是对数据进行采集的设备，可以将采集到的多条数据发送给时序数据库。The data collector may be a device for collecting data, and may send the collected multiple data to a time series database.
具体的，时序数据库可以接收采集数据并进行数据存储，时序数据库可以是Influx DB数据库，这里不做具体限定。详细的，在时序数据库将采集到的数据存储于第一时间粒度表，假设第一时间粒度表对应的是1分钟时间粒度表，也即在时序数据库中，是以较小的粒度进行数据存储的。Specifically, the time series database can receive the collected data and store the data. The time series database can be an Influx DB database, which is not specifically limited here. In detail, the collected data is stored in the first time granularity table in the time series database. Assume that the first time granularity table corresponds to a 1-minute time granularity table, that is, in the time series database, data is stored at a smaller granularity.
S120、指示聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果。S120, instructing the aggregation calculation engine to obtain each of the collected data from the time series database, and perform an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities.
其中，聚合计算引擎可以是对采集数据进行获取和聚合处理得到的引擎。具体的，聚合计算引擎可以是Apache Spark，这里不做具体限定。The aggregation computing engine may be an engine that acquires and aggregates the collected data. Specifically, the aggregation computing engine may be Apache Spark, which is not specifically limited here.
其中，联合聚合规则可以是预先设置的对采集数据进行聚合的规则。数据聚合结果可以是对采集数据进行聚合处理，得到的聚合结果。The joint aggregation rule may be a pre-set rule for aggregating the collected data, and the data aggregation result may be an aggregation result obtained by aggregating the collected data.
在本实施例中，假设聚合计算引擎是Apache Spark，时序数据库是Influx DB数据库。通过Apache Spark来获取Influx DB数据库中的采集数据，采集数据为第一时间粒度表中的数据。In this embodiment, it is assumed that the aggregation computing engine is Apache Spark and the time series database is Influx DB. The collected data in the Influx DB database is obtained through Apache Spark, and the collected data is the data in the first time granularity table.
进一步的，可以根据联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果。Furthermore, the collected data can be aggregated according to the joint aggregation rules to obtain data aggregation results of different granularities.
可选的，所述联合聚合规则包括第一联合聚合规则、第二联合聚合规则和第三联合聚合规则；所述根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果，包括：通过所述第一联合聚合规则来进行采集数据聚合操作，得到第一数据聚合结果；通过所述第二联合聚合规则来进行采集数据聚合操作，得到第二数据聚合结果；通过所述第三联合聚合规则来进行采集数据聚合操作，得到第三数据聚合结果。Optionally, the joint aggregation rules include a first joint aggregation rule, a second joint aggregation rule and a third joint aggregation rule; the collecting data aggregation operation is performed according to the pre-set joint aggregation rules to obtain data aggregation results of different granularities, including: performing the collecting data aggregation operation through the first joint aggregation rule to obtain a first data aggregation result; performing the collecting data aggregation operation through the second joint aggregation rule to obtain a second data aggregation result; performing the collecting data aggregation operation through the third joint aggregation rule to obtain a third data aggregation result.
其中，第一联合聚合规则可以是对第一时间粒度表中的数据通过第一时间进行聚合的规则。第二联合聚合规则可以是对第一时间粒度表中的数据通过第二时间进行聚合的规则。第三联合聚合规则可以是对第一时间粒度表中的数据通过第三时间进行聚合的规则。这里，第一时间小于第二时间，第二时间小于第三时间。The first joint aggregation rule may be a rule for aggregating data in the first time granularity table by the first time. The second joint aggregation rule may be a rule for aggregating data in the first time granularity table by the second time. The third joint aggregation rule may be a rule for aggregating data in the first time granularity table by the third time. Here, the first time is less than the second time, and the second time is less than the third time.
示例性的，假设第一时间是5分钟，第二时间是30分钟，第三时间是60分钟。Exemplarily, it is assumed that the first time is 5 minutes, the second time is 30 minutes, and the third time is 60 minutes.
具体的，对第一时间粒度表(1分钟时间粒度表)中的采集数据，每次满5分钟进行一次数据聚合，并且对聚合后的结果进行取平均值的处理，来得到第一数据聚合结果，并且需要记录当前存储时间，发送于Influx DB数据库。Specifically, for the collected data in the first time granularity table (1-minute time granularity table), data aggregation is performed every 5 minutes, and the average value of the aggregated results is taken to obtain the first data aggregation result, and the current storage time needs to be recorded and sent to the Influx DB database.
同理可知，对第一时间粒度表(1分钟时间粒度表)中的采集数据，每次满30分钟进行一次数据聚合，并且对聚合后的结果进行取平均值的处理，来得到第二数据聚合结果，并且需要记录当前存储时间，发送于Influx DB数据库。Similarly, for the collected data in the first time granularity table (1-minute time granularity table), data aggregation is performed every 30 minutes, and the average value of the aggregated results is taken to obtain the second data aggregation result, and the current storage time needs to be recorded and sent to the Influx DB database.
相应的，对第一时间粒度表(1分钟时间粒度表)中的采集数据，每次满60分钟进行一次数据聚合，并且对聚合后的结果进行取平均值的处理，来得到第三数据聚合结果，并且需要记录当前存储时间，发送于Influx DB数据库。Correspondingly, for the collected data in the first time granularity table (1-minute time granularity table), data aggregation is performed every 60 minutes, and the average value of the aggregated results is taken to obtain the third data aggregation result, and the current storage time needs to be recorded and sent to the Influx DB database.
S130、指示所述时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中。S130, instructing the time series database to obtain each data aggregation result, and storing each data aggregation result in each pre-constructed time granularity table.
其中，时间粒度表包括第一时间粒度表、第二时间粒度表、第三时间粒度表和第四时间粒度表。第一时间粒度表的时间粒度小于第二时间粒度表；第二时间粒度表的时间粒度小于第三时间粒度表；第三时间粒度表的时间粒度小于第四时间粒度表。The time granularity table includes a first time granularity table, a second time granularity table, a third time granularity table and a fourth time granularity table. The time granularity of the first time granularity table is smaller than that of the second time granularity table; the time granularity of the second time granularity table is smaller than that of the third time granularity table; and the time granularity of the third time granularity table is smaller than that of the fourth time granularity table.
续前例的，可以设置第一时间粒度表为1分钟时间粒度表；第二时间粒度表为5分钟时间粒度表；第三时间粒度表为30分钟时间粒度表；第四时间粒度表为60分钟时间粒度表。Continuing with the previous example, the first time granularity table can be set to a 1-minute time granularity table; the second time granularity table can be set to a 5-minute time granularity table; the third time granularity table can be set to a 30-minute time granularity table; and the fourth time granularity table can be set to a 60-minute time granularity table.
可选的，所述将各数据聚合结果分别存储于预先构建的各时间粒度表中，包括：将所述第一数据聚合结果存储于所述第二时间粒度表；将所述第二数据聚合结果存储于所述第三时间粒度表；将所述第三数据聚合结果存储于所述第四时间粒度表。Optionally, the data aggregation results are stored separately in pre-constructed time granularity tables, including: storing the first data aggregation result in the second time granularity table; storing the second data aggregation result in the third time granularity table; storing the third data aggregation result in the fourth time granularity table.
续前例的，将满5分钟进行一次数据聚合而得到的第一数据聚合结果，存储于第二时间粒度表，并且需要记录相应的当前存储时间。将满30分钟进行一次数据聚合而得到的第二数据聚合结果，存储于第三时间粒度表，并且需要记录相应的当前存储时间。将满60分钟进行一次数据聚合而得到的第三数据聚合结果，存储于第四时间粒度表，并且需要记录相应的当前存储时间。Continuing with the previous example, the first data aggregation result obtained by performing data aggregation once every 5 minutes is stored in the second time granularity table, and the corresponding current storage time needs to be recorded. The second data aggregation result obtained by performing data aggregation once every 30 minutes is stored in the third time granularity table, and the corresponding current storage time needs to be recorded. The third data aggregation result obtained by performing data aggregation once every 60 minutes is stored in the fourth time granularity table, and the corresponding current storage time needs to be recorded.
S140、指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。S140, instructing the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.
其中，数据聚合存储结果可以是对各个时间粒度表中的采集数据或者采集聚合数据进行数据压缩处理，得到的处理结果。The data aggregation storage result may be a processing result obtained by performing data compression processing on the collected data or collected aggregated data in each time granularity table.
可选的，在所述指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果之后，还包括：在各所述时间粒度表中，依次获取一个当前时间粒度表；获取所述当前时间粒度表中的各采集数据或者各聚合数据分别对应的当前存储时间，以及获取与所述当前时间粒度表相匹配的预先设置的删除策略；根据所述当前存储时间和当前删除策略，来进行各采集数据或者各聚合数据的删除操作；返回执行所述依次获取一个当前时间粒度表的操作，直至遍历完成各所述时间粒度表。Optionally, after instructing the time series database to perform data compression processing on each time granularity table to obtain the data aggregation storage result, it also includes: obtaining a current time granularity table in each time granularity table in turn; obtaining the current storage time corresponding to each collected data or each aggregated data in the current time granularity table, and obtaining a preset deletion policy that matches the current time granularity table; deleting each collected data or each aggregated data according to the current storage time and the current deletion policy; returning to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.
其中，删除策略可以是对不同的时间粒度表中的采集数据进行删除处理的策略。The deletion strategy may be a strategy for deleting the collected data in tables of different time granularities.
其中，第一时间粒度表对应第一删除策略；第二时间粒度表对应第二删除策略；第三时间粒度表对应第三删除策略；第四时间粒度表对应第四删除策略。Among them, the first time granularity table corresponds to the first deletion policy; the second time granularity table corresponds to the second deletion policy; the third time granularity table corresponds to the third deletion policy; and the fourth time granularity table corresponds to the fourth deletion policy.
在本实施例中，由于第一时间粒度表存储的数据是未聚合的数据，也即为采集数据，则需要对各个采集数据对应的当前存储时间，并进行判断是否满足第一删除策略。另外的，由于第二时间粒度表、第三时间粒度表和第四时间粒度表是聚合的数据，也即为聚合数据，则需要获取每个聚合数据分别对应的当前存储时间，并进行判断是否满足各自对应的删除策略。In this embodiment, since the data stored in the first time granularity table is unaggregated data, that is, collected data, it is necessary to obtain the current storage time corresponding to each collected data and determine whether it satisfies the first deletion policy. In addition, since the second time granularity table, the third time granularity table, and the fourth time granularity table are aggregated data, that is, aggregated data, it is necessary to obtain the current storage time corresponding to each aggregated data and determine whether it satisfies the corresponding deletion policy.
示例性的，假设第一删除策略可以是采集数据对应的当前存储时间满足1个月则进行数据删除处理的策略。假设第二删除策略可以是聚合数据对应的当前存储时间满足3个月则进行数据删除处理的策略。假设第三删除策略可以是聚合数据对应的当前存储时间满足6个月则进行数据删除处理的策略。假设第四删除策略可以是聚合数据对应的当前存储时间满足12个月则进行数据删除处理的策略。Exemplarily, it is assumed that the first deletion policy may be a policy for deleting data if the current storage time corresponding to the collected data meets 1 month. It is assumed that the second deletion policy may be a policy for deleting data if the current storage time corresponding to the aggregated data meets 3 months. It is assumed that the third deletion policy may be a policy for deleting data if the current storage time corresponding to the aggregated data meets 6 months. It is assumed that the fourth deletion policy may be a policy for deleting data if the current storage time corresponding to the aggregated data meets 12 months.
具体的，在各所述时间粒度表中，依次获取一个当前时间粒度表，假设当前时间粒度表为第一时间粒度表，则获取第一时间粒度表中的各采集数据分别对应的当前存储时间，以及获取第一删除策略(当前存储时间满足1个月则进行数据删除处理的策略)；接着根据当前存储时间和当前删除策略，来进行各采集数据的删除操作。Specifically, in each of the time granularity tables, a current time granularity table is obtained in turn. Assuming that the current time granularity table is the first time granularity table, the current storage time corresponding to each collected data in the first time granularity table is obtained, and the first deletion policy (a policy for performing data deletion processing when the current storage time meets 1 month) is obtained; then, the deletion operation of each collected data is performed according to the current storage time and the current deletion policy.
另外的，假设当前时间粒度表为第二时间粒度表，则获取第二时间粒度表中的各聚合数据分别对应的当前存储时间，以及获取第二删除策略(当前存储时间满足3个月则进行数据删除处理的策略)；接着根据当前存储时间和当前删除策略，来进行各采集数据的删除操作。In addition, assuming that the current time granularity table is the second time granularity table, obtain the current storage time corresponding to each aggregate data in the second time granularity table, and obtain the second deletion policy (a policy for data deletion if the current storage time meets 3 months); then perform deletion operations on each collected data according to the current storage time and the current deletion policy.
可选的，所述根据所述当前存储时间和当前删除策略，来进行各采集数据或者各聚合数据的删除操作，包括：判断各所述当前存储时间是否能与当前删除策略进行匹配，若是，则删除当前采集数据或者各聚合数据；若否，则返回执行判断各所述存储时间是否能与预先设置的删除策略进行匹配的操作，直至遍历完成各采集数据或者各聚合数据。Optionally, the deletion operation of each collected data or each aggregated data is performed according to the current storage time and the current deletion policy, including: judging whether each current storage time can match the current deletion policy, and if so, deleting the current collected data or each aggregated data; if not, returning to execute the operation of judging whether each storage time can match the preset deletion policy, until all collected data or each aggregated data are traversed.
续前例的，获取第一时间粒度表中的各采集数据分别对应的当前存储时间(假设当前存储时间为25天)，以及获取第一删除策略(当前存储时间满足1个月则进行数据删除处理的策略)。由于当前存储时间不满足第一删除策略，则返回执行判断各所述存储时间是否能与预先设置的删除策略进行匹配的操作，直至遍历完成各采集数据。另外的，假设当前存储时间为35天，由于当前存储时间满足第一删除策略，则删除当前采集数据。Continuing with the previous example, the current storage time corresponding to each collected data in the first time granularity table is obtained (assuming that the current storage time is 25 days), and the first deletion policy (a policy for data deletion if the current storage time meets 1 month). Since the current storage time does not meet the first deletion policy, the operation of determining whether each storage time can match the preset deletion policy is returned until all collected data are traversed. In addition, assuming that the current storage time is 35 days, since the current storage time meets the first deletion policy, the current collected data is deleted.
另外的，获取第二时间粒度表中的各聚合数据分别对应的当前存储时间(假设当前存储时间为50天)，以及获取第二删除策略(当前存储时间满足3个月则进行数据删除处理的策略)。由于当前存储时间不满足第二删除策略，则返回执行判断各所述存储时间是否能与预先设置的删除策略进行匹配的操作，直至遍历完成各聚合数据。另外的，假设当前存储时间为95天，由于当前存储时间满足第二删除策略，则删除当前聚合数据。In addition, the current storage time corresponding to each aggregate data in the second time granularity table is obtained (assuming that the current storage time is 50 days), and the second deletion policy (a policy for data deletion if the current storage time meets 3 months). Since the current storage time does not meet the second deletion policy, the operation of determining whether each storage time can match the preset deletion policy is returned until all aggregate data are traversed. In addition, assuming that the current storage time is 95 days, since the current storage time meets the second deletion policy, the current aggregate data is deleted.
可选的，在所述根据所述当前存储时间和当前删除策略，来进行各采集数据或者各聚合数据的删除操作之后，还包括：断当前时间粒度表是否为第一时间粒度表，若是，则指示所述时序数据库将与所述当前时间粒度表对应的各采集数据发送于冷数据库中；否否，则返回执行所述依次获取一个当前时间粒度表的操作，直至遍历完成各所述时间粒度表。Optionally, after performing the deletion operation of each collected data or each aggregated data according to the current storage time and the current deletion policy, it also includes: determining whether the current time granularity table is the first time granularity table, and if so, instructing the time series database to send each collected data corresponding to the current time granularity table to the cold database; if not, returning to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.
在本实施例中，对于第一时间粒度表中的采集数据如果满足第一删除策略之后，不仅需要将采集数据从时序数据库进行删除处理，还需要将各采集数据发送于冷数据库。由于冷数据库为不经常使用的数据库，当需要对历史采集数据进行追溯的情况下，可以从冷数据库中来获取采集数据，从而能更好地进行数据的存储，提高数据存储的灵活性。In this embodiment, if the collected data in the first time granularity table meets the first deletion policy, it is not only necessary to delete the collected data from the time series database, but also to send each collected data to the cold database. Since the cold database is a database that is not frequently used, when it is necessary to trace the historical collected data, the collected data can be obtained from the cold database, so that data can be better stored and the flexibility of data storage can be improved.
可选的，在所述实时获取数据采集器中的多条采集数据之前，还包括：在前端查询页面进行埋点处理，获取各历史采集数据；其中，所述历史采集数据包括：历史查询粒度和历史时间范围；对历史查询粒度和历史时间范围进行分析聚类处理，来构建不同的时间粒度表，并对各时间粒度表分别配置相应的删除策略和联合聚合规则。Optionally, before the real-time acquisition of multiple collected data in the data collector, it also includes: performing point-of-sale processing on the front-end query page to obtain various historical collected data; wherein the historical collected data includes: historical query granularity and historical time range; analyzing and clustering the historical query granularity and historical time range to construct different time granularity tables, and configuring corresponding deletion strategies and joint aggregation rules for each time granularity table.
其中，历史查询粒度可以是用来描述历史时刻采集的数据的查询粒度的大小。历史时间范围可以是用来描述历史时刻采集的数据的时间范围的大小。The historical query granularity may be the size of the query granularity used to describe the data collected at the historical moment. The historical time range may be the size of the time range used to describe the data collected at the historical moment.
在本实施例中，可以通过在前端查询页面进行埋点处理，来获取历史时刻的采集数据，采集数据可以包括历史查询粒度和历史时间范围。通过对不同历史采集数据分别对应的历史查询粒度和历史时间范围来进行分析聚类处理，可以根据聚类结果，来得到相应的各个时间粒度表，以及根据各个时间粒度表来分别确定相应的删除策略和联合聚合规则。In this embodiment, the collected data of historical moments can be obtained by performing point embedding processing on the front-end query page, and the collected data may include historical query granularity and historical time range. By performing analysis and clustering processing on the historical query granularity and historical time range corresponding to different historical collected data, the corresponding time granularity tables can be obtained according to the clustering results, and the corresponding deletion strategies and joint aggregation rules can be determined according to the various time granularity tables.
这样设置的好处在于，可以通过对历史采集数据的分析和聚类，来确定不同的时间粒度表、删除策略和联合聚合规则，这样可以更加适合当前页面的数据存储优化，提高数据存储的效率和准确率。The advantage of this setting is that different time granularity tables, deletion strategies, and joint aggregation rules can be determined through analysis and clustering of historically collected data. This can be more suitable for data storage optimization of the current page and improve the efficiency and accuracy of data storage.
可选的，在所述将各数据聚合结果分别存储于预先构建的各时间粒度表中之后，还包括：实时获取与前端查询页面对应的当前前端查询请求；对所述当前前端查询请求进行解析，得到当前查询粒度和当前时间范围；根据所述当前查询粒度和当前时间范围，在各所述时间粒度表中进行查询，确定当前前端查询请求结果，并将所述当前前端查询请求结果反馈给所述前端查询页面。Optionally, after storing each data aggregation result in each pre-constructed time granularity table respectively, it also includes: obtaining the current front-end query request corresponding to the front-end query page in real time; parsing the current front-end query request to obtain the current query granularity and the current time range; querying in each of the time granularity tables according to the current query granularity and the current time range to determine the current front-end query request result, and feeding back the current front-end query request result to the front-end query page.
在本实施例中，在数据存储优化完成之后，还需要对存储的数据进行查询处理。其中，当前前端查询请求可以是对数据进行查询的请求。通过对当前前端查询请求进行解析之后，可以得到相应的当前查询粒度和当前时间范围。In this embodiment, after the data storage optimization is completed, the stored data needs to be queried. The current front-end query request may be a request to query the data. By parsing the current front-end query request, the corresponding current query granularity and current time range can be obtained.
进一步的，可以在时序数据库中的各个时间粒度表中来匹配相应的当前查询粒度和当前时间范围，来得到当前前端查询请求结果，并将当前前端查询请求结果反馈给前端查询页面，向用户进行页面展示，这样提高了数据查询的准确率和速度，提高用户数据查询的体验感。Furthermore, the corresponding current query granularity and current time range can be matched in each time granularity table in the time series database to obtain the current front-end query request result, and the current front-end query request result can be fed back to the front-end query page for page display to the user, thereby improving the accuracy and speed of data query and improving the user's data query experience.
本发明实施例的技术方案，通过实时获取数据采集器中的多条采集数据，并指示预设的时序数据库将各所述采集数据进行存储；指示预设的聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；指示时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；指示时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。解决了对采集到的数据直接存储原始数据而导致的数据量大和查询不方便的问题，减少了数据存储成本，提高了快速聚合数据和加快数据查询的效率。The technical solution of the embodiment of the present invention obtains multiple collected data in the data collector in real time, and instructs the preset time series database to store each of the collected data; instructs the preset aggregation calculation engine to obtain each of the collected data from the time series database, and performs the collection data aggregation operation according to the preset joint aggregation rules to obtain data aggregation results of different granularities; instructs the time series database to obtain each data aggregation result, and stores each data aggregation result in each pre-built time granularity table; instructs the time series database to perform data compression processing on each time granularity table to obtain data aggregation storage results, so as to complete the storage optimization operation of the collected data according to the data aggregation storage results. The problem of large data volume and inconvenient query caused by directly storing the original data of the collected data is solved, the data storage cost is reduced, and the efficiency of fast data aggregation and accelerated data query is improved.
实施例二Embodiment 2
图2为本发明实施例二提供的一种数据存储优化装置的结构示意图。本实施例所提供的一种数据存储优化装置可以通过软件和/或硬件来实现，可配置于终端设备或者服务器中来实现本发明实施例中的一种数据存储优化方法。如图2所示，该装置包括：采集数据获取模块210、数据聚合结果确定模块220、数据聚合结果存储模块230和数据聚合存储结果确定模块240。FIG2 is a schematic diagram of the structure of a data storage optimization device provided in Embodiment 2 of the present invention. The data storage optimization device provided in this embodiment can be implemented by software and/or hardware, and can be configured in a terminal device or a server to implement a data storage optimization method in an embodiment of the present invention. As shown in FIG2 , the device includes: a data acquisition module 210, a data aggregation result determination module 220, a data aggregation result storage module 230, and a data aggregation storage result determination module 240.
其中，采集数据获取模块210，用于实时获取数据采集器中的多条采集数据，并指示时序数据库将各所述采集数据进行存储；The collected data acquisition module 210 is used to acquire multiple collected data in the data collector in real time and instruct the time series database to store each of the collected data;
数据聚合结果确定模块220，用于指示聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；The data aggregation result determination module 220 is used to instruct the aggregation calculation engine to obtain each of the collected data from the time series database, and perform an aggregation operation on the collected data according to a preset joint aggregation rule to obtain data aggregation results of different granularities;
数据聚合结果存储模块230，用于指示所述时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；The data aggregation result storage module 230 is used to instruct the time series database to obtain each data aggregation result and store each data aggregation result in each pre-constructed time granularity table;
数据聚合存储结果确定模块240，用于指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。The data aggregation storage result determination module 240 is used to instruct the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.
本发明实施例的技术方案，通过实时获取数据采集器中的多条采集数据，并指示预设的时序数据库将各所述采集数据进行存储；指示预设的聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；指示时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；指示时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。解决了对采集到的数据直接存储原始数据而导致的数据量大和查询不方便的问题，减少了数据存储成本，提高了快速聚合数据和加快数据查询的效率。The technical solution of the embodiment of the present invention obtains multiple collected data in the data collector in real time, and instructs the preset time series database to store each of the collected data; instructs the preset aggregation calculation engine to obtain each of the collected data from the time series database, and performs the collection data aggregation operation according to the preset joint aggregation rules to obtain data aggregation results of different granularities; instructs the time series database to obtain each data aggregation result, and stores each data aggregation result in each pre-built time granularity table; instructs the time series database to perform data compression processing on each time granularity table to obtain data aggregation storage results, so as to complete the storage optimization operation of the collected data according to the data aggregation storage results. The problem of large data volume and inconvenient query caused by directly storing the original data of the collected data is solved, the data storage cost is reduced, and the efficiency of fast data aggregation and accelerated data query is improved.
在上述各实施例的基础上，所述时间粒度表包括第一时间粒度表、第二时间粒度表、第三时间粒度表和第四时间粒度表。Based on the above embodiments, the time granularity table includes a first time granularity table, a second time granularity table, a third time granularity table and a fourth time granularity table.
在上述各实施例的基础上，还包括，采集数据删除模块，包括：当前时间粒度表获取单元，可以具体用于：在所述指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果之后，在各所述时间粒度表中，依次获取一个当前时间粒度表；删除策略获取单元，可以具体用于：获取所述当前时间粒度表中的各采集数据或者各聚合数据分别对应的当前存储时间，以及获取与所述当前时间粒度表相匹配的预先设置的删除策略；其中，第一时间粒度表对应第一删除策略；第二时间粒度表对应第二删除策略；第三时间粒度表对应第三删除策略；第四时间粒度表对应第四删除策略；第一时间粒度表的时间粒度小于第二时间粒度表；第二时间粒度表的时间粒度小于第三时间粒度表；第三时间粒度表的时间粒度小于第四时间粒度表；采集数据删除单元，可以具体用于：根据所述当前存储时间和当前删除策略，来进行各采集数据或者各聚合数据的删除操作；返回执行单元，可以具体用于：返回执行所述依次获取一个当前时间粒度表的操作，直至遍历完成各所述时间粒度表。On the basis of the above embodiments, it also includes a collection data deletion module, including: a current time granularity table acquisition unit, which can be specifically used to: after the time series database is instructed to perform data compression processing on each time granularity table and obtain the data aggregation storage result, a current time granularity table is sequentially acquired in each time granularity table; a deletion policy acquisition unit, which can be specifically used to: acquire the current storage time corresponding to each collection data or each aggregation data in the current time granularity table, and acquire a preset deletion policy matching the current time granularity table; wherein the first time granularity table corresponds to the first deletion policy; the second time granularity table corresponds to the first deletion policy; The granularity table corresponds to the second deletion strategy; the third time granularity table corresponds to the third deletion strategy; the fourth time granularity table corresponds to the fourth deletion strategy; the time granularity of the first time granularity table is smaller than that of the second time granularity table; the time granularity of the second time granularity table is smaller than that of the third time granularity table; the time granularity of the third time granularity table is smaller than that of the fourth time granularity table; the collected data deletion unit can be specifically used to: perform deletion operations on each collected data or each aggregated data according to the current storage time and the current deletion strategy; the return execution unit can be specifically used to: return to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.
在上述各实施例的基础上，所述采集数据删除单元，可以具体用于：判断各所述当前存储时间是否能与当前删除策略进行匹配，若是，则删除当前采集数据或者各聚合数据；若否，则返回执行判断各所述存储时间是否能与预先设置的删除策略进行匹配的操作，直至遍历完成各采集数据或者各聚合数据。Based on the above embodiments, the collected data deletion unit can be specifically used to: determine whether each current storage time can match the current deletion policy, and if so, delete the current collected data or each aggregated data; if not, return to execute the operation of determining whether each storage time can match the preset deletion policy until all collected data or each aggregated data are traversed.
在上述各实施例的基础上，还包括，第一时间粒度表判断单元，可以具体用于：在所述根据所述当前存储时间和当前删除策略，来进行各采集数据的删除操作之后，判断当前时间粒度表是否为第一时间粒度表，若是，则指示所述时序数据库将与所述当前时间粒度表对应的各采集数据发送于冷数据库中；否否，则返回执行所述依次获取一个当前时间粒度表的操作，直至遍历完成各所述时间粒度表。On the basis of the above embodiments, it also includes a first time granularity table judgment unit, which can be specifically used to: after the deletion operation of each collected data is performed according to the current storage time and the current deletion policy, determine whether the current time granularity table is the first time granularity table. If so, instruct the time series database to send each collected data corresponding to the current time granularity table to the cold database; if not, return to execute the operation of obtaining a current time granularity table in turn until all the time granularity tables are traversed.
在上述各实施例的基础上，所述联合聚合规则包括第一联合聚合规则、第二联合聚合规则和第三联合聚合规则。On the basis of the above embodiments, the joint aggregation rule includes a first joint aggregation rule, a second joint aggregation rule and a third joint aggregation rule.
在上述各实施例的基础上，所述数据聚合结果确定模块220，可以具体用于：通过所述第一联合聚合规则来进行采集数据聚合操作，得到第一数据聚合结果；通过所述第二联合聚合规则来进行采集数据聚合操作，得到第二数据聚合结果；通过所述第三联合聚合规则来进行采集数据聚合操作，得到第三数据聚合结果。Based on the above embodiments, the data aggregation result determination module 220 can be specifically used to: perform the data aggregation operation through the first joint aggregation rule to obtain the first data aggregation result; perform the data aggregation operation through the second joint aggregation rule to obtain the second data aggregation result; perform the data aggregation operation through the third joint aggregation rule to obtain the third data aggregation result.
在上述各实施例的基础上，所述数据聚合结果存储模块230，可以具体用于：将所述第一数据聚合结果存储于所述第二时间粒度表；将所述第二数据聚合结果存储于所述第三时间粒度表；将所述第三数据聚合结果存储于所述第四时间粒度表。Based on the above embodiments, the data aggregation result storage module 230 can be specifically used to: store the first data aggregation result in the second time granularity table; store the second data aggregation result in the third time granularity table; store the third data aggregation result in the fourth time granularity table.
在上述各实施例的基础上，还包括，时间粒度表构建模块，可以具体用于：在所述实时获取数据采集器中的多条采集数据之前，在前端查询页面进行埋点处理，获取各历史采集数据；其中，所述历史采集数据包括：历史查询粒度和历史时间范围；对历史查询粒度和历史时间范围进行分析聚类处理，来构建不同的时间粒度表，并对各时间粒度表分别配置相应的删除策略和联合聚合规则。On the basis of the above embodiments, it also includes a time granularity table construction module, which can be specifically used for: before the real-time acquisition of multiple collected data in the data collector, performing point-of-sale processing on the front-end query page to obtain various historical collected data; wherein the historical collected data includes: historical query granularity and historical time range; analyzing and clustering the historical query granularity and the historical time range to construct different time granularity tables, and configuring corresponding deletion strategies and joint aggregation rules for each time granularity table.
在上述各实施例的基础上，还包括，当前前端查询请求结果确定模块，可以具体用于：在所述将各数据聚合结果分别存储于预先构建的各时间粒度表中之后，实时获取与前端查询页面对应的当前前端查询请求；对所述当前前端查询请求进行解析，得到当前查询粒度和当前时间范围；根据所述当前查询粒度和当前时间范围，在各所述时间粒度表中进行查询，确定当前前端查询请求结果，并将所述当前前端查询请求结果反馈给所述前端查询页面。On the basis of the above embodiments, it also includes a current front-end query request result determination module, which can be specifically used to: after storing each data aggregation result in each pre-built time granularity table respectively, obtain the current front-end query request corresponding to the front-end query page in real time; parse the current front-end query request to obtain the current query granularity and the current time range; query in each of the time granularity tables according to the current query granularity and the current time range to determine the current front-end query request result, and feed back the current front-end query request result to the front-end query page.
本发明实施例所提供的数据存储优化装置可执行本发明任意实施例所提供的数据存储优化方法，具备执行方法相应的功能模块和有益效果。The data storage optimization device provided in the embodiment of the present invention can execute the data storage optimization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
实施例三Embodiment 3
图3示出了可以用来实施本发明的实施例三的电子设备10的结构示意图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备(如头盔、眼镜、手表等)和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本发明的实现。FIG3 shows a schematic diagram of the structure of an electronic device 10 that can be used to implement the third embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and/or required herein.
如图3所示，电子设备10包括至少一个处理器11，以及与至少一个处理器11通信连接的存储器，如只读存储器(ROM)12、随机访问存储器(RAM)13等，其中，存储器存储有可被至少一个处理器执行的计算机程序，处理器11可以根据存储在只读存储器(ROM)12中的计算机程序或者从存储单元18加载到随机访问存储器(RAM)13中的计算机程序，来执行各种适当的动作和处理。在RAM 13中，还可存储电子设备10操作所需的各种程序和数据。处理器11、ROM 12以及RAM 13通过总线14彼此相连。输入/输出(I/O)接口15也连接至总线14。As shown in FIG3 , the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11 in communication, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to the bus 14.
电子设备10中的多个部件连接至I/O接口15，包括：输入单元16，例如键盘、鼠标等；输出单元17，例如各种类型的显示器、扬声器等；存储单元18，例如磁盘、光盘等；以及通信单元19，例如网卡、调制解调器、无线通信收发机等。通信单元19允许电子设备10通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据。A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.
处理器11可以是各种具有处理和计算能力的通用和/或专用处理组件。处理器11的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的处理器、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。处理器11执行上文所描述的各个方法和处理，例如数据存储优化方法。The processor 11 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data storage optimization method.
在一些实施例中，数据存储优化方法可被实现为计算机程序，其被有形地包含于计算机可读存储介质，例如存储单元18。在一些实施例中，计算机程序的部分或者全部可以经由ROM 12和/或通信单元19而被载入和/或安装到电子设备10上。当计算机程序加载到RAM 13并由处理器11执行时，可以执行上文描述的数据存储优化方法的一个或多个步骤。备选地，在其他实施例中，处理器11可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行数据存储优化方法。In some embodiments, the data storage optimization method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data storage optimization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data storage optimization method in any other appropriate manner (e.g., by means of firmware).
该方法包括：实时获取数据采集器中的多条采集数据，并指示时序数据库将各所述采集数据进行存储；指示聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；指示所述时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。The method includes: acquiring multiple collected data in a data collector in real time, and instructing a time series database to store each of the collected data; instructing an aggregation computing engine to acquire each of the collected data from the time series database, and perform aggregating operations on the collected data according to a pre-set joint aggregation rule to obtain data aggregation results of different granularities; instructing the time series database to acquire each data aggregation result, and store each data aggregation result in a pre-constructed time granularity table respectively; instructing the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.
本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、芯片上系统的系统(SOC)、负载可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
用于实施本发明的方法的计算机程序可以采用一个或多个编程语言的任何组合来编写。这些计算机程序可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器，使得计算机程序当由处理器执行时使流程图和/或框图中所规定的功能/操作被实施。计算机程序可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions/operations specified in the flow chart and/or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
在本发明的上下文中，计算机可读存储介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的计算机程序。计算机可读存储介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。备选地，计算机可读存储介质可以是机器可读信号介质。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
为了提供与用户的交互，可以在电子设备上实施此处描述的系统和技术，该电子设备具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给电子设备。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)、区块链网络和互联网。The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
计算系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，又称为云计算服务器或云主机，是云计算服务体系中的一项主机产品，以解决了传统物理主机与VPS服务中，存在的管理难度大，业务扩展性弱的缺陷。A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本发明中记载的各步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本发明的技术方案所期望的结果，本文在此不进行限制。It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
上述具体实施方式，并不构成对本发明保护范围的限制。本领域技术人员应该明白的是，根据设计要求和其他因素，可以进行各种修改、组合、子组合和替代。任何在本发明的精神和原则之内所作的修改、等同替换和改进等，均应包含在本发明保护范围之内。The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
实施例四Embodiment 4
本发明实施例四还提供一种包含计算机可读存储介质，所述计算机可读指令在由计算机处理器执行时用于执行一种数据存储优化方法，该方法包括：实时获取数据采集器中的多条采集数据，并指示时序数据库将各所述采集数据进行存储；指示聚合计算引擎从所述时序数据库获取各所述采集数据，并根据预先设置的联合聚合规则来进行采集数据聚合操作，得到不同粒度的数据聚合结果；指示所述时序数据库获取各数据聚合结果，并将各数据聚合结果分别存储于预先构建的各时间粒度表中；指示所述时序数据库对各时间粒度表进行数据压缩处理，得到数据聚合存储结果，以根据数据聚合存储结果，来完成采集数据的存储优化的操作。Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a data storage optimization method when executed by a computer processor, the method comprising: acquiring multiple collected data in a data collector in real time, and instructing a time series database to store each of the collected data; instructing an aggregation computing engine to acquire each of the collected data from the time series database, and perform a collection data aggregation operation according to a preset joint aggregation rule to obtain data aggregation results of different granularities; instructing the time series database to acquire each data aggregation result, and store each data aggregation result in a pre-constructed time granularity table respectively; instructing the time series database to perform data compression processing on each time granularity table to obtain a data aggregation storage result, so as to complete the storage optimization operation of the collected data according to the data aggregation storage result.
当然，本发明实施例所提供的一种包含计算机可读存储介质，其计算机可执行指令不限于如上所述的方法操作，还可以执行本发明任意实施例所提供的数据存储优化方法中的相关操作。Of course, the computer-readable storage medium provided in the embodiment of the present invention has computer executable instructions that are not limited to the method operations described above, and can also execute related operations in the data storage optimization method provided in any embodiment of the present invention.
通过以上关于实施方式的描述，所属领域的技术人员可以清楚地了解到，本发明可借助软件及必需的通用硬件来实现，当然也可以通过硬件实现，但很多情况下前者是更佳的实施方式。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(RandomAccess Memory，RAM)、闪存(FLASH)、硬盘或光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述的方法。Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.
值得注意的是，上述数据存储优化装置的实施例中，所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本发明的保护范围。It is worth noting that in the embodiment of the above-mentioned data storage optimization device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.
上述具体实施方式，并不构成对本发明保护范围的限制。本领域技术人员应该明白的是，根据设计要求和其他因素，可以进行各种修改、组合、子组合和替代。任何在本发明的精神和原则之内所作的修改、等同替换和改进等，均应包含在本发明保护范围之内。The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410498584.3ACN118410021B (en) | 2024-04-24 | 2024-04-24 | Data storage optimization method and device, electronic equipment and storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410498584.3ACN118410021B (en) | 2024-04-24 | 2024-04-24 | Data storage optimization method and device, electronic equipment and storage medium |
| Publication Number | Publication Date |
|---|---|
| CN118410021Atrue CN118410021A (en) | 2024-07-30 |
| CN118410021B CN118410021B (en) | 2025-03-07 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410498584.3AActiveCN118410021B (en) | 2024-04-24 | 2024-04-24 | Data storage optimization method and device, electronic equipment and storage medium |
| Country | Link |
|---|---|
| CN (1) | CN118410021B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119692718A (en)* | 2024-12-20 | 2025-03-25 | 国网浙江省电力有限公司舟山供电公司 | A power grid adaptive dispatching method combined with short-term power source and load forecasting |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108460094A (en)* | 2018-01-30 | 2018-08-28 | 上海天旦网络科技发展有限公司 | The method and system of storage statistical data |
| CN110147357A (en)* | 2019-05-07 | 2019-08-20 | 浙江科技学院 | The multi-source data polymerization methods of sampling and system under a kind of environment based on big data |
| CN110389967A (en)* | 2019-07-26 | 2019-10-29 | 深圳市腾讯计算机系统有限公司 | Data storage method, device, server and storage medium |
| US10635315B1 (en)* | 2019-03-22 | 2020-04-28 | EMC IP Holding Company LLC | Performing compression and deduplication at different granularities |
| CN112800061A (en)* | 2021-01-29 | 2021-05-14 | 北京锐安科技有限公司 | A data storage method, device, server and storage medium |
| CN115034436A (en)* | 2022-05-10 | 2022-09-09 | 华东交通大学 | Electric power short-term prediction method based on generalized hierarchy and multiple granularities |
| CN115408214A (en)* | 2022-08-09 | 2022-11-29 | 北京捷科智诚科技有限公司 | Distributed, time-sequence and high-concurrency performance test tool collection sample calculation method and system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108460094A (en)* | 2018-01-30 | 2018-08-28 | 上海天旦网络科技发展有限公司 | The method and system of storage statistical data |
| US10635315B1 (en)* | 2019-03-22 | 2020-04-28 | EMC IP Holding Company LLC | Performing compression and deduplication at different granularities |
| CN110147357A (en)* | 2019-05-07 | 2019-08-20 | 浙江科技学院 | The multi-source data polymerization methods of sampling and system under a kind of environment based on big data |
| CN110389967A (en)* | 2019-07-26 | 2019-10-29 | 深圳市腾讯计算机系统有限公司 | Data storage method, device, server and storage medium |
| CN112800061A (en)* | 2021-01-29 | 2021-05-14 | 北京锐安科技有限公司 | A data storage method, device, server and storage medium |
| CN115034436A (en)* | 2022-05-10 | 2022-09-09 | 华东交通大学 | Electric power short-term prediction method based on generalized hierarchy and multiple granularities |
| CN115408214A (en)* | 2022-08-09 | 2022-11-29 | 北京捷科智诚科技有限公司 | Distributed, time-sequence and high-concurrency performance test tool collection sample calculation method and system |
| Title |
|---|
| TIMKO, I; BÖHLEN, M AND GAMPER, J: ""Sequenced spatiotemporal aggregation for coarse query granularities"", 《VLDB JOURNAL》, 31 December 2011 (2011-12-31), pages 721 - 741* |
| 金祝光: ""基于Memcached时序数据缓存的设计与实现"", 《中国优秀硕士学位论文全文数据库 (基础科学辑)》, 15 November 2022 (2022-11-15), pages 137 - 53* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119692718A (en)* | 2024-12-20 | 2025-03-25 | 国网浙江省电力有限公司舟山供电公司 | A power grid adaptive dispatching method combined with short-term power source and load forecasting |
| Publication number | Publication date |
|---|---|
| CN118410021B (en) | 2025-03-07 |
| Publication | Publication Date | Title |
|---|---|---|
| CN113568940B (en) | Method, device, equipment and storage medium for data query | |
| CN118643120A (en) | Code retrieval method and device based on large language model | |
| CN118410021A (en) | Data storage optimization method, device, electronic device and storage medium | |
| CN118820400A (en) | Multi-channel recall retrieval enhancement generation method and device based on real-time data index | |
| CN118823008A (en) | Image processing method, device, electronic device and storage medium | |
| CN118503516A (en) | Star table data retrieval method and device, electronic equipment and storage medium | |
| CN118673437A (en) | Knowledge graph-based abnormal examination method and device for examination rules and electronic equipment | |
| CN118797108A (en) | Data optimization method and device, electronic device and storage medium | |
| CN115860121B (en) | Text reasoning method, device, equipment and storage medium | |
| CN113051505B (en) | Document display method, device and electronic device | |
| CN114416414B (en) | Fault information locating method, device, equipment and storage medium | |
| CN113326416B (en) | Method for searching data, method and device for sending search data to client | |
| CN116594709A (en) | Method, device and computer program product for acquiring data | |
| CN112989224B (en) | Retrieval method, retrieval device, electronic equipment and storage medium | |
| CN112948461B (en) | Method, apparatus, storage medium and program product for calendar data processing | |
| CN112559886B (en) | Method, device, equipment, storage medium and product for sorting history documents | |
| CN112214290B (en) | Log information processing method, edge node, center node and system | |
| CN117156398B (en) | Message processing method, device, electronic equipment and storage medium | |
| CN117827425A (en) | Resource elimination method, device, electronic device and readable storage medium | |
| CN118503224A (en) | Object data storage and processing method, device, electronic device and storage medium | |
| CN118656276A (en) | A log data display method, device, medium and program product | |
| CN117885760A (en) | Automatic driving capability determining method, device, equipment and storage medium | |
| CN118379390A (en) | A stock curve drawing method, device, electronic device and storage medium | |
| CN118916307A (en) | Descriptor transmission method, device, equipment and storage medium | |
| CN118734338A (en) | Method, device, electronic device and readable storage medium for processing access data of application program interface |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |