CN113961579A - Dynamic updating method and device based on ClickHouse, electronic device and storage medium - Google Patents

Abstract

The application relates to a dynamic updating method, a device, an electronic device and a storage medium based on a ClickHouse, wherein the dynamic updating method based on the ClickHouse comprises the following steps: acquiring screening information of the equipment portrait; screening the target database according to the screening information to obtain a screening result; acquiring update data of a target database; screening the updating data according to the screening information to obtain updating information; and dynamically updating the screening result according to the updating information. By the method and the device, the updated data of the target database is screened, the screening result is dynamically updated, the problem that the updating time of the screening result of the equipment portrait is slow is solved, the screening result is updated in real time according to the updated data, and the data updating efficiency is improved.

Description

Dynamic updating method and device based on ClickHouse, electronic device and storage medium

Technical Field

The present application relates to the field of big data analysis technologies, and in particular, to a dynamic update method and apparatus based on clickwouse, an electronic apparatus, and a storage medium.

Background

With the large-scale popularization and application of the internet, the explosive increase of data volume marks the coming of a big data era, useful value data needs to be analyzed by a data analyst, and big data analysis becomes a very important link. The development of mass data brings great convenience to the life of people, and under the current big background of data explosion, each industry starts to rely more and more on the support of mass data and corresponding analysis to help companies to make more intelligent and reasonable decisions on strategic layout and marketing strategies. Entrepreneurs can help the entrepreneurs to better cater to larger user groups and increase sales by analyzing purchasing trends and purchasing habits of different user groups and purchasing feedback data.

An enterprise has tens of thousands or hundreds of thousands of devices around the world, each device works all the time, and you need to acquire running state information of the devices (such as recently accessed client addresses, latitude and longitude of the devices, whether alarm information exists) from the working machines for statistics, so that each device generates tens of thousands of pieces of data, and the data needs to be appropriately stored for tracing and analysis, and the storage bottleneck of the traditional type relational database including Mysql, Oracle, SqlServer, DB2 and the like can only reach millions of pieces and face the problem of query timeout. Large data-oriented databases such as elastic search, Hive and Vertica have respective characteristics and occupy a place in the large data field. However, when the data magnitude reaches billions, billions or even upwards, the efficiency between the storage space occupation and various aggregation queries starts to be improved, and when equipment portrait analysis is carried out, aggregation statistics of various intersections, union sets and difference sets is often required, wherein the memory occupation of the data and the response speed of an aggregation function are not satisfactory.

Aiming at the problem that the updating time of the screening result of the equipment portrait is slow in the related technology, no effective solution is provided at present.

Disclosure of Invention

In the embodiment, a clickwouse-based dynamic updating method, a clickwouse-based dynamic updating device, an electronic device and a storage medium are provided to solve the problem that the updating time of the screening result of the device portrait is slow in the related art.

In a first aspect, in this embodiment, a method for dynamic update based on clickwouse is provided, where the method includes:

acquiring screening information of the equipment portrait;

screening the target database according to the screening information to obtain a screening result;

acquiring update data of a target database;

screening the updating data according to the screening information to obtain updating information;

and dynamically updating the screening result according to the updating information.

In some embodiments, the obtaining the filtering information of the device representation includes:

obtaining a plurality of screening fields of a target database, wherein the screening fields include: at least one of service time of the equipment, last overhaul time, the number of faults, the number of error log entries, equipment ID, log type, error log, address of a recently accessed client, longitude and latitude of the equipment and alarm information;

and selecting the corresponding screening field to generate the screening information.

In some embodiments, the screening the target database according to the screening information to obtain a screening result includes:

obtaining a plurality of screening information of the equipment image;

screening a target database according to the screening information to obtain a plurality of corresponding screening results;

and aggregating the results of the plurality of screening tasks through an aggregation function to obtain a final screening result.

In some embodiments, the aggregating the results of the plurality of screening tasks through an aggregation function to obtain a final screening result includes;

performing aggregation statistics according to screening fields contained in the plurality of screening information to obtain a screening field set;

and screening a plurality of screening task results according to the screening field set to obtain a final screening result.

In some of these embodiments, the obtaining update data of the target database includes:

monitoring the updating data according to the screening information;

and loading the updated data containing the screening information into a cache.

In some embodiments, the loading the update data containing the filtering information into the cache comprises:

detecting whether the existing time of the updated data in the cache exceeds a preset time threshold;

and if the existing time of the updated data in the cache exceeds a preset time threshold, emptying the updated data in the cache.

In some embodiments, the loading the update data containing the filtering information into the cache comprises:

detecting whether the data volume of the updating information in the cache exceeds a preset data volume threshold value or not;

and if the data quantity of the updated data in the cache exceeds a preset data quantity threshold value, emptying the updated data in the cache.

In a second aspect, in this embodiment, a clickwouse-based dynamic update apparatus is provided, including:

the information acquisition module is used for acquiring screening information of the equipment portrait;

the database screening module is used for screening the target database according to the screening information to obtain a screening result;

the data updating module is used for acquiring the updating data of the target database;

the updating data screening module is used for screening the updating data according to the screening information to obtain updating information;

and the dynamic updating module is used for dynamically updating the screening result according to the updating information.

In a third aspect, in this embodiment, there is provided an electronic apparatus, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the clickwouse-based dynamic update method according to the first aspect.

In a fourth aspect, in the present embodiment, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the clickwouse-based dynamic update method described in the first aspect above.

Compared with the related art, the dynamic updating method, the device, the electronic device and the storage medium based on the clickwouse provided by the embodiment filter the updated data of the target database and dynamically update the filtering result, so that the problem of slow updating time of the filtering result of the equipment portrait is solved, the filtering result is updated in real time according to the updated data, and the data updating efficiency is improved.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is a block diagram of a hardware structure of a terminal of the dynamic update method based on clickwouse according to the present embodiment;

FIG. 2 is a flowchart of a ClickHouse-based dynamic update method according to the embodiment;

FIG. 3 is a flow chart of another ClickHouse-based dynamic update method according to the embodiment;

fig. 4 is a block diagram of the dynamic update apparatus based on clickwouse according to the present embodiment.

Detailed Description

For a clearer understanding of the objects, aspects and advantages of the present application, reference is made to the following description and accompanying drawings.

Unless defined otherwise, technical or scientific terms used herein shall have the same general meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The use of the terms "a" and "an" and "the" and similar referents in the context of this application do not denote a limitation of quantity, either in the singular or the plural. The terms "comprises," "comprising," "has," "having," and any variations thereof, as referred to in this application, are intended to cover non-exclusive inclusions; for example, a process, method, and system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or modules, but may include other steps or modules (elements) not listed or inherent to such process, method, article, or apparatus. Reference throughout this application to "connected," "coupled," and the like is not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference to "a plurality" in this application means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. In general, the character "/" indicates a relationship in which the objects associated before and after are an "or". The terms "first," "second," "third," and the like in this application are used for distinguishing between similar items and not necessarily for describing a particular sequential or chronological order.

The method embodiments provided in the present embodiment may be executed in a terminal, a computer, or a similar computing device. For example, the method is executed on a terminal, and fig. 1 is a block diagram of a hardware structure of the terminal according to the dynamic update method based on clickwouse in this embodiment. As shown in fig. 1, the terminal may include one or more processors 102 (only one shown in fig. 1) and amemory 104 for storing data, wherein theprocessor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA. The terminal may also include atransmission device 106 for communication functions and an input-output device 108. It will be understood by those of ordinary skill in the art that the structure shown in fig. 1 is merely an illustration and is not intended to limit the structure of the terminal described above. For example, the terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.

Thememory 104 can be used for storing computer programs, for example, software programs and modules of application software, such as a computer program corresponding to the clickwouse-based dynamic update method in the embodiment, and theprocessor 102 executes various functional applications and data processing by running the computer programs stored in thememory 104, so as to implement the above-mentioned method. Thememory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, thememory 104 may further include memory located remotely from theprocessor 102, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Thetransmission device 106 is used to receive or transmit data via a network. The network described above includes a wireless network provided by a communication provider of the terminal. In one example, thetransmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, thetransmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.

In this embodiment, a dynamic update method based on clickwouse is provided, and fig. 2 is a flowchart of the dynamic update method based on clickwouse of this embodiment, as shown in fig. 2, the flowchart includes the following steps:

in step S201, the screening information of the device portrait is obtained.

Specifically, the plug-in is uploaded to a server using a ClickHouse database, the plug-in is opened, and the screening information of the equipment portrait selected by the user is obtained.

The ClickHouse is an OLAP type database which is opened by Yadex, a Russian Max search engine company in 2016, supports sql operation and is low in learning cost. Its ultra-high read and write performance and high data compression rate make it reputable in the industry. At the same data level, the ClickHouse consumes 1/4 of ElasticSearch and 1/2 of Hive as the storage space under the default configuration, and the compression rate can reach 6% -17%. The writing efficiency can reach 4-20 ten thousand per second. Aggregated queries performed 2-25 times better than elastic search, 279 times more than Hive, and 5 times more than Vertica in terms of billions of data. Moreover, the ClickHouse is provided with a BitMap function and naturally supports bitmaps, so that the support on device portrayal is very large, and the space occupation is reduced by nearly 64 times for storage based on ID; in operation, the ClickHouse also provides a calculation function of intersection, union and difference sets of the bitmaps, and the self-realization process and maintenance are omitted.

The screening information of the obtained equipment portrait comprises: and acquiring a plurality of screening fields of the target database, selecting the corresponding screening fields, and generating the screening information.

Wherein the screening field comprises: at least one of service time of the equipment, last overhaul time, failure times, error log number, equipment ID, log type, error log, recently accessed client address, longitude and latitude of the equipment and alarm information.

Specifically, screening fields of the target database are obtained, and a user selects one or more screening fields to generate screening information. More specifically, the user selects the filtering field a plurality of times, generates a plurality of filtering conditions, and generates filtering information based on the plurality of filtering conditions.

In one embodiment, the user can form screening information of the dangerous equipment portrait according to the indexes of the service time, the last time of maintenance, the number of faults and the number of error reporting logs of the equipment.

And S202, screening the target database according to the screening information to obtain a screening result.

Specifically, according to the screening fields contained in the screening information, the information containing the screening fields in the target database is screened out to obtain screening results, and a classification table is created according to the screening results.

The screening of the target database according to the screening information to obtain a screening result comprises the following steps: obtaining a plurality of screening information of the equipment image, and screening a target database according to the plurality of screening information to obtain a plurality of corresponding screening results; and aggregating the results of the plurality of screening tasks through an aggregation function to obtain a final screening result.

Specifically, the server establishes a plurality of screening tasks, each screening task corresponds to one piece of screening information, the plurality of screening tasks screen the target database in parallel to obtain a plurality of corresponding screening results, and the plurality of screening task results are aggregated through an aggregation function to obtain a final screening result.

Aggregating a plurality of screening task results through an aggregation function to obtain a final screening result; performing aggregation statistics according to screening fields contained in the plurality of screening information to obtain a screening field set; and screening a plurality of screening task results according to the screening field set to obtain a final screening result.

Wherein, the aggregation system is any one of intersection, union and difference.

Specifically, the screening fields contained in different screening information are subjected to set/union set/difference set statistics to obtain a screening field set. And screening data containing the screening field set in the screening task results according to the screening field set to obtain a final screening result.

Screening data containing screening field set in a plurality of screening task results

In step S203, update data of the target database is acquired.

Specifically, the server collects update data of the target database.

The acquiring of the update data of the target database comprises: monitoring the updating data according to the screening information; and loading the updated data containing the screening information into a cache.

Specifically, the server judges whether the updated data contains the screening field in the screening information, and if the updated data contains the screening field in the screening information, the updated data containing the screening information is loaded into the cache. And if the updated data does not contain the screening fields in the screening information, not loading the updated data into the cache. In one embodiment, if the updated data contains the screening field in the screening information, the screening field with the unique identifier in the updated data is extracted, and the screening field is loaded into the cache.

The loading the updated data containing the screening information into the cache comprises: detecting whether the existing time of the updated data in the cache exceeds a preset time threshold; and if the existing time of the updated data in the cache exceeds a preset time threshold, emptying the updated data in the cache.

Specifically, the server adopts fixed-length time window dynamic updating, and whether the existing time of the updated data in the cache exceeds a preset time threshold value is detected; and if the existing time of the updated data in the cache exceeds a preset time threshold, emptying the updated data in the cache. And if the existing time of the updated data in the cache does not exceed the preset time threshold, storing the updated data in the cache.

The loading the updated data containing the screening information into the cache comprises: detecting whether the data volume of the updating information in the cache exceeds a preset data volume threshold value or not; and if the data quantity of the updated data in the cache exceeds a preset data quantity threshold value, emptying the updated data in the cache.

Specifically, the server dynamically updates by adopting an increasing time window, and detects whether the data volume of the updating information in the cache exceeds a preset data volume threshold value or not; and if the data quantity of the updated data in the cache exceeds a preset data quantity threshold value, emptying the updated data in the cache. And if the data quantity of the updated data in the cache does not exceed the preset data quantity threshold, storing the updated data in the cache.

And step S204, screening the updating data according to the screening information to obtain updating information.

Specifically, the server screens the updated data in the cache according to the screening information to obtain the updated information. The server screens the incremental update data without traversing the target database again, so that the screening time is saved, the screening efficiency is improved, and the burden of the system is reduced.

And step S205, dynamically updating the screening result according to the updating information.

Specifically, the server dynamically updates the screening result in an incremental manner according to the update information, and does not need to re-screen all data in the target database.

Through the steps, the updated data of the target database is screened, the screening result is dynamically updated, the problem that the time for updating the screening result of the equipment portrait is slow is solved, the screening result is updated in real time according to the updated data, and the data updating efficiency is improved.

The embodiment also provides a dynamic updating method based on the ClickHouse. Fig. 3 is a flowchart of another dynamic update method based on clickwouse according to this embodiment, and as shown in fig. 3, the flowchart includes the following steps:

step S301, uploading the plug-in.

Specifically, the plug-in is uploaded to a server using a clickwouse database.

Step S302, starting ClickHouse, if started, ignoring.

Specifically, the clickwouse database is started, and if clickwouse data is already started, the operation is ignored.

Step S303, create a classification table.

Specifically, a user creates a classification table according to the screening fields of the target database, for example, the service time of the equipment, the last time of maintenance, the number of failures once, and the number of error reporting logs can form a dangerous equipment portrait, and then the screening fields are screened from the target database and added into the classification table to generate screening information.

And step S304, informing the screening information needing to be monitored by using a command line in the plug-in tool, and screening the target data according to the screening information to obtain a screening result.

Specifically, when the plug-in is started, the ClickHouseJDBC connection address needs to be sent into the plug-in as a command parameter.

Step S305, run aggregation for the first time.

Specifically, an aggregation function is executed in the ClickHouse database, and device IDs conforming to a plurality of screening information are aggregated to generate a screening result. For example, when a current user wants to acquire a device group which is likely to be hijacked and attacked in the future, the aggregated information may be a classification table of screening fields containing device IDs, service time of the devices, last overhaul time, the number of times of failures and the number of error log entries, and a classification table of screening fields containing device IDs, log types and error log entries, and then send an instruction to the plugin.

In step S306, the plug-in command line executes the start task.

Specifically, the plug-in receives the instruction, monitors the screened fields screened by the target database, and the user needs to inform the plug-in through a command line of the specific aggregated screening conditions of the screened fields.

In step S307, the data is updated.

Step S308, the window is lengthened.

And step S309, caching.

Specifically, the server dynamically updates by adopting an increasing time window, and detects whether the data volume of the updating information in the cache exceeds a preset data volume threshold value or not; and if the data quantity of the updated data in the cache exceeds a preset data quantity threshold value, emptying the updated data in the cache. And if the data quantity of the updated data in the cache does not exceed the preset data quantity threshold, storing the updated data in the cache.

Step S310, the long window is fixed.

Step 311, unique identification.

Specifically, when updated data enters the target database, the updated data is judged, and when the data meets the aggregation screening condition, a screening field with a unique identifier in the updated data is placed into a cache.

And step S312, caching.

In step S313, if the time threshold is exceeded, the deletion is performed.

Specifically, the server adopts fixed-length time window dynamic updating, and whether the existing time of the updated data in the cache exceeds a preset time threshold value is detected; and if the existing time of the updated data in the cache exceeds a preset time threshold, emptying the updated data in the cache. And if the existing time of the updated data in the cache does not exceed the preset time threshold, storing the updated data in the cache.

Through the steps, the ClickHouse database is used for overcoming the defects of the traditional mode of equipment portrait analysis, the BitMap function of the ClickHouse database supports a BitMap structure, the BitMap calculation is good, the memory occupation and the aggregation consumption time are simplified, the real-time mode of the equipment portrait is supported by adopting a plug-in mode, and the performance consumption is further reduced.

It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.

In this embodiment, a dynamic update device based on clickwouse is also provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and the description already made is omitted for brevity. The terms "module," "unit," "subunit," and the like as used below may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 4 is a block diagram of a dynamic update apparatus based on clickwouse according to the present embodiment, and as shown in fig. 4, the apparatus includes: aninformation acquisition module 410, adatabase filtering module 420, adata update module 430, an updatedata filtering module 440, and adynamic update module 450.

And aninformation obtaining module 410, configured to obtain screening information of the device portrait.

And thedatabase screening module 420 is configured to screen the target database according to the screening information to obtain a screening result.

And adata updating module 430, configured to obtain update data of the target database.

And the updatedata screening module 440 is configured to screen the update data according to the screening information to obtain update information.

And adynamic update module 450, configured to dynamically update the screening result according to the update information.

Theinformation obtaining module 410 is further configured to obtain a plurality of screening fields of the target database, where the screening fields include: at least one of service time of the equipment, last overhaul time, the number of faults, the number of error log entries, equipment ID, log type, error log, address of a recently accessed client, longitude and latitude of the equipment and alarm information; and selecting the corresponding screening field to generate the screening information.

Thedatabase screening module 420 is further configured to obtain a plurality of screening information of the device representation; screening a target database according to the screening information to obtain a plurality of corresponding screening results;

and aggregating the results of the plurality of screening tasks through an aggregation function to obtain a final screening result.

Thedatabase screening module 420 is further configured to perform aggregation statistics according to the screening fields included in the plurality of screening information, and obtain a screening field set; and screening a plurality of screening task results according to the screening field set to obtain a final screening result.

Thedata updating module 430 is further configured to monitor the updated data according to the screening information; and loading the updated data containing the screening information into a cache.

Thedata updating module 430 is further configured to detect whether the existing time of the updated data in the cache exceeds a preset time threshold; and if the existing time of the updated data in the cache exceeds a preset time threshold, emptying the updated data in the cache.

Thedata updating module 430 is further configured to detect whether a data amount of the update information in the cache exceeds a preset data amount threshold; and if the data quantity of the updated data in the cache exceeds a preset data quantity threshold value, emptying the updated data in the cache.

The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.

There is also provided in this embodiment an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

acquiring screening information of the equipment portrait;

screening the target database according to the screening information to obtain a screening result;

acquiring update data of a target database;

screening the updating data according to the screening information to obtain updating information;

and dynamically updating the screening result according to the updating information.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program: and acquiring a plurality of screening fields of the target database, selecting the corresponding screening fields, and generating the screening information.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program: obtaining a plurality of screening information of the equipment image; screening a target database according to the screening information to obtain a plurality of corresponding screening results; and aggregating the results of the plurality of screening tasks through an aggregation function to obtain a final screening result.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program: performing aggregation statistics according to screening fields contained in the plurality of screening information to obtain a screening field set; and screening a plurality of screening task results according to the screening field set to obtain a final screening result.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program: monitoring the updating data according to the screening information; and loading the updated data containing the screening information into a cache.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program: detecting whether the existing time of the updated data in the cache exceeds a preset time threshold; and if the existing time of the updated data in the cache exceeds a preset time threshold, emptying the updated data in the cache.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program: detecting whether the data volume of the updating information in the cache exceeds a preset data volume threshold value or not; and if the data quantity of the updated data in the cache exceeds a preset data quantity threshold value, emptying the updated data in the cache.

It should be noted that, for specific examples in this embodiment, reference may be made to the examples described in the foregoing embodiments and optional implementations, and details are not described again in this embodiment.

In addition, in combination with the clickwouse-based dynamic update method provided in the foregoing embodiment, a storage medium may also be provided to implement this embodiment. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements any of the above embodiments of the clickwouse-based dynamic update method.

It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to be limiting. All other embodiments, which can be derived by a person skilled in the art from the examples provided herein without any inventive step, shall fall within the scope of protection of the present application.

It is obvious that the drawings are only examples or embodiments of the present application, and it is obvious to those skilled in the art that the present application can be applied to other similar cases according to the drawings without creative efforts. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

The term "embodiment" is used herein to mean that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly or implicitly understood by one of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the patent protection. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims (10)

1. A dynamic updating method based on ClickHouse is characterized by comprising the following steps:

acquiring screening information of the equipment portrait;

screening the target database according to the screening information to obtain a screening result;

acquiring update data of a target database;

screening the updating data according to the screening information to obtain updating information;

and dynamically updating the screening result according to the updating information.

2. The clickwouse-based dynamic update method according to claim 1, wherein the obtaining of the device portrait screening information includes:

obtaining a plurality of screening fields of a target database, wherein the screening fields include: at least one of service time of the equipment, last overhaul time, the number of faults, the number of error log entries, equipment ID, log type, error log, address of a recently accessed client, longitude and latitude of the equipment and alarm information;

and selecting the corresponding screening field to generate the screening information.

3. The ClickHouse-based dynamic updating method according to claim 2, wherein the screening the target database according to the screening information to obtain the screening result comprises:

obtaining a plurality of screening information of the equipment image;

screening a target database according to the screening information to obtain a plurality of corresponding screening results;

and aggregating the results of the plurality of screening tasks through an aggregation function to obtain a final screening result.

4. The ClickHouse-based dynamic update method according to claim 3, wherein the aggregating the results of the plurality of screening tasks through an aggregation function to obtain a final screening result comprises;

performing aggregation statistics according to screening fields contained in the plurality of screening information to obtain a screening field set;

and screening a plurality of screening task results according to the screening field set to obtain a final screening result.

5. The ClickHouse-based dynamic update method according to claim 1, wherein the obtaining update data of the target database includes:

monitoring the updating data according to the screening information;

and loading the updated data containing the screening information into a cache.

6. The ClickHouse-based dynamic update method according to claim 5, wherein the loading the update data containing the filtering information into the cache comprises:

detecting whether the existing time of the updated data in the cache exceeds a preset time threshold;

and if the existing time of the updated data in the cache exceeds a preset time threshold, emptying the updated data in the cache.

7. The ClickHouse-based dynamic update method according to claim 5, wherein the loading the update data containing the filtering information into the cache comprises:

detecting whether the data volume of the updating information in the cache exceeds a preset data volume threshold value or not;

and if the data quantity of the updated data in the cache exceeds a preset data quantity threshold value, emptying the updated data in the cache.

8. A dynamic update apparatus based on ClickHouse, comprising:

the information acquisition module is used for acquiring screening information of the equipment portrait;

the database screening module is used for screening the target database according to the screening information to obtain a screening result;

the data updating module is used for acquiring the updating data of the target database;

the updating data screening module is used for screening the updating data according to the screening information to obtain updating information;

and the dynamic updating module is used for dynamically updating the screening result according to the updating information.

9. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is configured to execute the computer program to perform the ClickHouse-based dynamic update method according to any one of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the ClickHouse-based dynamic update method according to any one of claims 1 to 7.

Images