Detailed Description
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the application herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that, related information (including, but not limited to, user equipment information, user personal information, etc.) and data (including, but not limited to, data for presentation, analyzed data, etc.) related to the present disclosure are information and data authorized by a user or sufficiently authorized by each party. For example, an interface is provided between the system and the relevant user or institution, before acquiring the relevant information, the system needs to send an acquisition request to the user or institution through the interface, and acquire the relevant information after receiving the consent information fed back by the user or institution.
The elastiscearch is a distributed, highly extended, high real-time search and data analysis engine. It can conveniently make a large amount of data have the capability of searching, analyzing and searching. Currently, the method is commonly used for e-commerce commodity searching, app searching, enterprise internal information searching, IT system searching and the like. Most of these products need to store and query massive data, and the complexity of the data during query is very high, which means that with the increase of the data volume, the memory space needed during data retrieval is larger and larger. The saving and releasing of memory becomes particularly important when storing the same data amount and providing the same retrieval function, and therefore, an elastic search-based data search method is proposed based on the above technical background.
The present application will be described with reference to preferred implementation steps, and fig. 1 is a flowchart of an elastic search-based data searching method according to an embodiment of the present application, as shown in fig. 1, and the method includes the following steps:
step S101, determining an index to be queried according to a data search request, and determining an index state of the index to be queried, wherein the index state is one of a first index state and a second index state, data stored in the index of the first index state is cached in an elastic search memory, and data stored in the index of the second index state is not cached in the elastic search memory;
specifically, a data search request initiated by a user is received, wherein the data search request comprises relevant information of data which the user needs to inquire. For example, the user wants to search log data of 10 months 11 days. And determining an index to be queried according to the data search request, wherein the index to be queried stores data searched by a user. After the index to be queried is determined according to the data search request, the index state of the index to be queried is determined.
The index states include a first index state, which may also be referred to as a hot index state, and a second index state, which may also be referred to as a cold index state. The data stored in the index of the hot index state is cached in the elastic search memory, and the data stored in the index of the cold index state is not cached in the elastic search memory. Generally, the index of the data which is not frequently searched is set to be in a cold index (freeze) state, and the index in the cold index state is not loaded into the memory of the elastic search virtual machine when storing the data, so that the memory space is not occupied, and the aim of releasing ELASTICSEARCH memories is fulfilled.
Step S102, if the index to be queried is an index in a first index state, searching from an elastic search memory to obtain first data, and storing the first data in a cache region;
specifically, if the index to be queried is in a hot index state, the first data is directly obtained by searching from the elastic search memory according to the data search request and the index to be queried, and then the first data is stored in the cache region.
Step S103, if the index to be queried is an index in a second index state, caching data stored in the index to be queried into an elastic search memory, searching from the elastic search memory according to a data searching request and the index to be queried to obtain second data, and storing the second data in a cache region;
Specifically, if the index to be queried is an index in a cold index state, firstly caching data stored in the index to be queried into an elastic search memory, then searching from the elastic search memory according to a data search request to obtain second data, and storing the second data in a cache region. After the second data is obtained, deleting the data corresponding to the index to be queried of the elastic search memory, and timely releasing the memory space.
Step S104, determining the search result of the data search request from the data in the buffer area.
And finally, determining the search result of the data search request through the data in the cache area.
It should be noted that, the index to be queried may be one index or a plurality of indexes, and if the index to be queried is a plurality of indexes, the above steps are performed on each index to realize data retrieval.
In summary, in the present embodiment, the index is divided into the first index state and the second index state. For the index in the second index state, the index is in a non-loading (unloading) state at ordinary times, the memory space of the elastic search is not occupied, when the data stored in the second index state is searched, the data corresponding to the index in the second index state is loaded into the elastic search memory, the corresponding data is searched after the loading, the memory space is released by removing the loading processing after the searching is finished, the data searching is realized by increasing the memory space, and the effect of reducing the cost required by the memory is further achieved.
In order to improve accuracy of determining indexes to be queried, in the data searching method based on the elastic search, determining the indexes to be queried according to the data searching request comprises determining a searching time range and attribute information of data to be searched according to the data searching request, and determining the indexes to be queried according to the searching time range and the attribute information of the data to be searched.
Specifically, a data search request initiated by a user is received, wherein the data search request comprises relevant information of data which the user needs to inquire. And determining a search time range which the user wants to search and attribute information of the data to be searched according to the data search request. For example, the user wants to search log data of 10 months 11-10 months 17 days. And then determining the index to be queried according to the searching time range and the attribute information of the data to be searched. Since the index is named by using the creation date of the index and the data type stored in the index when the index is created, the index to be queried can be accurately identified through the search time range and the attribute information of the data to be searched.
In order to improve efficiency and accuracy of data detection, in the data searching method based on the elastic search provided by the embodiment of the application, if the index to be queried is an index in a first index state, searching the first data from the elastic search memory comprises the steps of converting a searching time range corresponding to a data searching request to obtain a time stamp range, and searching the first data from the elastic search memory according to the time stamp range and the index to be queried.
Specifically, when data is stored in the index, a timestamp corresponding to the data is correspondingly marked, so that a search time range corresponding to the data search request is converted into a timestamp range, and then the first data is searched from the elastic search memory according to the timestamp range and the index to be queried.
In order to alleviate the problem of excessive occupation of an elastic search memory, in the data searching method based on the elastic search provided by the embodiment of the application, after the second data is stored in the cache region, the method further comprises deleting the data corresponding to the index to be queried in the elastic search memory.
Specifically, after the second data is obtained, deleting the data corresponding to the index to be queried of the elastic search memory, and timely releasing the memory space.
In an alternative embodiment, when there are multiple indexes to be queried in a cold index state, the data retrieval step is sequentially executed according to creation time of the indexes to be queried, firstly, one index with earliest creation time in all indexes in the cold index state is found, corresponding data is loaded into an elastic search memory (load), then the data (read) to be retrieved in the index is queried in the elastic search memory, and the data (read) to be retrieved is temporarily stored in a cache region. And then unloading the data corresponding to the index from the elastic search memory, releasing the memory space, and continuing to search the index of the cold index state with the earliest creation time until all indexes of the cold index state are subjected to load- > read- > unloading flow, and completing the search of the index data of the cold index state.
In the data searching method based on the elastic search, before the index to be queried is determined according to the data searching request, the method further comprises the steps of establishing the index in a preset time period, and storing data in a preset time range into the index, wherein the index is named according to the creation date of the index and the data type stored in the index.
Specifically, an index is established for a preset time period, and then data of the preset time range is stored in the index. In an alternative embodiment, the data is stored in days, with zero-point per day creating an index based on the current system time, with index names named log type + current date. For example, the stored data is syslog and the current date is 2022-10-03, then an index named "syslog-2022-10-03" is created at the zero point of 2022-10-03, and all syslog logs on subsequent 2022-10-03 days are all stored in this index.
In order to improve accuracy of updating an index state, in the data searching method based on the elastic search provided by the embodiment of the application, after data in a preset time range is stored in an index, whether the index state of the index is updated is determined by one of setting an occupation threshold of an elastic search memory and the threshold number of the index in a first index state, determining whether the index state of the index is updated according to the occupation threshold and the threshold number, setting a creation time threshold, and determining whether the index state of the index is updated according to the creation time threshold.
Determining whether to update the index state of the index according to the occupation threshold value and the threshold number comprises detecting the occupation proportion of the elastic search memory and the number of indexes currently in a first index state, updating the index state of a first target index into a second index state if the occupation proportion is greater than or equal to the occupation threshold value, and deleting data corresponding to the first target index in the elastic search memory until the occupation proportion is smaller than the occupation threshold value or the number of indexes currently in the first index state is equal to the threshold number, wherein the creation time of the first target index is earlier than that of the rest indexes.
Determining whether to update the index state of the index according to the creation time threshold comprises counting the creation time of the index currently in the first index state, judging whether the difference value between the creation time of the index currently in the first index state and the current time is larger than or equal to the creation time threshold, if the difference value between the creation time of the second target index in the first index state and the current time is larger than or equal to the creation time threshold, updating the index state of the second target index into the second index state, and deleting the data corresponding to the second target index in the elastic search memory.
Specifically, an index cold processing scheme is set according to the actual memory size and the use condition. The method is mainly divided into the following two modes:
(1) Setting an occupation threshold value of an elastic search memory and setting the threshold value number of indexes in a first index state, and determining whether to update the index state of the indexes according to the occupation threshold value and the threshold value number;
(2) Setting a creation time threshold, and determining whether to update the index state of the index according to the creation time threshold.
Setting an occupancy threshold of an elastic search memory and the number of index thresholds in a hot index state at least according to the first mode. And monitoring the occupation condition of the elastic search memory in real time/periodically, and updating the index state of one or more indexes with the oldest index creation time into a cold index state when the occupation condition of the elastic search memory reaches a set threshold value, so as to release the memory space. If the memory occupancy of the elastic search is not smaller than the set occupancy threshold, continuing to update the index state until the memory occupancy is smaller than the set threshold or the number of indexes in the hot index state reaches the set threshold number.
For the second way, the earliest creation time (i.e., the creation time threshold described above) of the indexes in the hot index state is set, e.g., to 30 days, and index state checks are made periodically (e.g., daily) to see if there are indexes in the hot index state 30 days earlier than the current time, and if so, the index states of these indexes are updated to the cold index state.
Through the scheme, the memory space is saved and released, and larger-scale data storage and data retrieval are supported under the condition that the memory space is not increased, so that the fund and labor investment brought by increasing the memory space are greatly reduced.
In an alternative embodiment, the data search based on the elastic search may be implemented using a flowchart as shown in FIG. 2.
(1) All indexes are found according to the queried time range and the name (creation time) of the index.
(2) And inquiring the index states of all indexes, if the indexes are in a hot index state (open state), directly inquiring the corresponding data in the indexes from the elastic search and returning the data, and temporarily storing the inquired data in a cache region.
(3) If the index state of the index is a cold index state (freeze state), the following operations are performed:
a) Finding all the freeze indexes, creating an index with earliest time, and loading corresponding data into a memory (load);
b) Inquiring the data (read) to be searched in the index, and temporarily storing the data (read) in a cache area;
c) Unloading the data corresponding to the index from the memory to release the memory space;
d) Removing the index from the indexes to be queried, and continuing to query the next freeze state index with the earliest creation time;
e) Until all the freeze indexes undergo load- > read- > unloading flow, and the freeze state index data query is completed;
(4) Summarizing all the data queried by the open state index and the freeze state index, and returning the summarized data to a querying user;
(5) This query ends.
The data searching method based on the elastic search provided by the embodiment of the application determines the index to be queried according to the data searching request and determines the index state of the index to be queried, wherein the index state is one of a first index state and a second index state, the data stored in the index of the first index state is cached in the elastic search memory, the data stored in the index of the second index state is not cached in the elastic search memory, if the index to be queried is the index of the first index state, the first data is obtained by searching in the elastic search memory and is stored in a cache region, and if the index to be queried is the index of the second index state, the data stored in the index to be queried is cached in the elastic search memory and is searched according to the data searching request and the index to be queried, the second data is obtained by searching in the elastic search memory, and the second data is stored in the cache region, and the searching result of the data searching request is determined from the data in the cache region, so that the problem that the memory is not occupied by the data which is too large to be used up when the data is searched based on the elastic search in the related technology is solved. In this scheme, the index is divided into a first index state and a second index state. For the index in the second index state, the index is in a non-loading (unloading) state at ordinary times, the memory space of the elastic search is not occupied, when the data stored in the second index state is searched, the data corresponding to the index in the second index state is loaded into the elastic search memory, the corresponding data is searched after the loading, the memory space is released by removing the loading processing after the searching is finished, the data searching is realized by increasing the memory space, and the effect of reducing the cost required by the memory is further achieved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
The embodiment of the application also provides a data searching device based on the elastic search, and it should be noted that the data searching device based on the elastic search of the embodiment of the application can be used for executing the data searching method based on the elastic search provided by the embodiment of the application. The following describes an elastiscearch-based data search device provided by an embodiment of the present application.
Fig. 3 is a schematic diagram of an elastiscearch-based data search apparatus according to an embodiment of the present application. As shown in fig. 3, the apparatus includes a first determining unit 301, a first searching unit 302, a second searching unit 303, and a second determining unit 304.
A first determining unit 301, configured to determine an index to be queried according to a data search request, and determine an index state of the index to be queried, where the index state is one of a first index state and a second index state, where data stored in the index of the first index state is cached in an elastic search memory, and data stored in the index of the second index state is not cached in the elastic search memory;
The first search unit 302 is configured to search from the elastic search memory to obtain first data if the index to be queried is an index in a first index state, and store the first data in the buffer;
the second search unit 303 is configured to cache, if the index to be queried is an index in the second index state, data stored in the index to be queried into an elastic search memory, search from the elastic search memory according to the data search request and the index to be queried to obtain second data, and store the second data in the cache region;
a second determining unit 304 for determining a search result of the data search request from the data in the buffer.
Specifically, a data search request initiated by a user is received, wherein the data search request comprises relevant information of data which the user needs to inquire. For example, the user wants to search log data of 10 months 11 days. And determining an index to be queried according to the data search request, wherein the index to be queried stores data searched by a user. After the index to be queried is determined according to the data search request, the index state of the index to be queried is determined.
The index states include a first index state, which may also be referred to as a hot index state, and a second index state, which may also be referred to as a cold index state. The data stored in the index of the hot index state is cached in the elastic search memory, and the data stored in the index of the cold index state is not cached in the elastic search memory. Generally, the index of the data which is not frequently searched is set to be in a cold index (freeze) state, and the index in the cold index state is not loaded into the memory of the elastic search virtual machine when storing the data, so that the memory space is not occupied, and the aim of releasing ELASTICSEARCH memories is fulfilled.
If the index to be queried is in a hot index state, the first data is directly obtained by searching from the elastic search memory according to the data searching request and the index to be queried, and then the first data is stored in the cache region.
If the index to be queried is in the cold index state, firstly caching the data stored in the index to be queried into an elastic search memory, then searching from the elastic search memory according to a data search request to obtain second data, and storing the second data in a cache region. After the second data is obtained, deleting the data corresponding to the index to be queried of the elastic search memory, and timely releasing the memory space. And finally, determining the search result of the data search request through the data in the cache area.
It should be noted that, the index to be queried may be one index or a plurality of indexes, and if the index to be queried is a plurality of indexes, the above steps are performed on each index to realize data retrieval.
In summary, in the present embodiment, the index is divided into the first index state and the second index state. For the index in the second index state, the index is in a non-loading (unloading) state at ordinary times, the memory space of the elastic search is not occupied, when the data stored in the second index state is searched, the data corresponding to the index in the second index state is loaded into the elastic search memory, the corresponding data is searched after the loading, the memory space is released by removing the loading processing after the searching is finished, the data searching is realized by increasing the memory space, and the effect of reducing the cost required by the memory is further achieved.
According to the data searching device based on the elastic search, the first determining unit 301 determines the index to be queried according to the data searching request and determines the index state of the index to be queried, wherein the index state is one of a first index state and a second index state, data stored in the index of the first index state are cached in the elastic search memory, data stored in the index of the second index state are not cached in the elastic search memory, the first searching unit 302 searches for the first data from the elastic search memory if the index to be queried is the index of the first index state, the first data is stored in the cache region, the second searching unit 303 caches the data stored in the index to be queried into the elastic search memory if the index to be queried is the index of the second index state, searches for the second data from the elastic search memory according to the data searching request and the index to be queried, the second data is stored in the cache region, and the second data is determined to be stored in the cache region when the second determining unit 304 searches for the data from the search region and even the memory is not occupied by the related data. In this scheme, the index is divided into a first index state and a second index state. For the index in the second index state, the index is in a non-loading (unloading) state at ordinary times, the memory space of the elastic search is not occupied, when the data stored in the second index state is searched, the data corresponding to the index in the second index state is loaded into the elastic search memory, the corresponding data is searched after the loading, the memory space is released by removing the loading processing after the searching is finished, the data searching is realized by increasing the memory space, and the effect of reducing the cost required by the memory is further achieved.
Optionally, in the data searching device based on the elastic search provided by the embodiment of the application, the first determining unit 301 includes a first determining module configured to determine a searching time range and attribute information of data to be searched according to a data searching request, and a second determining module configured to determine an index to be queried according to the searching time range and the attribute information of the data to be searched.
Specifically, a data search request initiated by a user is received, wherein the data search request comprises relevant information of data which the user needs to inquire. And determining a search time range which the user wants to search and attribute information of the data to be searched according to the data search request. For example, the user wants to search log data of 10 months 11-10 months 17 days. And then determining the index to be queried according to the searching time range and the attribute information of the data to be searched. Since the index is named by using the creation date of the index and the data type stored in the index when the index is created, the index to be queried can be accurately identified through the search time range and the attribute information of the data to be searched.
Optionally, in the data searching device based on the elastic search provided by the embodiment of the application, the first searching unit 302 includes a conversion module, configured to convert a search time range corresponding to a data searching request to obtain a timestamp range, and a searching module, configured to search for first data from an elastic search memory according to the timestamp range and an index to be queried.
Specifically, when data is stored in the index, a timestamp corresponding to the data is correspondingly marked, so that a search time range corresponding to the data search request is converted into a timestamp range, and then the first data is searched from the elastic search memory according to the timestamp range and the index to be queried.
Optionally, in the data searching device based on the elastic search provided by the embodiment of the application, the device further comprises a building unit, which is used for building an index in a preset time period and storing data in a preset time range into the index before determining the index to be queried according to the data searching request, wherein the index is named according to the creation date of the index and the data type stored in the index.
Specifically, an index is established for a preset time period, and then data of the preset time range is stored in the index. In an alternative embodiment, the data is stored in days, with zero-point per day creating an index based on the current system time, with index names named log type + current date. For example, the stored data is syslog and the current date is 2022-10-03, then an index named "syslog-2022-10-03" is created at the zero point of 2022-10-03, and all syslog logs on subsequent 2022-10-03 days are all stored in this index.
Optionally, in the data searching device based on the elastic search provided by the embodiment of the application, after the data in the preset time range is stored in the index, one of a first setting unit, which is used for setting an occupation threshold of the elastic search memory and the threshold number of the indexes in the first index state, and determining whether to update the index state of the index according to the occupation threshold and the threshold number, and a second setting unit, which is used for setting a creation time threshold, and determining whether to update the index state of the index according to the creation time threshold, is adopted.
Optionally, in the data searching device based on the elastic search provided by the embodiment of the application, the first setting unit comprises a detection module, a first updating module and a first setting module, wherein the detection module is used for detecting the occupation proportion of the elastic search memory and the number of indexes currently in a first index state, and the first updating module is used for updating the index state of the first target index into a second index state if the occupation proportion is greater than or equal to an occupation threshold value, deleting data corresponding to the first target index in the elastic search memory until the occupation proportion is smaller than the occupation threshold value or the number of indexes currently in the first index state is equal to the threshold number, wherein the creation time of the first target index is earlier than that of the rest indexes.
Optionally, in the data searching device based on the elastic search provided by the embodiment of the application, the second setting unit comprises a statistics module, a judging module and a second replacing module, wherein the statistics module is used for counting the creation time of the index in the first index state currently, the judging module is used for judging whether the difference value between the creation time of the index in the first index state currently and the current time is larger than or equal to a creation time threshold value, and the second replacing module is used for updating the index state of the second target index into the second index state and deleting the data corresponding to the second target index in the elastic search memory if the difference value between the creation time of the second target index in the first index state and the current time is larger than or equal to the creation time threshold value.
Specifically, an index cold processing scheme is set according to the actual memory size and the use condition. The method is mainly divided into the following two modes:
(1) Setting an occupation threshold value of an elastic search memory and setting the threshold value number of indexes in a first index state, and determining whether to update the index state of the indexes according to the occupation threshold value and the threshold value number;
(2) Setting a creation time threshold, and determining whether to update the index state of the index according to the creation time threshold.
Setting an occupancy threshold of an elastic search memory and the number of index thresholds in a hot index state at least according to the first mode. And monitoring the occupation condition of the elastic search memory in real time/periodically, and updating the index state of one or more indexes with the oldest index creation time into a cold index state when the occupation condition of the elastic search memory reaches a set threshold value, so as to release the memory space. If the memory occupancy of the elastic search is not smaller than the set occupancy threshold, continuing to update the index state until the memory occupancy is smaller than the set threshold or the number of indexes in the hot index state reaches the set threshold number.
For the second way, the earliest creation time (i.e., the creation time threshold described above) of the indexes in the hot index state is set, e.g., to 30 days, and index state checks are made periodically (e.g., daily) to see if there are indexes in the hot index state 30 days earlier than the current time, and if so, the index states of these indexes are updated to the cold index state.
Through the scheme, the memory space is saved and released, and larger-scale data storage and data retrieval are supported under the condition that the memory space is not increased, so that the fund and labor investment brought by increasing the memory space are greatly reduced.
Optionally, in the data searching device based on the elastic search provided by the embodiment of the application, the device further comprises a deleting unit, which is used for deleting the data corresponding to the index to be queried in the elastic search memory after the second data is stored in the cache region.
Specifically, after the second data is obtained, deleting the data corresponding to the index to be queried of the elastic search memory, and timely releasing the memory space.
In an alternative embodiment, when there are multiple indexes to be queried in a cold index state, the data retrieval step is sequentially executed according to creation time of the indexes to be queried, firstly, one index with earliest creation time in all indexes in the cold index state is found, corresponding data is loaded into an elastic search memory (load), then the data (read) to be retrieved in the index is queried in the elastic search memory, and the data (read) to be retrieved is temporarily stored in a cache region. And then unloading the data corresponding to the index from the elastic search memory, releasing the memory space, and continuing to search the index of the cold index state with the earliest creation time until all indexes of the cold index state are subjected to load- > read- > unloading flow, and completing the search of the index data of the cold index state.
In an alternative embodiment, the data search based on the elastic search may be implemented using a flowchart as shown in FIG. 2.
(1) All indexes are found according to the queried time range and the name (creation time) of the index.
(2) And inquiring the index states of all indexes, if the indexes are in a hot index state (open state), directly inquiring the corresponding data in the indexes from the elastic search and returning the data, and temporarily storing the inquired data in a cache region.
(3) If the index state of the index is a cold index state (freeze state), the following operations are performed:
a) Finding all the freeze indexes, creating an index with earliest time, and loading corresponding data into a memory (load);
b) Inquiring the data (read) to be searched in the index, and temporarily storing the data (read) in a cache area;
c) Unloading the data corresponding to the index from the memory to release the memory space;
d) Removing the index from the indexes to be queried, and continuing to query the next freeze state index with the earliest creation time;
e) Until all the freeze indexes undergo load- > read- > unloading flow, and the freeze state index data query is completed;
(4) Summarizing all the data queried by the open state index and the freeze state index, and returning the summarized data to a querying user;
(5) This query ends.
The elastic search-based data search device includes a processor and a memory, and the first determination unit 301, the first search unit 302, the second search unit 303, the second determination unit 304, and the like described above are stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.
The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one, and the data search based on the elastiscearch is realized by adjusting kernel parameters.
The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.
The embodiment of the invention provides a computer-readable storage medium having a program stored thereon, which when executed by a processor, implements an elastosearch-based data search method.
The embodiment of the invention provides a processor, which is used for running a program, wherein the program runs to execute an elastiscearch-based data searching method.
As shown in FIG. 4, the embodiment of the invention provides an electronic device, which comprises a processor, a memory and a program stored on the memory and capable of running on the processor, wherein the processor determines an index to be queried according to a data search request and determines an index state of the index to be queried, the index state is one of a first index state and a second index state, data stored in the index of the first index state is cached in an elastic search memory, data stored in the index of the second index state is not cached in the elastic search memory, if the index to be queried is the index of the first index state, the first data is searched from the elastic search memory and stored in a cache region, and if the index to be queried is the index of the second index state, the data stored in the index to be queried is cached in the elastic search memory and searched from the elastic search memory according to the data search request and the index to be queried, the second data is obtained and stored in the second data is stored in the search region, and the cache result of the data in the cache region is determined from the data search request.
Optionally, determining the index to be queried according to the data search request comprises determining a search time range and attribute information of the data to be searched according to the data search request, and determining the index to be queried according to the search time range and the attribute information of the data to be searched.
Optionally, if the index to be queried is an index in a first index state, searching the first data from the elastic search memory includes converting a search time range corresponding to the data search request to obtain a time stamp range, and searching the first data from the elastic search memory according to the time stamp range and the index to be queried.
Optionally, after the second data is stored in the buffer, the method further includes deleting the data corresponding to the index to be queried in the elastic search memory.
Optionally, before determining the index to be queried according to the data search request, the method further comprises establishing the index in a preset time period, and storing data in a preset time range into the index, wherein the index is named according to the creation date of the index and the data type stored in the index.
Optionally, after storing the data in the preset time range in the index, determining whether to update the index state of the index is performed by one of setting an occupancy threshold of the elastic search memory and a threshold number of indexes in the first index state, determining whether to update the index state of the index according to the occupancy threshold and the threshold number, setting a creation time threshold, and determining whether to update the index state of the index according to the creation time threshold.
Optionally, determining whether to update the index state of the index according to the occupancy threshold and the threshold number includes detecting an occupancy proportion of the elastic search memory and the number of indexes currently in the first index state, updating the index state of the first target index to the second index state if the occupancy proportion is greater than or equal to the occupancy threshold, and deleting data corresponding to the first target index in the elastic search memory until the occupancy proportion is less than the occupancy threshold or the number of indexes currently in the first index state is equal to the threshold number, wherein the creation time of the first target index is earlier than that of the remaining indexes.
Optionally, determining whether to update the index state of the index according to the creation time threshold includes counting creation time of the index currently in the first index state, judging whether a difference value between the creation time of the index currently in the first index state and the current time is greater than or equal to the creation time threshold, if the difference value between the creation time of the second target index in the first index state and the current time is greater than or equal to the creation time threshold, updating the index state of the second target index to the second index state, and deleting data corresponding to the second target index in the elastomer search memory.
The device herein may be a server, PC, PAD, cell phone, etc.
The application also provides a computer program product which is suitable for executing a program initialized with the following method steps when being executed on data processing equipment, wherein the program is used for determining an index to be queried according to a data search request and determining an index state of the index to be queried, the index state is one of a first index state and a second index state, data stored in the index of the first index state are cached in an elastic search memory, data stored in the index of the second index state are not cached in the elastic search memory, if the index to be queried is the index of the first index state, the first data are searched from the elastic search memory and stored in a cache region, and if the index to be queried is the index of the second index state, the data stored in the index to be queried are cached in the elastic search memory, searching is carried out from the elastic search memory according to the data search request and the index to be queried to obtain the second data, the second data are stored in the cache region, and the search result of the data search request is determined from the data in the cache region.
Optionally, determining the index to be queried according to the data search request comprises determining a search time range and attribute information of the data to be searched according to the data search request, and determining the index to be queried according to the search time range and the attribute information of the data to be searched.
Optionally, if the index to be queried is an index in a first index state, searching the first data from the elastic search memory includes converting a search time range corresponding to the data search request to obtain a time stamp range, and searching the first data from the elastic search memory according to the time stamp range and the index to be queried.
Optionally, after the second data is stored in the buffer, the method further includes deleting the data corresponding to the index to be queried in the elastic search memory.
Optionally, before determining the index to be queried according to the data search request, the method further comprises establishing the index in a preset time period, and storing data in a preset time range into the index, wherein the index is named according to the creation date of the index and the data type stored in the index.
Optionally, after storing the data in the preset time range in the index, determining whether to update the index state of the index is performed by one of setting an occupancy threshold of the elastic search memory and a threshold number of indexes in the first index state, determining whether to update the index state of the index according to the occupancy threshold and the threshold number, setting a creation time threshold, and determining whether to update the index state of the index according to the creation time threshold.
Optionally, determining whether to update the index state of the index according to the occupancy threshold and the threshold number includes detecting an occupancy proportion of the elastic search memory and the number of indexes currently in the first index state, updating the index state of the first target index to the second index state if the occupancy proportion is greater than or equal to the occupancy threshold, and deleting data corresponding to the first target index in the elastic search memory until the occupancy proportion is less than the occupancy threshold or the number of indexes currently in the first index state is equal to the threshold number, wherein the creation time of the first target index is earlier than that of the remaining indexes.
Optionally, determining whether to update the index state of the index according to the creation time threshold includes counting creation time of the index currently in the first index state, judging whether a difference value between the creation time of the index currently in the first index state and the current time is greater than or equal to the creation time threshold, if the difference value between the creation time of the second target index in the first index state and the current time is greater than or equal to the creation time threshold, updating the index state of the second target index to the second index state, and deleting data corresponding to the second target index in the elastomer search memory.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.