Disclosure of Invention
In view of the above, embodiments of the present invention provide a method, an apparatus, an electronic device, and a storage medium for acquiring report data, which can solve the problem that when report data is acquired, a data cluster runs slowly or is down due to an excessive acquired data amount, so that the system performance of the data cluster is reduced.
In order to achieve the above object, according to an aspect of the embodiment of the present invention, a report data acquisition method is provided.
The report data acquisition method of the embodiment of the invention comprises the following steps: receiving a report data query request sent by a terminal, wherein the query request carries a report identifier; inquiring data generation time corresponding to the report data identifier in a report data identifier set based on the report identifier, splitting the data generation time corresponding to the report data identifier into a plurality of time intervals, and generating data requests corresponding to the time intervals by combining the corresponding report data identifier; and issuing the data request to a data cluster, so that the data cluster processes the data request based on a preset data request execution rule, and further receives report data processed by the data cluster and returns the report data to the terminal.
In one embodiment, further comprising:
Inquiring expiration time of report data corresponding to the report identifier from a first-level cache so as to judge whether the report data corresponding to the report identifier in the first-level cache is data-expired;
Under the condition that the data is not expired, acquiring report data corresponding to the report identifier from the first-level cache so as to send the report data to the terminal;
And under the condition that the data is out of date, executing the steps of generating and issuing the data request, inquiring the expiration time of the report data corresponding to the report identifier from a second-level cache to judge whether the report data corresponding to the report identifier in the second-level cache is out of date, if not, acquiring the report data corresponding to the report identifier from the second-level cache, and sending the report data to the terminal, and if so, returning the report data processed by the data cluster to the terminal, wherein the priority of the first-level cache is higher than that of the second-level cache.
In yet another embodiment, after receiving the report data after the data cluster processing, the method further includes:
updating report data corresponding to the report type in the first-level cache and the second-level cache based on the report data processed by the data cluster, and setting expiration time of the updated report data in the first-level cache and the second-level cache.
In yet another embodiment, the setting the expiration time of the updated report data in the first level cache and the second level cache includes:
Inquiring report update time corresponding to the report identifier to set the report update time as the expiration time of the updated report data in the first-level cache;
Inquiring the preset expiration time of the report identifier corresponding to the second-level cache to set the expiration time of the updated report data in the second-level cache, wherein the preset expiration time of the report identifier corresponding to the second-level cache is larger than the report update time.
In yet another embodiment, receiving the report data after the data clustering processing and returning to the terminal includes:
inquiring a summary column identifier corresponding to the report identifier, summarizing the report data processed by the data cluster based on column data corresponding to the summary column identifier, and sending the summarized report data to the terminal.
In yet another embodiment, the querying the report data identifier set for the data generation time corresponding to the report data identifier based on the report identifier includes:
Under the condition that the target task lock is in a locking state, querying the data generation time corresponding to the report data identifier in the report data identifier set to fail;
And under the condition that the target task lock is in an unlocked state, inquiring data generation time corresponding to the report data identifier in the report data identifier set based on the report identifier, and updating the state of the target task lock to be a locked state.
In order to achieve the above object, according to another aspect of the embodiments of the present invention, there is provided an apparatus for acquiring report data.
The report data acquisition device of the embodiment of the invention comprises: the receiving unit is used for receiving a report data query request sent by the terminal and obtaining a report identifier in the query request;
The processing unit is used for inquiring the data generation time corresponding to the report data identifier in the report data identifier set based on the report identifier, splitting the data generation time corresponding to the report data identifier into a plurality of time intervals, and generating data requests corresponding to the time intervals by combining the corresponding report data identifier; and issuing the data request to a data cluster, so that the data cluster processes the data request based on a preset data request execution rule, and further receives report data processed by the data cluster and returns the report data to the terminal.
In one embodiment, the processing unit is specifically configured to:
Inquiring expiration time of report data corresponding to the report identifier from a first cache to judge whether the report data corresponding to the report identifier in the first-level cache is data-expired;
Under the condition that the data is not expired, acquiring report data corresponding to the report identifier from the first-level cache so as to send the report data to the terminal;
And under the condition that the data is out of date, executing the steps of generating and issuing the data request, inquiring the expiration time of the report data corresponding to the report identifier from a second-level cache to judge whether the report data corresponding to the report identifier in the second-level cache is out of date, if not, acquiring the report data corresponding to the report identifier from the second-level cache, and sending the report data to the terminal, and if so, returning the report data processed by the data cluster to the terminal, wherein the priority of the first-level cache is higher than that of the second-level cache.
In yet another embodiment, the apparatus further comprises:
and the updating unit is used for updating the report data corresponding to the report type in the first-level cache and the second-level cache based on the report data processed by the data cluster, and setting the expiration time of the updated report data in the first-level cache and the second-level cache.
In a further embodiment, the updating unit is specifically configured to:
Inquiring report update time corresponding to the report identifier to set the report update time as the expiration time of the updated report data in the first-level cache;
Inquiring the preset expiration time of the report identifier corresponding to the second-level cache to set the expiration time of the updated report data in the second-level cache, wherein the preset expiration time of the report identifier corresponding to the second-level cache is larger than the report update time.
In a further embodiment, the processing unit is specifically configured to:
inquiring a summary column identifier corresponding to the report identifier, summarizing the report data processed by the data cluster based on column data corresponding to the summary column identifier, and sending the summarized report data to the terminal.
In a further embodiment, the processing unit is specifically configured to:
Under the condition that the target task lock is in a locking state, querying the data generation time corresponding to the report data identifier in the report data identifier set to fail;
And under the condition that the target task lock is in an unlocked state, inquiring data generation time corresponding to the report data identifier in the report data identifier set based on the report identifier, and updating the state of the target task lock to be a locked state.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided an electronic apparatus.
An electronic device according to an embodiment of the present invention includes: one or more processors; and the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors realize the report data acquisition method provided by the embodiment of the invention.
To achieve the above object, according to still another aspect of an embodiment of the present invention, a computer-readable medium is provided.
The computer readable medium of the embodiment of the invention stores a computer program, and the program is executed by a processor to realize the report data acquisition method provided by the embodiment of the invention.
One embodiment of the above invention has the following advantages or benefits: in the embodiment of the invention, after receiving a report data query request sent by a terminal, a report identifier in the report data query request can be obtained, and the data generation time corresponding to the report data identifier in a corresponding report data identifier set can be queried based on the report identifier, namely, the generation time of the report data is required to be obtained, then each data generation time can be divided into a plurality of time intervals, each time interval is combined with the corresponding report data identifier to generate a data request, the data request is issued to a data cluster, the data cluster can process the data request issued to the data cluster based on a preset data request execution rule, the processed report data is returned, and the report data processed by the data cluster can be returned to the terminal. In the embodiment of the invention, when the report data is acquired from the data cluster, the data generation time corresponding to each report data identifier can be divided into a plurality of time intervals, and the data request is generated based on the time intervals, and because the data generation time is divided, the time interval included in each data request is smaller than the data generation time, namely the report data volume originally acquired by one request is divided into a plurality of data requests to be acquired, so that the plurality of data requests are issued to the data cluster, the data cluster can process the received plurality of data requests based on the preset data request execution rule, and each data request can process the report data corresponding to the data generation time in batches based on the data request execution rule, namely the data request execution rule is not required to process the report data corresponding to the data generation time for a plurality of times, so that the data volume of the data cluster once processed is reduced, the data volume of the data cluster once processed based on the data request execution rule is reduced even in special periods such as a huge promotion period, the running is avoided from being slow or down, and the system performance of the data cluster is improved.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It is noted that embodiments of the invention and features of the embodiments may be combined with each other without conflict.
The embodiment of the invention provides a report data acquisition method, which can be executed by a server, as shown in fig. 1, and comprises the following steps:
S101: and receiving a report data query request sent by the terminal, and acquiring a report identifier in the query request.
The terminal can be report generation equipment, and can generate a report by sending a report data query request to the server so as to query corresponding report data. The server side can be provided with a report data acquisition system, in particular a large storage system which can be connected with each data cluster to acquire report data required by generating each report from the data clusters. The data clusters may include a plurality of data clusters, and each data cluster may be used to store data corresponding to each report.
In this step, the server may receive a report data query request sent by the terminal, where the report data query request includes a report identifier, and the report data query request is used to indicate which report is queried, so that the report identifier may be obtained from the report data query request.
In the embodiment of the invention, the server can set a unified portal for communication with the terminal to receive the report data query request sent by the terminal, specifically can be jsf unified portals, and can be based on the implementation configuration class corresponding to the report type, so that the data can be not configured independently when newly accessed in the report, and the configuration efficiency is improved. The report identifier can be an identifier obtained by combining the report data identifier and the service parameter in a report data identifier set included in the corresponding report.
S102: based on the report identifications, inquiring data generation time corresponding to the report data identifications in the report data identification set, dividing the data generation time corresponding to the report data identifications into a plurality of time intervals, and generating data requests corresponding to the time intervals by combining the corresponding report data identifications.
The report data identifier represents a data identifier included in the report data queried at this time, the report data identifier set is a set of data identifiers included in the report data queried at this time, for example, the report data queried is yesterday order-placing amount, and the report data identifier is order-placing amount identifier. The data generation time represents the generation time of the queried report data, namely the data generated by the report data identification represented by the data in the corresponding data generation time is the report data of the query request. For example, if the report data is the order amount within 0 to 12 points yesterday, the data generation time corresponding to the order amount is from 0 to 12 points yesterday.
In the embodiment of the invention, on one hand, because report data required by report generation usually has regularity, the server side can pre-configure each report data identification set corresponding to the report data required by each report generation and data generation time corresponding to each report data identification in the report data identification set, and establish a corresponding relation between the report identification and the report data identification set and the data generation time corresponding to each report data identification in the report data identification set, so that the report identification can be queried based on the corresponding relation in the step, and further query the report data identification set corresponding to the report identification and the data generation time corresponding to each report data identification in the report data identification set. On the other hand, the report data query request may include a report data identifier set corresponding to the report data of the query and a data generation time corresponding to each report data identifier in the report data identifier set, so in this step, each report data identifier and the corresponding data generation time may be obtained from the query request based on the report identifiers.
After the data generation time corresponding to each report data identifier is obtained, each data generation time can be split into a plurality of time intervals, namely, the data generation time is segmented, and the corresponding data generation time can be obtained by combining the time intervals. Specifically, in the embodiment of the present invention, a segmentation period corresponding to the report data identifier may be preset, so that the data generation time of the report data identifier is split into a plurality of time intervals based on the segmentation period. If a certain report data identifier can be set with a segmentation period of 60 minutes, then the starting time and the ending time of the corresponding data generation time can be obtained based on the report data identifier, and the starting time and the ending time of each split time interval can be specifically realized through the following codes.
@Override
public String getBeginTime(ReportReqBean bean){
return bean.getOutsideBeginTime()；
}
The current time is denoted by// nul1
@Override
public String getEndTime(ReportReqBean bean){
return bean.getOutsideEndTime()；
}
The units of// are minutes, the query period is segmented, -1 represents the non-segmentation
@Override
public int intervalCycle(){
return 60；
}
Let the start time of the data generation time (begin time) be: 2021-03-22 09:00:00, end time (EndTime) is: 2021-03-22 11:59:59, with a segmentation period (INTERVALCYCLE) of 60 minutes, the code is run to split the data generation time into time intervals of 60 minutes in duration. In the above code operation, if the end time of the data generation time is nul a 1, the default end time may be the current time. In the embodiment of the invention, the data generation time is not split, for example, the segmentation period is set to be-1 so as to indicate that the data generation time is not split, 60 can be modified to be-1 in the code, and the data generation time can be determined not to be split when the code runs.
After each data generation time is split into a plurality of time intervals, a corresponding data request can be generated for each time interval by combining a corresponding report data identifier, so that a plurality of data requests are generated. Specifically, in the embodiment of the present invention, a code template corresponding to the data request may be configured in advance, and then parameters such as a time interval and a corresponding report data identifier are substituted into the code template to generate the data request.
It should be noted that, the data request may further include a plurality of parameters for acquiring report data, for example, report identifier, interface information, cluster information, etc., where each parameter may be stored in advance, and may be obtained and brought into the code template based on the report identifier query when generating the data request, so as to generate the data request.
S103: and sending a data request to the data cluster so that the data cluster processes the data request based on a preset data request execution rule, and further receiving report data processed by the data cluster and returning the report data to the terminal.
After the data request is generated, the generated data request may be issued to the data cluster, so as to obtain report data from the data cluster.
After each data request is received in the data cluster, rule processing may be performed based on a preset data request, for example, each data request is added to a queue and then sequentially processed. The data cluster processes each data request to obtain report data in a time interval corresponding to the data request, and then the processed report data can be sent to the server, and after the server receives the report data processed by the data cluster, the server obtains the report data queried by the report data query request in step S101, and then the report data can be returned to the terminal.
In the embodiment of the invention, after receiving report data processed by the data cluster, the server can summarize the report data and then return the report data to the terminal. In the embodiment of the invention, a data summarizing mode can be preset, specifically, a summarizing column identifier can be preset, namely, report data is summarized by data in a column corresponding to the summarizing column identifier, and for example, the data with the same value in the column corresponding to the summarizing column identifier can be combined. For example, the summary column identifier may be a warehouse number, and summarizing the report data may be performed to merge data with the same warehouse number in the received report data.
For example, in the embodiment of the present invention, the summary key, that is, the summary column identifier, may be represented by summaryKey in the code, and the data that needs to be combined (added) when the summary is represented by the count word, so as to improve the code development efficiency. Specifically, the following code is shown.
String sql＝”select e.wh_no as
summaryKey,substring_index(e.wh_no,'-',-2)aswhNo,count(distinct(e.order_source_no))asexpCount"+"from obd_order_exception e join obd_orderoo on e.order_source_no＝oo.source_no andoo.yn＝1"+"WHERE(oo.create_time>＝str_to_date('"+time Section[0]+"','％Y-％m-％d％H:％i:％S')andoo.create_time<＝str_to_date('"+timeSection[1]+"','％Y--％m-％d％H:％i:％S')+param+"and oo.order_type＝1and e.is_finished＝0and oo.wh_no＝e.wh_no"+"GROUP BYoo.wh_no"";
In the above code, wh_no is the summary column identifier, denoted by summaryKey, and the data fields that need to be added when summarized are denoted by expCount.
In the embodiment of the invention, the summary column identifier can be set according to requirements, and can be specifically an identifier of one field or a combination of a plurality of field identifiers. For example, as shown in fig. 2, the data is stored in a general database, in which the time field is in the format shown in the create_time field in fig. 2, the data in one week needs to be summarized from the data stored in this way by hour+by warehouse (wh_no), the time field needs to intercept the information of the create_time field (year-month-day and hour) and the warehouse field for groupby summarization, and the data code in 7 days of summarization is written into the create_time >2021-01-01 00:00:00and create_time<2021-01-0623:59:59 generally, but the data volume in this time span is too large, so that slow sql is easy to be generated to cause the database cpu to rise. The specific code of each time interval at the time of data summarization can be as follows.
String sql＝"SELETC concat(oo.wh_no,'_',ifnull(substring_index(oo.shipped_time,':'1),"))"+"as summaryKey,substring_index(substring_index(oo.shipped_time,':'1),':'1)as time,oo.dc_no as dcNo"+",SUM(CASE WHENoo.order_status IN('9')THEN 1ELSE 0END)AScount"+"FROM obd_order oo LEFT JOIN obd_wave owON oo.wave_no＝ow.wave_no AND oo.wh_no＝oo.wh_no"+"AND oo.yn＝1 AND ow.yn＝1JOIN mst_dcdc ON oo.dc_no＝da.da_no and dc.new_org＝1left joinmst_area area"+"on dc.area_no＝area.area_no left joinmst_wh wh on oo.wh_no＝wh.wh_no"+"WHERE"+"(oo.shipped_time<＝str_to_date('"+timeSection[0]+"','％Y-％m-％d％H:％i:％S')"+"andoo.shipped_time<＝str_to_date('"+timeSection[1]+"','％Y--％m-％d％H:％i:％S'))"+param+"and oo.order_type＝"1"and oo.is_customer_cancel＝0and wh.wh_business_type in(1,2)"+"AND oo.yn＝1GROUP BY oo.wh_no";
In the code, the summary dimension is wh_no field and the year-month-day-hour part intercepted in shipped _time, so that the summary dimension is summaryKey, namely the summary column is identified as wh_no field and the year-month-day-hour part intercepted in shipped _time, and the code finally passes through wh_no field group by.
It should be noted that, the data cluster returns the processed report data according to the received data request, that is, each report data is sent to the server end in multiple times, so in the embodiment of the invention, the report data belonging to the report can be determined based on the report identifier, then the report data in each time interval can be combined into a whole, and summarized by the summary column identifier, or in order to avoid that the data volume is too large after the report data in each time interval is combined into a whole, the report data corresponding to each time interval can be summarized by the summary column identifier, and then the summarized results of each time interval are combined. If the data clusters comprise a plurality of data clusters, the report data of each data cluster also needs to be combined and summarized.
In the embodiment of the invention, report data corresponding to each time interval is received, and then the report data can be summarized through summarykey, so that one large data size is split into a plurality of small data sizes to be acquired and summarized, and the performance risk of the system is reduced.
In the embodiment of the invention, when the report data is acquired from the data cluster, the data generation time corresponding to the report data identifier can be divided into a plurality of time intervals, and the data request is generated based on the time intervals, and because the data generation time is divided, the time interval included in each data request is smaller than the data generation time, namely the report data volume acquired by the original request is divided into a plurality of data requests to be acquired, so that the plurality of data requests are issued to the data cluster, the data cluster can process the received plurality of data requests based on the preset data request execution rule, and each data request can be processed in batches based on the data request execution rule, namely the report data corresponding to the data generation time is not required to be processed for one time, but is processed for a plurality of times based on the data request execution rule, so that the data volume of the data cluster for one time processing data is reduced, and even in special periods such as large promotion, the data volume of the data cluster for one time processing the report data can be reduced based on the data request execution rule, thereby the occurrence of slow running or downtime is avoided, and the system performance of the data cluster is improved.
In connection with the embodiment shown in fig. 1, a system architecture of a report data acquisition system in the embodiment of the present invention may be shown in fig. 3. In order to improve the system performance and stability of the acquisition system, the embodiment of the invention can set a multi-level cache mode to store the acquired report data.
In fig. 3, the system architecture includes a unified portal, an Sql rule, a report data summary mode, a multithreaded trigger, and a cache. The unified portal may be used to receive report data query requests.
In order to improve the system performance and stability of the acquisition system, in the embodiment of the invention, the acquired report data may be stored in a two-level cache manner, which is a first-level cache (first-level cache in fig. 3) and a second-level cache (second-level cache in fig. 3), respectively, and the priority of the first-level cache is set higher than that of the second-level cache. The report data stored in the first-level cache and the second-level cache are set with expiration time, and whether the report data in the caches can be used for report generation can be judged based on the expiration time. Specifically, after step S101, the expiration time of the report data corresponding to the report identifier may be queried from the first-level cache, so as to determine whether the report data corresponding to the report identifier in the first-level cache is data-expired. If the report data corresponding to the report identifier in the first-level cache is judged to be not outdated, the report data corresponding to the report identifier can be directly obtained from the first-level cache and sent to the terminal; if the report data corresponding to the report identifier in the first-level cache is judged to be data expiration, the report data acquisition process can be triggered, that is, the step S102 is executed and a data request is issued to the data cluster, and meanwhile, the expiration time of the report data corresponding to the report identifier can be queried from the second-level cache to judge whether the report data corresponding to the report identifier in the second-level cache is data expiration. If the report data corresponding to the report identifier in the second-level cache is judged to be data unexpired, which means that the report data in the second-level cache can be used, the report data corresponding to the report identifier can be obtained from the second-level cache and sent to the terminal, and if the report data corresponding to the report identifier in the second-level cache is judged to be data unexpired, which means that the report data in the second-level cache is expired, which means that the report data returned from the received data cluster in step S103 is needed to be sent to the terminal. In the embodiment of the invention, the first-level cache and the second-level cache can be jimdb caches.
Parameters required for generating the data request corresponding to each report and rules for generating the data request are configured in the Sql rule, for example, report data identifier corresponding to a report data identifier set and data generation time corresponding to the report data identifier in the report data identifier set, and splitting modes, segmentation periods, cluster information and the like of the data generation time can be used for acquiring the data request of the report data.
The cluster summarization, the time interval summarization and the mark summarization are summarization modes of report data, and can be used for realizing summarization of received report data. The cluster summarization represents summarizing report data sent by different clusters, the time interval summarization represents summarizing report data corresponding to each time interval, and the mark summarization represents summarizing the report data based on the summarization column mark.
The multi-thread trigger can be used for executing the generation of data requests and the summarization of report data, and can send trigger signals to the multi-thread trigger when the report data acquisition flow is triggered, the multi-thread trigger can generate data requests based on configuration information in the Sql rule and send the data requests to each data cluster, and after receiving the report data sent by each data cluster, the data clusters are summarized in the summarization mode and then synchronously stored in the primary cache, the secondary cache and the database cache.
In order to break down the mode, in the embodiment of the invention, a distributed task lock, namely a target task station, can be set, and only one report data acquisition flow is triggered within the same time period of the set value, namely only one trigger signal is issued to the multithread trigger, so that when executing the step S102: under the condition that the target task lock is in a locking state, querying the data generation time failure corresponding to the report data identification in the report data identification set, namely, the execution failure in the step S102; and under the condition that the target task lock is in an unlocked state, inquiring the data generation time corresponding to the report data identifier in the report data identifier set based on the report identifier, and updating the state of the target task lock to be a locked state, namely executing step S102.
The method for acquiring report data in the embodiment of the present invention is specifically described below with reference to the embodiment shown in fig. 1 and the system architecture shown in fig. 3. As shown in fig. 4, the method includes:
S401: and receiving a report data query request sent by the terminal, and acquiring a report identifier in the query request.
S402: inquiring expiration time of report data corresponding to the report identifier from the first-level cache to judge whether the report data corresponding to the report identifier in the first-level cache is out of date, and if not, executing step S403; if yes, steps S404 and S405 are performed.
The first level cache is a level one cache in the structure shown in fig. 3. The method comprises the steps of firstly inquiring expiration time of report data corresponding to report identifiers from a first-level cache so as to judge whether the report data corresponding to the report identifiers in the first-level cache is data-expired.
It should be noted that, in the embodiment of the present invention, expiration time is set for each report data in the first-level cache, and this step can compare the current time with the expiration time when executing, and if the current time is greater than the expiration time, it can determine that the data is expired; if the current time is not greater than the expiration time, it may be determined that the non-data is expired.
S403: and acquiring report data corresponding to the report identifier from the first-level cache so as to send the report data to the terminal.
If step S402 determines that the data is not expired, it indicates that the report data corresponding to the report identifier in the first-level cache can be used, so this step may be performed.
S404: inquiring data generation time corresponding to the report data identifier in the report data identifier set based on the report identifier, splitting the data generation time corresponding to the report data identifier into a plurality of time intervals, and generating data requests corresponding to the time intervals by combining the corresponding report data identifier; and sending a data request to the data cluster, and further receiving report data processed by the data cluster.
If the report data corresponding to the report identifier in the first-level cache is out of date, the report data corresponding to the report identifier is required to be updated, and the report data acquisition process can be triggered, namely the step is executed.
S405: inquiring expiration time of report data corresponding to the report identifier from the second-level cache to judge whether the report data corresponding to the report identifier in the second-level cache is out of date, and if not, executing step S406; if yes, go to step S407.
The second level cache is the second level cache in the structure shown in fig. 3. If the report data corresponding to the report identifier in the first-level cache is out of date, the report data acquisition process may be triggered, that is, step S102 may be executed, and at the same time, the report data corresponding to the report identifier may be attempted to be acquired from the second-level cache, that is, the expiration time of the report data corresponding to the report identifier may be queried from the second-level cache, so as to determine whether the data is out of date.
In the embodiment of the invention, the expiration time is set for each report data in the second-level cache, the step can compare the current time with the expiration time when the step is executed, and if the current time is greater than the expiration time, the expiration of the data can be judged; if the current time is not greater than the expiration time, it may be determined that the non-data is expired.
S406: and acquiring report data corresponding to the report identifier from the second-level cache, and sending the report data to the terminal.
If it is determined in step S405 that the non-data is expired, this step may be executed because it is indicated that the report data corresponding to the report identifier in the second-level cache may be used.
S407: and returning the report data after the received data cluster processing to the terminal.
If the step S405 determines that the data is out of date, it indicates that the report data corresponding to the report identifier in the second-level cache is not usable, so that the report data after the data cluster processing received after the step S404 is executed may be returned to the terminal.
It should be noted that, in the embodiment of the present invention, in executing step S407, the report data after the data cluster processing may not be received yet, and at this time, report data corresponding to the report identifier may also be obtained from the database cache, so as to send the report data to the terminal. Therefore, before executing step S407, it may be further determined whether report data sent by the data cluster after executing step S404 is received, if yes, then executing step S407, and if not, obtaining report data corresponding to the report identifier from the database cache, so as to send the report data to the terminal.
According to the embodiment of the invention, the two-level cache is arranged, so that the access of the database cache is reduced, meanwhile, the report data acquisition flow is triggered when the report data in the first-level cache is out of date, the report data acquisition times are reduced, frequent report data acquisition is avoided, the system performance can be improved, and the system stability is ensured.
Meanwhile, by adopting the report data acquisition mode in the embodiment of the invention, the multithreading processing process is shielded, the sql rule and the summarization mode are set, the report data acquisition and summarization details are shielded, and the system development time and the system risk are greatly saved.
Based on the system architecture shown in fig. 4, after executing step S404, the report data may also be updated, and the present invention provides an implementation manner to explain the method for updating the report data after step S404 in the embodiment shown in fig. 4. As shown in fig. 5, the method includes:
s501: and updating report data corresponding to the report type in the first-level cache and the second-level cache based on the report data processed by the data clusters.
After receiving the report data processed by the data cluster, the report data can be updated to report data corresponding to the report type in the first-level cache and the second-level cache. Specifically, the report data after the data cluster processing can be summarized first, and then the summarized report data is updated to the report data corresponding to the report types in the first-level cache, the second-level cache and the database cache. The updating mode can be specifically to store the corresponding relation between the report identifier and the summarized report data and delete the expired report data corresponding to the report identifier.
It should be noted that, in this step, after step S404, the report data corresponding to the report type in the second-level cache may be updated, where the updated report data is newly acquired, and the data will not expire in general, so if the execution process of step S404 and this step is faster, when step S405 is executed, it is possible to query the report data corresponding to the report identifier from the second-level cache for the newly acquired report data, and at this time, the report data received in step S404 may be returned to the terminal.
S502: and setting expiration time of the updated report data in the first-level cache and the second-level cache.
And setting expiration time of the report data corresponding to the updated report data in the first-level cache and the second-level cache.
Specifically, in the embodiment of the present invention, the expiration time of the report data in the first-level cache may be set to be the update time of the report data, and the expiration time of the report data in the second-level cache may be set to be longer than the update time of the report data. The steps may be specifically performed as: inquiring report update time corresponding to the report identifier so as to set the report update time as expiration time of updated report data in the first-level cache; inquiring the preset expiration time of the report identifier corresponding to the second-level cache to set the expiration time of the updated report data in the second-level cache, wherein the preset expiration time of the report identifier corresponding to the second-level cache is larger than the report update time.
In the embodiment of the invention, the report data in each buffer memory is updated so as to provide corresponding report data for the terminal during inquiry.
In order to solve the problems in the prior art, an embodiment of the present invention provides a device 600 for acquiring report data, as shown in fig. 6, the device 600 includes:
The receiving unit 601 is configured to receive a report data query request sent by a terminal, and obtain a report identifier in the query request;
The processing unit 602 is configured to query, based on the report identifier, a data generation time corresponding to the report data identifier in the report data identifier set, split the data generation time corresponding to the report data identifier into a plurality of time intervals, and generate a data request corresponding to each time interval in combination with the corresponding report data identifier; and issuing the data request to a data cluster, so that the data cluster processes the data request based on a preset data request execution rule, and further receives report data processed by the data cluster and returns the report data to the terminal.
It should be understood that the manner of implementing the embodiment of the present invention is the same as that of implementing the embodiment shown in fig. 1, and will not be described herein.
In one implementation manner of the embodiment of the present invention, the processing unit 602 is specifically configured to:
Inquiring expiration time of report data corresponding to the report identifier from a first cache to judge whether the report data corresponding to the report identifier in the first-level cache is data-expired;
Under the condition that the data is not expired, acquiring report data corresponding to the report identifier from the first-level cache so as to send the report data to the terminal;
And under the condition that the data is out of date, executing the steps of generating and issuing the data request, inquiring the expiration time of the report data corresponding to the report identifier from a second-level cache to judge whether the report data corresponding to the report identifier in the second-level cache is out of date, if not, acquiring the report data corresponding to the report identifier from the second-level cache, and sending the report data to the terminal, and if so, returning the report data processed by the data cluster to the terminal, wherein the priority of the first-level cache is higher than that of the second-level cache.
In yet another implementation manner of the embodiment of the present invention, the apparatus 600 further includes:
and the updating unit is used for updating the report data corresponding to the report type in the first-level cache and the second-level cache based on the report data processed by the data cluster, and setting the expiration time of the updated report data in the first-level cache and the second-level cache.
In yet another implementation manner of the embodiment of the present invention, the updating unit is specifically configured to:
Inquiring report update time corresponding to the report identifier to set the report update time as the expiration time of the updated report data in the first-level cache;
Inquiring the preset expiration time of the report identifier corresponding to the second-level cache to set the expiration time of the updated report data in the second-level cache, wherein the preset expiration time of the report identifier corresponding to the second-level cache is larger than the report update time.
In yet another implementation manner of the embodiment of the present invention, the processing unit 602 is specifically configured to:
inquiring a summary column identifier corresponding to the report identifier, summarizing the report data processed by the data cluster based on column data corresponding to the summary column identifier, and sending the summarized report data to the terminal.
In yet another implementation manner of the embodiment of the present invention, the processing unit 602 is specifically configured to:
Under the condition that the target task lock is in a locking state, querying the data generation time corresponding to the report data identifier in the report data identifier set to fail;
And under the condition that the target task lock is in an unlocked state, inquiring data generation time corresponding to the report data identifier in the report data identifier set based on the report identifier, and updating the state of the target task lock to be a locked state.
It should be understood that the manner of implementing the embodiments of the present invention is the same as that of implementing the embodiments shown in fig. 1, 4 or 5, and will not be described herein.
In the embodiment of the invention, when the report data is acquired from the data cluster, the data generation time corresponding to the report data identifier can be divided into a plurality of time intervals, and the data request is generated based on the time intervals, and because the data generation time is divided, the time interval included in each data request is smaller than the data generation time, namely the report data volume acquired by the original request is divided into a plurality of data requests to be acquired, so that the plurality of data requests are issued to the data cluster, the data cluster can process the received plurality of data requests based on the preset data request execution rule, and each data request can be processed in batches based on the data request execution rule, namely the report data corresponding to the data generation time is not required to be processed for one time, but is processed for a plurality of times based on the data request execution rule, so that the data volume of the data cluster for one time processing data is reduced, and even in special periods such as large promotion, the data volume of the data cluster for one time processing the report data can be reduced based on the data request execution rule, thereby the occurrence of slow running or downtime is avoided, and the system performance of the data cluster is improved.
According to an embodiment of the present invention, an electronic device and a readable storage medium are also provided.
The electronic equipment of the embodiment of the invention comprises: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor, so that the at least one processor executes the report data acquisition method provided by the embodiment of the invention.
Fig. 7 illustrates an exemplary system architecture 700 of a report data acquisition method or report data acquisition apparatus to which embodiments of the present invention may be applied.
As shown in fig. 7, a system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. The network 704 is the medium used to provide communication links between the terminal devices 701, 702, 703 and the server 705. The network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 705 via the network 704 using the terminal devices 701, 702, 703 to receive or send messages or the like. Various client applications may be installed on the terminal devices 701, 702, 703.
The terminal devices 701, 702, 703 may be, but are not limited to, smartphones, tablets, laptop and desktop computers, and the like.
The server 705 may be a server providing various services, and may perform processing such as analysis on received report data query request and the like, and feed back processing results (e.g., report data—only an example) to the terminal device.
It should be noted that, the report data acquiring method provided in the embodiment of the present invention is generally executed by the server 705, and accordingly, the report data acquiring device is generally disposed in the server 705.
It should be understood that the number of terminal devices, networks and servers in fig. 7 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 8, there is illustrated a schematic diagram of a computer system 800 suitable for use in implementing embodiments of the present invention. The computer system shown in fig. 8 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU) 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, mouse, etc.; an output portion 807 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 808 including a hard disk or the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. The drive 810 is also connected to the I/O interface 805 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as needed so that a computer program read out therefrom is mounted into the storage section 808 as needed.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 809, and/or installed from the removable media 811. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 801.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a unit, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present invention may be implemented in software or in hardware. The described units may also be provided in a processor, for example, described as: a processor includes a receiving unit, a generating unit, and a transmitting unit. The names of these units do not in any way constitute a limitation of the unit itself, for example, a receiving unit may also be described as a "unit of the function of the receiving unit".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by one of the apparatuses, cause the apparatus to perform the report data acquisition method provided by the present invention.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.