Invention content
The embodiment of the present invention provides a kind of daily record monitoring method and daily record monitoring system, to solve at existing daily record dataThe problem of reason mode, daily record data processing is rough, and obtained monitor control index accuracy is poor.
In order to solve the above-mentioned technical problem, an embodiment of the present invention provides a kind of daily record monitoring method, the method includes：
Acquire the daily record data in goal systems；
The daily record data is parsed, obtains the parsing data of the daily record data；
It is calculated using the parsing data, so as to obtain weighing the monitor control index of goal systems operation conditions.
Further, it is described acquisition goal systems in daily record data the step of after, the method further includes：
Store the collected daily record data；
After the step of being parsed described to the daily record data, obtaining the parsing data of the daily record data, instituteThe method of stating includes：
Store the parsing data obtained later to daily record data parsing；
Parsing data after the parsing using the daily record data are calculated, so as to obtain weighing goal systemsAfter the step of monitor control index of operation conditions, the method includes：
Store the monitor control index being calculated.
Further, it is calculated described using the parsing data, so as to obtain weighing goal systems operation conditionsMonitor control index the step of after, the method includes：
Show the monitor control index being calculated.
Further, it is described that the daily record data is parsed, the step of obtaining the parsing data of the daily record data,Including：
The daily record data according to different fields is parsed, obtains the field parsing data of the daily record data.
Further, it is described to be calculated using the parsing data, so as to obtain weighing goal systems operation conditionsThe step of monitor control index, including：
The first parsing data of the first daily record data in object time window are obtained in real time, obtain the first parsing data setIt closes；
Based on the first parsing data acquisition system, the first monitor control index of the goal systems is calculated；
The first preset time after the first parsing data acquisition system is obtained, obtains shadow from history log dataSecond parsing data acquisition system of the second daily record data in time window, wherein, the object time window and the shadow timeThe start time point of window is identical with time point is terminated；
Based on the described second parsing data acquisition system, the second monitor control index of the goal systems is calculated；
First monitor control index is corrected using second monitor control index.
Further, described the step of correcting first monitor control index using second monitor control index, including：
First monitor control index is covered using second monitor control index.
Further, after described the step of correcting first monitor control index using second monitor control index, instituteThe method of stating includes：
" the k-th presumed time after the kth parsing data acquisition system is obtained, from history log data is performed successivelyThe kth+1 for obtaining corresponding+1 daily record data of kth in kth shadow time window parses data acquisition system, wherein, the object timeThe start time point of window and the kth shadow time window and to terminate time point identical ", " data are parsed based on the kth+1Set calculates+1 monitor control index of kth of the goal systems " and " correct kth monitoring using+1 monitor control index of kth to refer toThe step of mark "；
Until k reaches preset quantity N, stop cycle calculations, wherein, N is the integer more than or equal to 2, k to be more than orPerson is equal to 2 and the integer less than or equal to N.
The embodiment of the present invention also provides a kind of daily record monitoring system, and the daily record monitoring system includes：
Log acquisition module, for acquiring the daily record data in goal systems；
Log analyzing module for being parsed to the daily record data, obtains the parsing data of the daily record data；
Monitor control index computing module, for being calculated using the parsing data, so as to obtain weighing goal systems fortuneThe monitor control index of row situation.
Further, the daily record monitoring system includes：
Daily record memory module, for storing the collected daily record data of the log acquisition module；
Data memory module is parsed, what is obtained later is parsed to the daily record data for storing the log analyzing moduleThe parsing data；
Monitor control index memory module, for storing the monitor control index that the monitor control index computing module is calculated.
Further, the daily record monitoring system further includes：
Monitor control index display module, for showing the monitor control index that the monitor control index computing module is calculated.
Further, the log analyzing module is additionally operable to parse the daily record data according to different fields,Obtain the field parsing data of the daily record data.
Further, the monitor control index computing module includes：
First acquisition unit for obtaining first of the first daily record data in object time window the parsing data in real time, obtainsTo the first parsing data acquisition system；
First computing unit for being based on the first parsing data acquisition system, calculates the first monitor control index of the goal systems；
Second acquisition unit, for the first preset time after the first parsing data acquisition system is obtained, from historyThe second parsing data acquisition system of the second daily record data in the first shadow time window is obtained in daily record data, wherein, the targetTime window is identical with the start time point of the first shadow time window and termination time point；
Second computing unit for being based on the second parsing data acquisition system, calculates the second monitoring of the goal systemsIndex；
First amending unit, for second monitor control index to be used to correct first monitor control index.
Further, first amending unit is also used for second monitor control index covering, first monitoring and refers toMark.
Further, the monitor control index computing module further includes：
Recycle execution unit, for perform successively " the k-th presumed time after kth parsing data acquisition system is obtained,The kth+1 that corresponding+1 daily record data of kth in kth shadow time window is obtained from history log data parses data acquisition system,Wherein, the start time point of the object time window and the kth shadow time window and to terminate time point identical ", " be based onThe kth+1 parses data acquisition system, calculates+1 monitor control index of kth of the goal systems " and " monitored using the kth+1Index corrects kth monitor control index " the step of；
Stop unit for reaching preset quantity N up to k, stops cycle calculations, wherein, N is more than or equal to 2Integer, k are the integer more than or equal to 2 and less than or equal to N.
A kind of computer readable storage medium is stored with computer program on the computer readable storage medium, describedThe step of above-mentioned daily record monitoring method is realized when computer program is executed by processor.
Daily record monitoring method provided in an embodiment of the present invention and daily record monitoring system, acquire the daily record number in goal systemsAccording to；The daily record data is parsed, obtains the parsing data of the daily record data；It is counted using the parsing dataIt calculates, so as to obtain weighing the monitor control index of goal systems operation conditions.In this way, calculate mesh by the parsing data of daily record dataThe monitor control index of mark system, the calculating of monitor control index can be very accurate and fine, so as to accurately reflect target systemOperating status and health status of system etc. facilitate operation maintenance personnel the work such as to be monitored and safeguard to goal systems, mitigate O＆MThe operating pressure of personnel, and then goal systems is reduced because monitoring the probability for causing to break down not in time, to reduce the system failureCaused by lose.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, completeSite preparation describes, it is clear that described embodiment is part of the embodiment of the present invention, instead of all the embodiments.Based on this hairEmbodiment in bright, the every other implementation that those of ordinary skill in the art are obtained without creative effortsExample, shall fall within the protection scope of the present invention.
Referring to Fig. 1, Fig. 1 is the flow chart for the daily record monitoring method that one embodiment of the invention provides.The method can answerSystem is monitored for daily record, as shown in Figure 1, the described method comprises the following steps：
Daily record data in step 101, acquisition goal systems.
The goal systems is the business event IT system for needing to monitor.There are many application journeys in business event IT systemSequence, these application programs assume responsibility for the various businesses of enterprise, such as the business such as deposit at bank, loan, enchashment, these business systemsSystem can generate daily record during the various customer demands of normal handling, and each of log recording operation system was runJourney, the business such as current IT, internet, mobile Internet flourish, and are the very big facility of society band, meanwhile, constantlyThe portfolio of growth also proposes higher requirement to the maintenance work of business event IT system.Business IT system is carried out completeMonitoring to include daily record monitoring be the premise that enterprise ensures the steady health operation of business IT system.
Therefore, in order to be monitored in time to goal systems, in this step, the daily record monitoring system can be describedWhen goal systems is run, the daily record data in the goal systems is acquired.
Wherein, the daily record data in goal systems is acquired, can be real-time progress data acquisition.
Wherein, the daily record data in goal systems is acquired, can be by data processing pipeline, such as servers' dataProcessing pipeline logstash carrys out collector journal for log collection, polymerization and the system of transmission Flume.In addition,Logstash can also be stored in the daily record data of acquisition the distributed of distributed memory system high-throughput such as kafka and send outCloth is subscribed in message system or RabbitMQ message queues.
Step 102 parses the daily record data, obtains the parsing data of the daily record data.
In the step, the daily record monitors system after the daily record data is collected, can to the daily record data intoThe fine parsing of row, obtains the parsing data of the daily record data.
Wherein, the daily record data is parsed, obtains the parsing data of the daily record data, can be by the dayWill data are parsed according to different fields, so as to obtain the field of daily record data parsing data.According to the daily recordThe fields such as date, time, daily record rank, log information in the field in data, such as the daily record data, extract institute respectivelyThe field divided in daily record data is stated, to be parsed to daily record data.Wherein, the task of daily record parsing can be by above-mentionedWhat logstash was completed, it can read daily record data on one side, be parsed on one side.
In this way, the field in daily record data, parses daily record data, daily record data can be carried out more accurateAnalysis, to reflect more detailed situation, convenient for subsequently to the calculating of monitor control index.
For example, daily record data is got as daily record monitors system, the daily record of expression is 17-09-2114:29:34.846|endpoint[TCP_CPS]-thread-19|4dcca2a3-c4|E|Begin|touda.syslog.endpoint.TCP_CPS|poup_qrpay_web|J_PCS_022_0111|182.180.202.155|30323030f238202020 | failed, then the daily record monitoring system can be according to the field in daily record data, by carryingIt the modes such as takes, daily record data is parsed into different subfields, if multiple subfields after splitting can be 17-09-21,14:29:34.846、endpoint[TCP_CPS]-thread-19、4dcca2a3-c4、E、Begin、touda.syslog.endpoint.TCP_CPS、poup_qrpay_web、J_PCS_022_0111、182.180.202.155、30323030f238202020, failed, then the corresponding daily record data of each subfield is obtained by these subfields, in this way,The daily record monitoring system can obtain the field parsing data of daily record data.
Step 103 is calculated using the parsing data, so as to which the monitoring for obtaining weighing goal systems operation conditions refers toMark.
In the step, the daily record monitors system after the parsing data of the daily record data are obtained, it is possible to use instituteIt states parsing data to be calculated, the monitor control index of goal systems is calculated, the target is weighed with the monitor control indexThe operation conditions of system.
Wherein, monitor control index can be the index of the operation conditions and monitored condition that represent goal systems, for example transaction is totalAmount, Fail Transaction amount, merchandise successfully amount, success rate of merchandising, transaction time, the various dimensions index of abnormal number etc. of merchandising.Citing comesSay there are many application programs in business event IT system, the primary calling between these application programs, for a transaction, instituteIt can be call number total between each application program in certain time period to state transaction total amount, and the Fail Transaction amount can beThe number of total malloc failure malloc in the period, the successfully amount of merchandising can be that calling total in the period is successfulNumber, it is described transaction success rate can be within the period merchandise successfully amount with merchandise total amount ratio, the transactionIt can be that the average of transaction in the period takes to take, and the abnormal number of transaction can be that the period interior transaction occursAbnormal stroke count.
In this way, by using the monitor control index that is calculated of field parsing data, reaction that can be very accurate and fineGo out operating status and health status of goal systems etc., operation maintenance personnel facilitated the work such as to be monitored and safeguard to goal systems,Mitigate the operating pressure of operation maintenance personnel, and then reduce goal systems because monitoring the probability for causing to break down not in time, to reduceIt is lost caused by the system failure.
In the embodiment of the present invention, above-mentioned daily record monitors system, can be put down for developing and running the backstage of processing dataPlatform etc. is realized in the cluster formed in a large amount of computers and carries out Distributed Calculation to mass data, it is preferred that the daily record monitoringSystem is big data platform.
Above-mentioned daily record monitors system, can be applied to the big data application of financial system, medical system and educational system etc.Scene, such as the monitoring of bank data system, hospital data system and school's data system.
In daily record monitoring method provided in an embodiment of the present invention, the daily record data in goal systems is acquired；To the daily recordData are parsed, and obtain the parsing data of the daily record data；It is calculated using the parsing data, so as to be weighedThe monitor control index of goal systems operation conditions.In this way, referred to by the parsing data of daily record data to calculate the monitoring of goal systemsMark, the calculating of monitor control index can be very accurate and fine, so as to accurately reflect the operating status of goal systems andHealth status etc. facilitates operation maintenance personnel the work such as to be monitored and safeguard to goal systems, mitigates the operating pressure of operation maintenance personnel,And then goal systems is reduced because monitoring the probability for causing to break down not in time, it is lost caused by the system failure to reduce.
Referring to Fig. 2, Fig. 2 is the flow chart for the daily record monitoring method that another embodiment of the present invention provides.The method applicationSystem is monitored in daily record, as shown in Fig. 2, the described method comprises the following steps：
Daily record data in step 201, acquisition goal systems.
Step 202 parses the daily record data, obtains the parsing data of the daily record data.
Step 203 is calculated using the parsing data, so as to which the monitoring for obtaining weighing goal systems operation conditions refers toMark.
The monitor control index that step 204, display are calculated.
In the step, after the monitor control index is obtained by calculation in the daily record monitoring system, it can control in target positionPut or display interface in, the monitor control index being calculated is shown, so that operation maintenance personnel is checked.
Wherein, step 101 the retouching to step 103 that the description of step 201 to step 203 is referred in above-described embodimentIt states, this will not be repeated here.
Optionally, after step 201, the method includes：
Store the collected daily record data.
In the step, the daily record monitors system after the daily record data is collected, and can control the daily record numberAccording to storage to corresponding target location.
Wherein, the collected daily record data is stored, can be that the daily record number is stored using distributed memory systemAccording to, such as distributed post subscribes to message system kafka or reusable enterprise messaging systems RabbitMQ and stores daily record,Wherein can preferred kafka, kafka receives the daily record that comes of logstash acquisitions, stores.
Wherein, the daily record data parsed in step 202 can be from the collected daily record data of above-mentioned storageIt is obtaining or obtain from the daily record data acquired in real time.Preferably, in present embodiment, the daily record number of parsingAccording to being obtained from the collected daily record data of above-mentioned storage.
Optionally, after step 202, the method includes：
Store the parsing data obtained later to daily record data parsing.
In the step, the daily record monitoring system parses the daily record data, after obtaining the parsing data, instituteStating daily record monitoring system can control obtained parsing data storage to corresponding target location.
Wherein, the parsing data are stored, can store the parsing number using search engine ElasticSearchAccording to.Parsing data of the search engine ElasticSearch in the present embodiment for being obtained after storing and resolving use,ElasticSearch is a search engine in itself, the present invention is used as data warehouse, it can be to being stored thereinParsing data index, and support to search for data therein, so the parsing data deposit after parsing is wherein, so that it mayTo be scanned for and queried according to keyword.
Optionally, after step 203, the method includes：
Store the monitor control index being calculated.
In the step, after the monitor control index that the daily record monitoring system is calculated, it can control the prisonIndex storage is controlled to corresponding target location.
Wherein, the monitor control index can be stored in distributed data base.Preferably, in present embodiment, be byThe monitor control index is stored in distributed, expansible time series databases OpenTSDB.
Wherein, OpenTSDB is a kind of specialized database for storing time series data, because the monitor control index is with dayThe generation of will data and calculate, daily record generates sequentially in time, so monitor control index also can be a series ofTherefore monitor control index, can be stored in OpenTSDB by time series data.
Wherein, the monitor control index shown in step 204 can be obtained from the monitor control index of above-mentioned storageOr directly acquire from the monitor control index being calculated.
Optionally, please referring also to Fig. 3, Fig. 3 is the flow chart shown in Fig. 2 that monitor control index is calculated, such as Fig. 3 institutesShow, step 203 includes：
Step 2031, the first parsing data for obtaining the first daily record data in object time window in real time, obtain the first solutionAnalyse data acquisition system.
In this step, the daily record monitoring system needs in real time monitor the service conditions of goal systems in real time,The daily record monitoring system can obtain the daily record data in the goal systems in real time, so as to obtain a preset targetFirst parsing data of real-time first daily record data in time window, to obtain the first parsing data acquisition system.
Preferably, in present embodiment, the first parsing data are obtained, is obtained from the parsing data of storage.
It wherein, the termination time point of the object time window, can be with obtaining for the first time point for parsing data in real timeIt is identical.For example, as shown in Figure 4, Fig. 4 is the schematic diagram of object time window obtained in real time, such as in time shaft t,At the 0 current moment, in the object time window T of acquisition first parsing data when, the termination time point of object time window TBe 0 moment, and the start time point of object time window T can be 0 moment before a preset time point, such as -1 whenIt carves, wherein, the time span of object time window T can be set according to different monitoring requirements.Preferably, the time of TSpan can be 30 seconds.
Step 2032 parses data acquisition system based on described first, calculates the first monitor control index of the goal systems.
In the step, after daily record monitoring system gets the first parsing data acquisition system, the daily record monitoringSystem can be calculated according to the first parsing data in the described first parsing data acquisition system to parse data using first,So as to obtain the first monitor control index of the goal systems.
Wherein, the first monitor control index is the real-time condition monitoring index of the goal systems.
In this way, goal systems can be monitored in real time, with the operating whether monitoring objective system is normal and healthy.But during due to real time data acquisition, data need to expend certain time during acquisition and parsing etc., lead to the data havingThere may be certain delay so that the data of the first parsing data acquisition system are imperfect, have certain missing, but comparatively, numberWill not be very big according to missing amount, so the accuracy of the first monitor control index after calculating is unlikely to too low, it can be used as target temporarilyThe evaluation of system operation situation.
Step 2033, the first preset time after the first parsing data acquisition system is obtained, from history log dataMiddle the second parsing data acquisition system for obtaining the second daily record data in the first shadow time window, wherein, the object time windowIt is identical with the start time point of the first shadow time window and termination time point.
In view of above-mentioned, real-time first daily record data in the first parsing data acquisition system in the object time window of acquisitionThe first parsing data it is possible that with certain shortage of data, it is imperfect, in order to more accurately react the operation of goal systemsSituation can be modified the first monitor control index, with the situation of accurate display target system.
Therefore, in this step, daily record monitoring system is after the first parsing data acquisition system is got, can be withAfter one section of duration is spaced, i.e., the preset time after the first parsing data acquisition system is obtained, the daily record monitors systemData extraction can be carried out from historical data, so as to parse number to obtain second of the second daily record data in a shadow windowAccording to set.
Wherein, the start time point of the shadow window and termination time point, with obtaining the first parsing data setDuring conjunction, the start time point of the object time window is identical with time point is terminated, that is to say, that the shadow window representedPeriod, with the object time window when obtaining the when being used for calculating first monitor control index first parsing data acquisition systemThe period that oral thermometer shows is identical.
For example, referring to Fig. 5, Fig. 5 be object time window and the corresponding schematic diagram of shadow time window, such as whenOn countershaft t, obtain first parsing data when, object time window T represent 3 o'clock sharps to 3 points 01 minute, i.e., obtained in 3: 01 timesharingIt takes first in object time window T to parse data, as the time pushes ahead, after being spaced 14 points, i.e., is obtained in 3: 15 timesharingSecond in the first shadow time window T1 is taken to parse data, at this point, the first shadow time window for object time window TThe second parsing data in T1, it is corresponding to remain 3 o'clock sharps to 3 points of 01 minute this periods and obtained for 3 o'clock sharps to 3: 01The second parsing data in time segment.In addition, first in new object time window T ' can also be obtained in 3: 15 timesharingData are parsed, object time window T ' represents that 3: 14 assign at 3 points 15 minutes.That is when object time window T ' can be with the first shadowBetween window T1 calculate simultaneously.Optionally, object time window can be carried out at the same time with shadow time window based on scheduled frequencyIt calculates, for example, every 30 seconds calculate once, and object time window and the time span of shadow time window can be set to be more than or equal toThe time interval of the frequency i.e. 30 second are so that all data can be calculated.
Step 2034 parses data acquisition system based on described second, calculates the second monitor control index of the goal systems.
In the step, after daily record monitoring system gets the second parsing data acquisition system, the daily record prisonExamining system can according to acquisition it is described second parsing data acquisition system in second parsing data, come use the second parsing data intoRow calculates, so as to obtain the second monitor control index of the goal systems.
Wherein, due to being historical data, so being parsed in data acquisition system using second that the shadow time window obtainsDaily record data second parsing data, be complete data in this period, so the second monitor control index being calculatedIt is also accurate.
Step 2035 corrects first monitor control index using second monitor control index.
In the step, after daily record monitoring system gets second monitor control index, due to the described second prisonControlling index is calculated by history log data, and the data of acquisition are complete, so second monitor control index is compared to describedThe accuracy of first monitor control index is more preferable, so daily record monitoring system can use second monitor control index to described theOne monitor control index is modified, so as to more accurately embody the operating condition of the goal systems.
Wherein, first monitor control index is corrected using second monitor control index, can directly use described secondMonitor control index covers first monitor control index or based on preset algorithm, and the factors such as binding time and data pass throughSecond monitor control index and first monitor control index obtain a comprehensive monitor control index, and modified mode can not be done sternlyLattice limit.
In this way, using the mode of shadow window, by obtaining the first parsing data acquisition system in object time window in real timeAnd real-time first monitor control index is calculated, to be monitored in real time to the operating status of goal systems, obtaining the first solutionThe preset time after data acquisition system is analysed, by obtaining the shadow identical with the period of object time window in history log dataSecond parsing data acquisition system in sub- time window, and the second monitor control index is calculated, correct first using the second monitor control indexMonitor control index, since historical data is more complete, so the second monitor control index recalculated is more accurately and reliably, and during targetBetween window and shadow time window be two individual time windows, and between the two every preset time can be shorter, such as onlyDelay time of the daily record monitoring system to processes such as the acquisitions and parsing of daily record data need to be more than, so obtaining the first monitoringThe time difference of index and the second monitor control index is little, corrects the first monitor control index using the second monitor control index, can approximately waitThe first monitor control index is modified in real time, to ensure the accuracy being subsequently monitored to goal systems, furthermore targetThe time span of time window and shadow time window is all smaller, comparatively data acquisition, parsing and calculate speed compared withSoon, the time is shorter, and using the mode calculated respectively, can further reduce the calculation amount of data, save and calculate cost, carryRise system performance.
Optionally, step 2035 includes：
First monitor control index is covered using second monitor control index.
In the step, after daily record monitoring system gets second monitor control index, the daily record monitors systemFirst monitor control index can be directly covered using second monitor control index, it may also be said to be to delete first prisonAfter control, first monitor control index is replaced using the described second prison index.
Optionally, after step 2035, step 203 includes：
" the k-th presumed time after the kth parsing data acquisition system is obtained, from history log data is performed successivelyThe kth+1 for obtaining corresponding+1 daily record data of kth in kth shadow time window parses data acquisition system, wherein, the object timeThe start time point of window and the kth shadow time window and to terminate time point identical ", " data are parsed based on the kth+1Set calculates+1 monitor control index of kth of the goal systems " and " correct kth monitoring using+1 monitor control index of kth to refer toThe step of mark "；Until k reaches preset quantity N, stop cycle calculations, wherein, N is the integer more than or equal to 2, k be more thanOr the integer equal to 2 and less than or equal to N.
The monitor control index calculated by the corresponding parsing data acquisition system of shadow window carries out first monitor control indexIt corrects, can also be the monitor control index calculated using the corresponding parsing data acquisition system of multiple shadow windows to refer to the first monitoringMark is modified, to improve the modified accuracy of monitor control index.
So after second monitor control index is used to be modified first monitor control index, the daily record monitoringSystem can also be spaced one section of duration, i.e., after the second preset time after obtaining the second parsing data acquisition system, from going throughContinue data extraction in history data, so as to obtain the third solution in the second shadow window corresponding to corresponding daily record dataAnalyse data acquisition system；Then, the daily record monitoring system can parse the third solution in data acquisition system according to the third of acquisitionData are analysed, the third monitor control index of the goal systems is calculated；Then, the daily record monitoring system can use instituteIt states third monitor control index to be modified the second monitor control index, is finally reached and the monitor control index of the goal systems is modifiedPurpose；Correspondingly, after being modified to second monitor control index, the daily record monitoring system can also continue to obtainingAfter third preset time after the third parsing data acquisition system, continue to obtain from historical data corresponding in third shadow windowDaily record data corresponding to the 4th parsing data acquisition system；Then, the daily record monitoring system can be according to the 4th of acquisition the solutionData are analysed, the 4th monitor control index of the goal systems is calculated；Then, the daily record monitoring system can use instituteIt states the 4th monitor control index to be modified third monitor control index, is finally reached and the monitor control index of the goal systems is modifiedPurpose.
And so on, that is to say, that in this step, daily record monitoring system can successively be performed by controlling "The k-th presumed time after the kth parsing data acquisition system is obtained, kth shadow time window is obtained from history log dataThe kth+1 of interior+1 daily record data of corresponding kth parses data acquisition system, wherein, when the object time window is with the kth shadowBetween window start time point with terminate time point it is identical ", " based on the kth+1 parse data acquisition system, calculate the target systemThe step of+1 monitor control index of kth of system " and " correcting kth monitor control index using+1 monitor control index of kth ", to be recycledIt calculates k times, parses data, and data are parsed by the K+1 ,+1 monitor control index of kth is calculated to obtain kth+1, use+ 1 monitor control index of kth corrects kth monitor control index, reaches calculated using the corresponding parsing data acquisition system of multiple shadow windows successivelyMonitor control index monitor control index before is modified, improve the accuracy of monitoring；But to avoid increasing the daily recordThe computation burden of monitoring system is gone down without unconfined cycle calculations, so, until k reaches preset quantity N, that is, followAfter ring calculates n times, the daily record monitoring system can stop cycle calculations.
Wherein, N is the integer more than or equal to 2, and k is the integer more than or equal to 2 and less than or equal to N.
Wherein, the object time window and the start time point of the kth shadow time window and termination time point phaseTogether.
In this way, can multiple history logs be obtained using multiple discontinuous shadow windows by the cycle of above-mentioned stepsThe parsing data of data, so as to which multiple monitor control indexes be calculated, before being covered using the monitor control index value being calculatedThe index value of calculating realizes the modified accuracy of index.
Preferably, the time span of k discontinuous shadow windows is identical, and between k discontinuous shadow windowsInterval time can be identical, however, it is not limited to this, in other embodiments, between k discontinuous shadow windowsInterval time can also be different.
In daily record monitoring method provided in an embodiment of the present invention, the daily record data in goal systems is acquired；To the daily recordData are parsed, and obtain the parsing data of the daily record data；It is calculated using the parsing data, so as to be weighedThe monitor control index of goal systems operation conditions；Show the monitor control index being calculated.In this way, the parsing for passing through daily record dataData calculate the monitor control index of goal systems, and the calculating of monitor control index can be very accurate and fine, so as to accurateOperating status and health status of goal systems etc. is reflected, the monitor control index of storage can also be shown to operation maintenance personnel, sideJust operation maintenance personnel such as is monitored goal systems and safeguards at the work, mitigates the operating pressure of operation maintenance personnel, and then reduce targetSystem is lost because monitoring the probability for causing to break down not in time, to reduce caused by the system failure.
Referring to Fig. 6 to Fig. 9, Fig. 6 is one of structure chart of daily record monitoring system that one embodiment of the invention provides, and Fig. 7 isTwo, Fig. 8 of the structure chart of daily record monitoring system that one embodiment of the invention provides is the structure of monitor control index computing module in Fig. 6One of figure, Fig. 9 are two of the structure chart of monitor control index computing module in Fig. 6.As shown in fig. 6, daily record monitoring system 600 includes：
Log acquisition module 610, for acquiring the daily record data in goal systems.
Wherein, the log acquisition module 610 can be by data processing pipeline, such as servers' data processing tubeRoad logstash carrys out collector journal for log collection, polymerization and the system of transmission Flume.The operation system of goal systemsIt can be deployed on the server of hundreds and thousands of, the daily record that operational process generates can be also stored on these servers, in order to receiveCollect these daily records, a logstash can be disposed on each server, these logstash multiply 24 hours uninterruptedly 7Ground is run, and operation system once has daily record generation, will be collected by logstash.In addition, logstash can also be acquisitionDaily record data deposit distributed memory system high-throughput such as kafka distributed post subscribe to message system orIn RabbitMQ message queues.
Log analyzing module 620 for being parsed to the daily record data, obtains the parsing number of the daily record dataAccording to.
Specifically, log analyzing module 620 can parse the daily record data according to different fields, so as toObtain the field parsing data of the daily record data.
Wherein, the daily record data according to according to different fields is parsed, can is according in the daily record dataField, such as fields such as date, time, daily record rank, log information in the daily record data extract the daily record respectivelyThe field divided in data, to be parsed to daily record data.
Wherein, the task of daily record parsing can above-mentioned logstash be controlled to complete by log analyzing module 620, itDaily record data is read on one side, is parsed on one side.
Monitor control index computing module 630, for being calculated using the parsing data, so as to obtain weighing goal systemsThe monitor control index of operation conditions.
Wherein, the monitor control index computing module 620 can be comprising computing engines, for the data after parsing intoRow calculates, so as to obtain the monitor control index of measurement business IT system operation conditions, such as transaction total amount, Fail Transaction amount, transactionThe various dimensions index of success amount, transaction success rate, transaction time, the abnormal number of transaction etc..For example, in business event IT systemThere are many application programs, the primary calling between these application programs, for a transaction, the transaction total amount can be a certainCall number total between each application program in period, the Fail Transaction amount can be calling total in the periodThe number of failure, described merchandise that successfully to measure be the successful number of calling total in the period, the transaction success rateCan be the ratio of merchandise successfully amount and the transaction total amount within the period, the transaction time can be the periodThe average of interior transaction takes, and the abnormal number of transaction can be that abnormal stroke count occurs in transaction in the period.
Specifically, above-mentioned calculating generally using computing engines component, can also use Spark Streaming components,A kind of Storm (real-time computing engines) components or Flink components are calculated.Further, it is calculated in the monitor control index of daily recordThe computational methods (referring to attached drawing 3 and the corresponding embodiments of attached drawing 8-9) of shadow time window are used in the process, for calculating targetData in time window T, obtain monitor control index as a result, to improve the accuracy of monitor control index calculating, above-mentioned different dimensionsThe monitor control index calculating of index can be calculated using shadow window.
Optionally, as shown in fig. 7, daily record monitoring system 600 further includes：
Daily record memory module 640, for storing the collected daily record data of the log acquisition module.
Wherein, distributed memory system may be used in the daily record memory module 640, such as distributed post subscribes to messageSystem kafka or reusable enterprise messaging systems RabbitMQ stores daily record, wherein can preferred kafka, kafka connectsThe daily record that logstash acquisitions come is received, is stored, for module services such as subsequent parsing, calculating.
Data memory module 650 is parsed, after storing the log analyzing module 620 to daily record data parsingThe obtained parsing data.
Wherein, the parsing data memory module 650 can store the solution using search engine ElasticSearchAnalyse data.Memory modules of the search engine ElasticSearch as the data after parsing in this architectural schemes uses,ElasticSearch is a search engine in itself, the present invention is used as data warehouse, it can be to being stored thereinData index, and support to search for data therein, so the data deposit after parsing is wherein, it is possible to according to passKeyword is scanned for and be queried to daily record.
Specifically, log analyzing module 620 can control logstash to read daily record data from kafka on one side, solve on one sideAnalysis, in the data deposit ElasticSearch parsed after parsing.
Monitor control index memory module 660 refers to for storing the monitoring that the monitor control index computing module is calculatedMark.
Wherein, the monitor control index memory module 660 can be distributed data base.Preferably, in present embodiment, instituteIt can be in time series databases OpenTSDB to state monitor control index memory module 660.
Wherein, OpenTSDB is a kind of specialized database for storing time series data, because the monitor control index is with dayThe generation of will data and calculate, daily record generates sequentially in time, so monitor control index also can be a series ofTherefore monitor control index, can be stored in OpenTSDB by time series data.
Optionally, as shown in fig. 7, daily record monitoring system 600 further includes：
Monitor control index display module 670, for showing the monitoring that the monitor control index computing module 630 is calculatedIndex.Specifically, monitor control index display module can be a kind of data visualisation system.Calculate monitor control index purpose be in order toDisplaying so that operation maintenance personnel can see the operation conditions of goal systems in real time.
Wherein, the monitor control index that monitor control index display module 670 is shown can store mould from the monitor control indexIt is being obtained in the monitor control index stored in block 660 or be calculated from the monitor control index computing module 630It is directly acquired in the monitor control index.
Optionally, as shown in figure 8, the monitor control index computing module 630 includes：
First acquisition unit 631, for obtaining first of the first daily record data in object time window the parsing data in real time,Obtain the first parsing data acquisition system.
First computing unit 632, for being based on the first parsing data acquisition system, the first monitoring for calculating the goal systems refers toMark.
Second acquisition unit 633, for the first preset time after the first parsing data acquisition system is obtained, from going throughThe second parsing data acquisition system of the second daily record data in the first shadow time window is obtained in history daily record data, wherein, the meshStart time point and the termination time point for marking time window and the first shadow time window are identical.
Second computing unit 634 for being based on the second parsing data acquisition system, calculates the second prison of the goal systemsControl index.
First amending unit 635, for second monitor control index to be used to correct first monitor control index.
Optionally, first amending unit 635 is also used for the second monitor control index covering first monitoringIndex.
In this way, monitor control index computing module can use the mode of shadow window, by obtaining object time window in real timeReal-time first monitor control index is calculated in interior first parsing data set merging, come in real time to the operating status of goal systems intoRow monitoring, the preset time after the first parsing data acquisition system is obtained, by when acquisition is with target in history log dataBetween window period identical shadow time window in the second parsing data acquisition system, and the second monitor control index is calculated, usesSecond monitor control index corrects the first monitor control index, since historical data is more complete, so the second monitor control index recalculatedMore accurately and reliably, and object time window and shadow time window are two individual time windows, and between the two everyPreset time can be shorter, such as need to only be more than daily record and monitor system to the acquisition of daily record data and the delay time of parsing process, so the time difference for obtaining the first monitor control index and the second monitor control index is little, first is corrected using the second monitor control indexMonitor control index can approximately be equal to and the first monitor control index is modified in real time, to ensure subsequently to carry out goal systemsThe accuracy of monitoring, furthermore the time span of object time window and shadow time window is all smaller, and comparatively data are adoptedCollection, parsing and the speed calculated, the time is shorter, and using the mode calculated respectively, can further reduce dataCalculation amount saves and calculates cost, lifting system performance.
Optionally, as shown in figure 9, the monitor control index computing module 630 further includes：
Execution unit 636 is recycled, for performing successively " in the k-th presumed after obtaining the kth parsing data acquisition systemBetween, the kth+1 that corresponding+1 daily record data of kth in kth shadow time window is obtained from history log data parses data setIt closes, wherein, the object time window is identical with the start time point of the kth shadow time window and termination time point "," data acquisition system being parsed based on the kth+1, calculate+1 monitor control index of kth of the goal systems " and " use the kth+1Monitor control index corrects kth monitor control index " the step of；
Stop unit 637 for reaching preset quantity N up to k, stops cycle calculations, wherein, N is more than or equal to 2Integer, k be more than or equal to 2 and less than or equal to N integer.
In this way, the parsing data of multiple history log datas can be obtained by multiple discontinuous shadow windows, so as toMultiple monitor control indexes are calculated, the index value that the monitor control index value being calculated covering can be used to calculate before, realization refers toMark modified accuracy.
The daily record monitoring system 600 of the invention in real time that example provides can realize that daily record monitors system in above method embodimentThe each process realized, repeats, which is not described herein again to avoid.
The daily record monitoring system of the invention in real time that example provides, acquires the daily record data in goal systems；To the daily record numberAccording to being parsed, the parsing data of the daily record data are obtained；It is calculated using the parsing data, so as to obtain weighing meshThe monitor control index of mark system operation conditions.In this way, the monitor control index of goal systems is calculated by the parsing data of daily record data,The calculating of monitor control index can be very accurate and fine, so as to accurately reflect the operating status of goal systems and healthSituation etc. facilitates operation maintenance personnel the work such as to be monitored and safeguard to goal systems, mitigates the operating pressure of operation maintenance personnel, and thenGoal systems is reduced because monitoring the probability for causing to break down not in time, is lost caused by the system failure to reduce.
Example in real time of the invention also provides a kind of computer readable storage medium, and meter is stored on computer readable storage mediumCalculation machine program, the computer program realize each process of above-mentioned daily record monitoring method embodiment when being executed by processor, andIdentical technique effect can be reached, repeated to avoid, which is not described herein again.Wherein, the computer readable storage medium, such asRead-only memory (Read-Only Memory, abbreviation ROM), random access memory (Random Access Memory, abbreviationRAM), magnetic disc or CD etc..
The embodiment of the present invention is described above in conjunction with attached drawing, but the invention is not limited in above-mentioned specificEmbodiment, above-mentioned specific embodiment is only schematical rather than restricted, those of ordinary skill in the artUnder the enlightenment of the present invention, present inventive concept and scope of the claimed protection are not being departed from, can also made very muchForm is belonged within the protection of the present invention.