Disclosure of Invention
Aiming at the defects in the prior art, the invention provides an exception handling method, a system, a terminal and a medium based on log analysis, which can realize automatic exception data handling by combining with logs and improve the timeliness of system operation and maintenance monitoring.
In a first aspect, an exception handling method based on log analysis includes the steps of:
reading a log stored in a database;
analyzing the log according to a preset log template to obtain an analysis log;
extracting corresponding indexes in the analysis log according to preset log indexes;
and when the index abnormality is detected, alarming.
Preferably, the log includes a system log and an application log.
Preferably, the setting method of the log template comprises the following steps:
obtaining a log sample from an internal memory or network;
performing multi-row combination according to the head-of-line matching mode in the log sample to obtain a combination template;
extracting log indexes in the merging module through a regular expression to obtain an extraction template;
defining the extraction template as the log template.
Preferably, the categories of the log index include one or more combinations of:
application transaction classes, middleware classes, database classes, operating system classes, network classes, server classes, security device classes, and storage device classes.
Preferably, when the index abnormality is detected, the alerting specifically includes:
when the index extracted from the analysis log meets the preset alarm rule, alarming is carried out;
and acquiring the associated alarm associated with the alarm, and compressing the associated alarm.
Preferably, the method further comprises, after the alerting when the index anomaly is detected:
establishing a prediction model according to the linear relation between the index extracted from the analysis log and a preset standard index;
and carrying the analysis log into the prediction model, and predicting the index utilization rate of each time node according to time, wherein the index utilization rate of each time node exceeds the standard.
In a second aspect, an exception handling system based on log analysis includes:
the acquisition unit: for reading a log stored in a database;
an analysis unit: the method comprises the steps of analyzing the log according to a preset log template to obtain an analysis log;
extraction unit: extracting corresponding indexes in the analysis log according to preset log indexes;
and a processing unit: and the alarm is used for alarming when the index abnormality is detected.
In a third aspect, a terminal comprises a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory being interconnected, wherein the memory is adapted to store a computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method according to the first aspect.
In a fourth aspect, a computer readable storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of the first aspect.
According to the technical scheme, the exception handling method, the system, the terminal and the medium based on log analysis can analyze the log, realize automatic exception data handling by combining the log according to the exception condition of the operation and the maintenance of the log analysis system, and improve the timeliness of the operation and the maintenance monitoring of the system.
Detailed Description
Embodiments of the technical scheme of the present invention will be described in detail below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and thus are merely examples, and are not intended to limit the scope of the present invention. It is noted that unless otherwise indicated, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.
It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
Embodiment one:
an exception handling method based on log analysis, see fig. 1, comprises the following steps:
s1: reading a log stored in a database;
preferably, the log includes a system log and an application log. Specifically, the method may analyze a system log and an application log.
S2: analyzing the log according to a preset log template to obtain an analysis log;
preferably, the setting method of the log template comprises the following steps:
s11: obtaining a log sample from an internal memory or network;
s12: performing multi-row combination according to the head-of-line matching mode in the log sample to obtain a combination template;
s13: extracting log indexes in the merging module through a regular expression to obtain an extraction template;
s14: defining the extraction template as the log template.
Specifically, the method needs to determine a head-of-line matching mode in the log sample, and determine whether multi-line merging is needed.
S3: extracting corresponding indexes in the analysis log according to preset log indexes;
preferably, the categories of the log index include one or more combinations of:
application transaction classes, middleware classes, database classes, operating system classes, network classes, server classes, security device classes, and storage device classes.
Specifically, the application transaction classes mainly include time of transaction related logs, transaction operation codes, success-failure flags, transaction time-consuming, and the like. The middleware class mainly comprises time, log level, local IP, http transmission data volume, remote IP, request protocol, request mode, request URL and the like of logs such as Tomcat. Database classes mainly include log levels, instance names, hostnames, pid, etc. of database logs. The operating system class mainly comprises an operating system, virtualized related performance data, such as a CPU, a memory and the like; the network class mainly includes network-related data such as: source IP, destination IP, source port, destination port, protocol type, etc. The server class and the security device class mainly comprise hardware related performance data such as hardware temperature, SSL hardware performance and the like of the server, the security device and the like. Storage device classes mainly include storage related data such as: storage space usage, and so on.
S4: and when the index abnormality is detected, alarming.
Preferably, when the index abnormality is detected, the alerting specifically includes:
when the index extracted from the analysis log meets the preset alarm rule, alarming is carried out;
and acquiring the associated alarm associated with the alarm, and compressing the associated alarm.
Specifically, the alarm mainly takes care of the following points:
1) A rule engine;
the rule engine is mainly used for carrying out alarm management on unstructured data such as logs and the like, supporting defining alarm rules according to keyword search results, and determining whether to alarm and alarm level according to rule matching results.
2) Index management;
the index management is a precondition for setting a threshold, and includes, in addition to basic performance indexes, indexes aggregated based on basic data, such as the number of erroneous transactions in the last 5 minutes or the average transaction time of the last 5 minutes. The definition of the aggregate metrics is generated using a query interface provided by a query engine. Any statistical aggregate statement supported by the query engine can be stored as an index.
3) Threshold management;
the definition of the threshold depends on the index, the threshold management support sets different alarm values according to time periods, for example, the index of CPU utilization rate is set to be 0.8 for 8 am to 8 pm, and other time periods are 0.9, so that the user can conveniently and flexibly define the alarm threshold according to actual service conditions.
4) A scheduling engine;
after the alarm is started, the corresponding index or rule needs to be calculated in real time according to the time period when the alarm is defined, the index/rule of parallel real-time calculation which is required to be supported by the operation and maintenance big data platform reaches the millions, and the system needs to provide efficient dispatching and parallel real-time monitoring of the millions of indexes.
The framework of the alarm engine is realized based on AKKA Cluster, each started alarm management item is an Actor, the Actor is a lightweight parallel model, the weight of the Actor is lighter than that of threads, the rule of the alarm is required to be calculated, the index of the alarm, the threshold value of the alarm, the calculated period and other information are determined to be stored in the Actor when the Actor is created, each Actor is monitored and managed in real time, and when abnormality occurs, the Actor can be restarted or re-created automatically. A4 g virtual machine can easily create millions of level actors and support the improvement of the overall throughput of AKKA Cluster by adding nodes.
5) Alarm compression;
the actually running IT components have a close association relationship, besides the self-generated alarms, the anomalies of the bottom layer components can also cause the alarms of the upper layer components or services supported by the bottom layer components, if no alarms are compressed, alarm storms can be generated, the system needs to automatically identify the alarms of the bottommost layer according to the association relationship between the mastered and identified components, find out the associated alarms, compress the associated alarms, and maintain the association relationship between the components and store the association relationship in a Neo4j graph database.
6) Alarming and predicting;
depending on the feature model provided by the underlying analysis engine, intelligent prognosis can be provided for the overall operating condition of the system, and when the operation of the system does not conform to the model provided by the underlying analysis engine, alarm prediction can be performed.
7) Alarming by keywords;
after selection, the user can enter an index alarm interface, a fixed threshold configuration alarm of a certain index can be set, and the following diagram is set, namely, the threshold alarm of the maximum Duration index is set, and the situation that more than 3 times exceeds 300 seconds continuously occurs and is set as crisis alarm; setting a general alarm when more than 3 times of conditions exceeding 100 seconds continuously occur; setting up as an information alarm or the like more than 50 seconds for 3 times in succession.
According to the exception handling method based on log analysis, the log can be analyzed, automatic exception data processing is realized by combining the log according to the exception condition of the operation and maintenance of the log analysis system, and the timeliness of the operation and maintenance monitoring of the system is improved.
Preferably, the method further comprises, after the alerting when the index anomaly is detected:
establishing a prediction model according to the linear relation between the index extracted from the analysis log and a preset standard index;
and carrying the analysis log into the prediction model, and predicting the index utilization rate of each time node according to time, wherein the index utilization rate of each time node exceeds the standard.
Specifically, the capacity prediction is performed by analyzing the linear relationship between the indexes, creating a prediction model, and using a multiple linear regression model. Capacity prediction distinguishes between users, transactions, and workload of the system and business related metrics by capturing performance related data in existing systems. Using capacity prediction, the following results can be obtained:
1) Analyzing and predicting part of indexes according to time to predict the utilization rate of the indexes to reach a certain time node; how much time the index is predicted, when the point in time is.
2) Carrying out correlation analysis on part of important indexes and indexes, analyzing a mathematical formula between the two indexes, and estimating the utilization rate of the indexes according to total transaction amount of TPS and daily; the maximum transaction condition that the server can load can be estimated according to the index utilization rate.
Embodiment two:
an exception handling system based on log analysis, see fig. 2, comprising:
the acquisition unit: for reading a log stored in a database;
an analysis unit: the method comprises the steps of analyzing the log according to a preset log template to obtain an analysis log;
extraction unit: extracting corresponding indexes in the analysis log according to preset log indexes;
and a processing unit: and the alarm is used for alarming when the index abnormality is detected.
The exception handling system based on log analysis can analyze the log, and according to the exception condition of the operation and maintenance of the log analysis system, the automatic exception data processing is realized by combining the log, so that the timeliness of the operation and maintenance monitoring of the system is improved.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in this application, it should be understood that the disclosed system may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices, or elements, or may be an electrical, mechanical, or other form of connection.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present invention.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
For a brief description of the system provided by the embodiments of the present invention, reference may be made to the corresponding content in the foregoing method embodiments where the description of the embodiments is not mentioned.
Embodiment III:
a terminal, see fig. 3, comprising a processor 801, an input device 802, an output device 803 and a memory 804, the processor 801, the input device 802, the output device 803 and the memory 804 being interconnected by a bus 805, wherein the memory 804 is adapted to store a computer program comprising program instructions, the processor 801 being configured to invoke the program instructions to perform the method as described above.
It should be appreciated that in embodiments of the present invention, the processor 801 may be a central processing unit (Central Processing Unit, CPU) which may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSPs), application specific integrated circuits (Application Specific Integrated Circuit, ASICs), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The input device 802 may include a touch pad, a fingerprint sensor (for collecting fingerprint information of a user and direction information of a fingerprint), a microphone, etc., and the output device 803 may include a display (LCD, etc.), a speaker, etc.
The memory 804 may include read only memory and random access memory and provides instructions and data to the processor 801. A portion of the memory 804 may also include non-volatile random access memory. For example, the memory 804 may also store information of device type.
For a brief description, the terminal provided in the embodiment of the present invention may refer to the corresponding content in the foregoing method embodiment, where the embodiment section is not mentioned.
Embodiment four:
a computer readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method described above.
The computer readable storage medium may be an internal storage unit of the terminal according to any of the foregoing embodiments, for example, a hard disk or a memory of the terminal. The computer readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the terminal. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the terminal. The computer-readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
For a brief description, reference may be made to the corresponding contents of the foregoing method embodiments for the media provided in the embodiments of the present invention, where the description of the embodiments is not mentioned.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention, and are intended to be included within the scope of the appended claims and description.