CN113420032A - Classification storage method and device for logs - Google Patents

Abstract

The application provides a classified storage method and a classified storage device for logs, wherein the method comprises the following steps: classifying the collected logs according to key fields in the logs; and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database. According to the technical scheme, different types of logs are respectively sent to corresponding log processing programs for parallel processing, so that the efficiency of storing the logs into the database can be improved; in addition, different types of logs are respectively put in storage, and the readability and the operability of the whole log system can be improved.

Description

Classification storage method and device for logs

Technical Field

The present application relates to the field of information technologies, and in particular, to a method and an apparatus for storing logs in a classified manner, a computer device, and a computer-readable storage medium.

Background

At present, the service of data synchronization through the security isolation gatekeeper is in diversified form development, and the data exchange requirements are increasingly raised, such as video data exchange, WEB service data exchange and the like, and the security isolation gatekeeper has the typical characteristics of large data exchange amount and timely and efficient requirements. How to check the integrity of data transmitted through a network gate becomes important for constructing a complete log auditing system, so that the log information related to the service is ensured to be kept in the system, and the requirements of efficient and real-time recording are met.

However, the service throughput of the gatekeeper is increased day by day, the log quantity of various services and systems is also multiplied, and the time consumed for storing and warehousing a large amount of logs is increased more and more.

Therefore, in order to meet the auditing requirement of the current real-time service, it is necessary to provide a scheme for efficiently and quickly storing the logs into the database, so that the method can be applied to a network gate and other scenes with a large amount of log storage requirements.

Disclosure of Invention

The application aims to provide a log classified storage method and device, computer equipment and a computer readable storage medium, which are used for improving the efficiency of storing logs into a database, so that the log classified storage method and device are suitable for occasions with a large number of log storage requirements, such as a gatekeeper.

One aspect of the embodiments of the present application provides a method for storing logs in a classified manner, including:

classifying the collected logs according to key fields in the logs;

and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database.

Optionally, the different types of logs specifically include: at least one of a log of a service type, a log of an alarm type, and a log of a kernel type; and

the database includes at least one of the following data tables: the system comprises a service log data table for storing service type logs, an alarm log data table for storing alarm type logs and a kernel log data table for storing kernel type logs.

Optionally, when the log is a service type log, sending the log to a service log processing program for processing and then storing the processed log into the service log data table; the service type logs are divided into a plurality of subcategories, and the service log data tables are multiple and respectively correspond to each subcategory of the service type logs;

the method for processing the log by the service log processing program comprises the following steps: after the logs with correct formats are screened out by utilizing a regular matching technology, inserting the screened logs into a pre-constructed annular buffer queue;

reading logs one by one from the circular buffer queue;

aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category;

and aiming at each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.

Optionally, after the regular matching technology is used to screen out the logs with the correct format, the screened logs are inserted into a pre-constructed circular buffer queue, which specifically includes:

acquiring a log data block with a set size through a block analyzer, performing message format check on the acquired log data block, discarding unconventional log information in the log data block, and splitting the log data block into log message information with minimum information length;

distributing the split log message information of each row to a plurality of parallel running row resolvers through the block resolvers, and carrying out further rule check by the row resolvers:

and checking whether the information of a plurality of necessary fields in the input log message information of each line is in compliance or not through the line analyzer, further discarding the log message information which is not in compliance, and inserting the log message information which is in compliance into the ring buffer queue.

Optionally, the checking, by the line parser, whether information of a plurality of necessary fields in the input log message information of each line is compliant specifically includes:

the method comprises the steps that sub-categories of log message information input within a period of time are divided through a line analyzer; and aiming at the divided log message information of each sub-category, carrying out batch rule check on the log message information of the sub-category by using an SQL statement block corresponding to the sub-category.

Optionally, when the log is an alarm type log, sending the alarm type log to an alarm log processing program for processing and then storing the processed alarm type log into the alarm log data table; the method for processing the log by the alarm log processing program comprises the following steps:

the alarm log processing program compares the information abstract of the currently input log with the information abstract of each log stored in the first hash table; if the comparison result is inconsistent, storing the currently input log into a first hash table and a first cache queue; otherwise:

further comparing the status identification bits of the two logs with the same information abstract; if the status identification bits of the log and the hash table are different, storing the currently input log into a first cache queue, and updating the status identification bit of the corresponding log in a first hash table according to the status identification bit of the currently input log;

if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a first cache queue, and updating the timestamp of the corresponding log in a first hash table according to the timestamp of the currently input log;

and storing the log in the first cache queue into an alarm log data table in the database.

Optionally, when the log is a kernel-type log, sending the log to a kernel log processing program for processing and then storing the processed log into the kernel log data table; the method for processing the logs by the kernel log processing program comprises the following steps:

comparing the protocol information of the currently input log with the protocol information of each log stored in the second hash table; if the comparison result is inconsistent, storing the log into a second hash table and a second cache queue; otherwise:

further comparing the status identification bits of two logs with the same protocol information; if the status identification bits of the log and the hash table are different, storing the log into a second cache queue, and updating the status identification bit of the corresponding log in a second hash table according to the currently input status identification bit of the log;

if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a second cache queue, and updating the timestamp of the corresponding log in a second hash table according to the timestamp of the currently input log;

and storing the logs in the second cache queue to a kernel log data table in the database.

An aspect of an embodiment of the present application further provides a log classification storage apparatus, including:

the log collection module is used for collecting logs and classifying the collected logs according to key fields in the logs;

and the log processing modules correspond to the classified logs of different types respectively and are used for processing the logs of the corresponding types and storing the processed logs into the database.

An aspect of the embodiments of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the log classification storage method.

An aspect of the embodiments of the present application further provides a computer-readable storage medium, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the log classification storage method.

According to the classified storage method, the computer equipment and the computer readable storage medium of the log, the collected log is classified according to key fields in the log; and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database. Therefore, different types of logs are respectively sent to corresponding log processing programs for parallel processing, the efficiency of storing the logs into a database can be improved, and the method and the device are suitable for scenes with a large number of log storage requirements, such as a gatekeeper. In addition, different types of logs are respectively put in storage, and the readability and the operability of the whole log system can be improved.

Preferably, when the logs with correct formats are screened, the block analyzer and the line analyzer are adopted for screening the logs, the block analysis processing speed is high, the information source can be quickly obtained for preliminary screening, the line analysis granularity is accurate, the information source can be bound to multiple cores for calculation processing, the log screening efficiency is integrally improved, and the log warehousing efficiency can be further improved.

Preferably, the chunk parser is a production thread and the line parser is a plurality of consumption threads. The design mode utilizes the characteristics of a multi-core CPU system and shares the analysis calculation to each core, thereby improving the analysis efficiency, and also improving the efficiency of log screening and the efficiency of log storage.

Preferably, the line parser can perform centralized parsing, checking and buffering on the same sub-categories accumulated within a period of time, and avoid switching SQL sentences back and forth, so that the efficiency of screening logs by the line parser is greatly improved, and the efficiency of warehousing logs can also be improved.

In addition, the logs of different sub-categories are stored in the database in a table mode, and the logs of the same sub-category are stored in the same data table, so that the readability and the operability of the whole log system are improved.

Preferably, when the alarm/kernel logs are processed, the similar alarm/kernel logs can be merged according to a state edge triggering mode due to the comparison of information summary/protocol information and state identification bits; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.

Drawings

FIG. 1 is a schematic diagram illustrating a classified storage device of logs applied to a security isolation gatekeeper according to an embodiment of the present invention;

FIG. 2 is a block diagram schematically illustrating the internal structure of a sorting storage apparatus for implementing a log according to the present invention;

FIG. 3 is a flow chart of a method for processing a log by a service log processing program according to a first embodiment of the invention;

fig. 4 is a schematic diagram illustrating an internal structure of a service log processing module according to a first embodiment of the present invention;

FIG. 5 is a flow chart of a method for processing a log by an alarm log handler according to a second embodiment of the invention;

FIG. 6 is a flow chart of a method for processing logs by a kernel log handler according to a third embodiment of the present invention;

fig. 7 is a schematic diagram of a hardware architecture of a computer device suitable for implementing the classified storage method of the log according to a fourth embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the descriptions relating to "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.

In the description of the present application, it should be understood that the numerical references before the steps do not identify the order of performing the steps, but merely serve to facilitate the description of the present application and to distinguish each step, and therefore should not be construed as limiting the present application.

The inventor of the application finds that different types of logs generally have different warehousing processing modes; logs of the same type often have approximately the same warehousing operation mode; therefore, in the technical scheme of the application, the collected logs are classified according to the key fields in the logs; and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database. Therefore, different types of logs are respectively sent to corresponding log processing programs for parallel processing, the efficiency of storing the logs into a database can be improved, and the method and the device are suitable for scenes with a large number of log storage requirements, such as a gatekeeper. In addition, different types of logs are respectively put in storage, and the readability and the operability of the whole log system can be improved.

Fig. 1 shows a schematic diagram of a log classification storage device installed in a security isolation gatekeeper.

In the security isolation gatekeeper, file traffic, database traffic, and proxy traffic generate logs. The logs are sent to a log collector syslog-ng in a classification storage device of the logs by means of a pipeline file, a local network and the like.

The security isolation gatekeeper usually includes three types of logs, which are respectively a service type log (referred to as a service log for short), an alarm type log (referred to as an alarm log for short) and a kernel type log (referred to as a kernel log for short). The service log and the alarm log are generated by each large service module of the security isolation gatekeeper independently, and the format of the service log and the alarm log is optimized by dividing the data into fields on the basis of the syslog format which accords with the standard. That is, the data source of the alarm log is the same as that of the service log, and the data source is the log generated by the file service, the database service and the proxy service.

The kernel log is a log generated by the kernel system. The kernel log is generally the contents of the system itself such as start, stop, process abnormal information, etc., and the corresponding log will be generated when the gatekeeper is attacked by the network. The log processing is uniformly delivered to a log collector syslog-ng in the classification storage device for log collection.

In the log classifying and storing device, the collected logs are classified accurately according to the fields, then distributed to different log processing programs, and simultaneously stored in different data tables in the database, so that high concurrent processing logic of the logs is realized.

The log processing programs corresponding to the service type log (service log for short), the alarm type log (alarm log for short) and the kernel type log (kernel log for short) are a service log processing program, an alarm log processing program and a kernel log processing program, respectively.

Correspondingly, the database may also include at least one of the following data tables: the system comprises a service log data table for storing service type logs, an alarm log data table for storing alarm type logs and a kernel log data table for storing kernel type logs.

The classification storage device for the log provided by the application can comprise: the system comprises a log collection module and a plurality of log processing modules;

the log collection module may include the log collector syslog-ng, which is configured to collect logs and classify the collected logs according to key fields in the logs. Wherein the key field may be a rank field or a module field; for example, the log collection module may identify a level field in the log, and if the level field of the log is a normal level, the log is classified as a service type log (service log for short); if the level field of the log is specifically the alarm level, the log is classified as an alarm type log (alarm log for short); if the level field of the log is the kernel level, the log is classified as a kernel-type log (referred to as a kernel log for short).

And the log processing modules correspond to the classified logs of different types respectively and are used for processing the logs of the corresponding types and storing the processed logs into the database.

The internal structure of the classification storage device for logs provided by the present application can be as shown in fig. 2, and includes the following modules: alog collection module 201, a servicelog processing module 202, an alarmlog processing module 203 and a kernellog processing module 204.

The servicelog processing module 202 corresponds to a service type log (referred to as a service log for short) and is configured to process the service log;

the alarmlog processing module 203 is corresponding to a log of alarm types (called alarm log for short) and is used for processing the alarm log;

the kernellog processing module 204 corresponds to a kernel-type log (referred to as a kernel log for short) and is configured to process the kernel log.

Various embodiments are provided below, and various embodiments provided below can be used to implement the above-described scheme for sorted storage of logs.

Example one

Fig. 3 schematically shows a flowchart of a method for processing a log by a service log processing program according to a first embodiment of the present application.

As shown in fig. 3, a method for processing a service log by a service log processing program according to a first embodiment of the present application may include the following steps:

step S301: and for the service type logs, screening out logs with correct formats by utilizing a regular matching technology, and then inserting the screened logs into an annular buffer queue.

Specifically, the service log processing program mainly processes a system log, a management log, a service log and a tracking log; in this step, the service log processing process may first screen out dirty logs, such as logs with empty key field content and incorrect field format, by using a regular matching technique, and truncate the ultralong log content, so as to facilitate warehousing. The logs with correct format are inserted into a ring buffer queue which is constructed in advance; when the ring buffer queue is full, subsequent logs will be discarded.

Preferably, in this step, a log data block with a set size, for example, a log data block with a size of 512KB, 1MB or 10MB, may be obtained through the block parser, and a syslog message format check is performed on the obtained log data block, and after non-compliant log information in the log data block is discarded, the log data block is split into log message information with a minimum information length in a row, that is, log message information in a row;

the block parser distributes the split log message information of each row to a plurality of parallel running row parsers, and the row parsers perform further rule check: the line analyzer checks whether the information of a plurality of necessary fields in the line of log message information is in compliance or not aiming at the input log message information of each line, discards the log message information which is not in compliance and inserts the log message information which is in compliance into the annular buffer queue;

the block parser and the line parser are implemented by using a syntax interpreter respectively, and functionally are the difference between coarse-grained detection and fine-grained detection. The advantage of adopting block analyzer and line analyzer lies in, the fast information source that can acquire fast of block analysis processing rate carries out preliminary screening, and line analysis granularity is accurate can bind to and carry out the calculation processing on the multicore to improve the efficiency of log screening on the whole, just also can improve the efficiency that the log was put in storage.

A memory queue is arranged between the block parser and the row parser, the memory queue is mainly used for information transmission among threads, the block parser is a production thread, and the row parser is a plurality of consumption threads. The design mode utilizes the characteristics of a multi-core CPU system and shares the analysis calculation to each core, thereby improving the analysis efficiency, and also improving the efficiency of log screening and the efficiency of log storage. The memory queue supports a single-production-multi-consumption multithread processing mode, and a plurality of consumption threads can equally acquire messages output by the generation threads. With the concept of thread pool, the number of threads of the line parser can be configured in the initial loading stage of the program.

In general, the logs of traffic types can be divided into various sub-categories, for example, distinguished by the sub-categories of the traffic logs: web module logs, flow tracking logs, file tracking logs, database tracking logs and general service logs;

as a more preferable embodiment, the line parser classifies the message information of the log input in a period of time into sub-categories; the row parser maintains an SQL (Structured Query Language) statement block corresponding to each sub-type log for parsing and checking the sub-type log; and the row parser performs batch rule check on the log message information of each sub-category by using the SQL statement block corresponding to the sub-category aiming at the batch log message information of each sub-category. The line analyzer can perform centralized analysis, inspection and buffering on the same sub-categories accumulated in a period of time, and avoids switching SQL sentences back and forth, so that the efficiency of screening logs by the line analyzer is greatly improved, and the efficiency of warehousing the logs can also be improved.

Step S302: and reading logs from the circular buffer queue one by one.

The service log processing program can also read logs from the ring buffer queue one by one and use the read logs for warehousing.

The ring buffer queue is used for communication among multiple processes, and a multi-producer-multi-reader model can be realized by a memory sharing method; for example, in addition to reading the logs from the ring buffer queue during warehousing, the logs can also be read from the ring buffer queue for log snapshot, that is, real-time log information in the ring buffer queue is synchronized to each log snapshot. The log snapshot has the function of displaying the latest log information in real time, and has the advantages that the log snapshot is in a read-only mode, and the operation of acquiring the log by a single log snapshot does not block the log to be put in storage. The log snapshot can be used for the CLI viewing of a background manager and also can be used for the WebUI log audit viewing.

Step S303: and aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category.

Step S304: and aiming at each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.

Specifically, a plurality of service log data tables in the database respectively correspond to each subcategory of the service log; in this step, for each sub-category, when it is determined that the number of logs in the log storage queue corresponding to the sub-category reaches the set number index, the service log processing program inserts the logs in the log storage queue into the service log data table corresponding to the sub-category in the database in batches, and clears the log storage queue.

The logs of different sub-categories are stored in the database in a table mode, and the logs of the same sub-category are stored in the same data table, so that the readability and the operability of the whole log system are improved.

Corresponding to the method for processing the service log by the service log processing program, a servicelog processing module 202 provided in the first embodiment of the present application is specifically configured to, for a service-type log, filter out a log with a correct format by using a regular matching technique, and then insert the filtered log into a pre-constructed circular buffer queue; reading logs one by one from the circular buffer queue; aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category; and aiming at each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.

An internal structure of a servicelog processing module 202 provided in an embodiment of the present application is shown in fig. 4, and may include the following units: a log screening unit 401 and alog storage unit 402;

the log screening unit 401 is configured to, for logs of a service type, screen out logs with a correct format by using a regular matching technology, and then insert the screened logs into a pre-constructed circular buffer queue; preferably, the log filtering unit 401 may include the above block parser and line parser.

Thelog warehousing unit 402 is configured to read logs from the circular buffer queue one by one; aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category; and for each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.

According to the technical scheme, the block analyzer and the line analyzer are adopted for screening the logs, the block analysis processing speed is high, the information source can be rapidly acquired for preliminary screening, the line analysis granularity can be accurately bound to multiple cores for calculation processing, and therefore the efficiency of log screening is integrally improved, and the efficiency of log storage can be further improved.

Preferably, the chunk parser is a production thread and the line parser is a plurality of consumption threads. The design mode utilizes the characteristics of a multi-core CPU system and shares the analysis calculation to each core, thereby improving the analysis efficiency, and also improving the efficiency of log screening and the efficiency of log storage.

Preferably, the line parser can perform centralized parsing, checking and buffering on the same sub-categories accumulated within a period of time, and avoid switching SQL sentences back and forth, so that the efficiency of screening logs by the line parser is greatly improved, and the efficiency of warehousing logs can also be improved.

In addition, the logs of different sub-categories are stored in the database in a table mode, and the logs of the same sub-category are stored in the same data table, so that the readability and the operability of the whole log system are improved.

Example two

The second embodiment of the present application describes a scheme for processing an alarm log by an alarm log processing program.

Alarm logs are typically divided into eight subtypes: virus alarms, attack alarms, hardware exceptions, system exceptions, resource exceptions, configuration changes, log alarms, and policy alarms. Each seed type alarm log is individually identified by a type identification bit, and in addition, the status identification bit identifies the change of the state of the alarm log, such as a resource abnormal alarm log, taking a cpu alarm log as an example, the state identification bit is high (high)/medium (mid)/low (low); for example, the system abnormal alarm log takes a network card alarm log as an example, and the status flag bit is fault (fault)/recovery (recovery).

Fig. 5 is a flowchart schematically illustrating a method for processing a log by an alarm log processing program according to a second embodiment of the present application.

As shown in fig. 5, the method for processing an alarm log by an alarm log processing program according to the second embodiment of the present application may include the following steps:

and S501, screening the logs with the correct format by utilizing a regular matching technology for the logs with the alarm types.

In this step, the alarm log processing program screens out the log with the correct format by using the regular matching technology for the log of the alarm type, and the method for screening out the dirty log may be the same as the method for screening out the log in step S301 in fig. 3, and is not described here again.

Because the alarm log can be matched with an outgoing program to send alarm contents to users in time, such as short messages, mailboxes and the like, a plurality of alarm logs with the same contents can be sent within a period of time. Therefore, in order to improve the warehousing efficiency of the logs, the following steps are adopted to merge the screened logs with correct formats so as to greatly reduce the number of logs needing to be warehoused.

Step S502: the alarm log processing program compares the information abstract of the currently input log with the information abstract of each log stored in the first hash table; if the comparison result is inconsistent, executing the following step S503 to store the currently input log into the first hash table and the first cache queue; otherwise, the following step S504 is executed;

specifically, the msg (information) field of the currently input alarm log with the correct format may be hashed to extract a fixed-length information digest, where the information digest is a unique field of the alarm log. The digest has a function of being index information and compression information.

Comparing the information abstract of the alarm log with the correct format with the information abstract of each log stored in the hash table; if there is no log with the same information digest as the currently input alarm log in the hash table, executing the following step S503 to store the currently input log in the first hash table and the first cache queue; otherwise, the following step S504 is performed.

Step S503: and storing the currently input log into a first hash table and a first cache queue.

In this step, the currently input log is stored in the first hash table, and the currently input log is also stored in the first buffer queue to be put in storage.

Step S504: further comparing the status identification bits of the two logs with the same information abstract; if the status flag bits of the first hash table and the second hash table are different, executing step S505 to store the currently input log into the first cache queue, and updating the status flag bit of the corresponding log in the first hash table according to the status flag bit of the currently input log; if the status flag bits are the same, step S506 is executed.

Specifically, if there is a log with the same information digest as the currently input alarm log in the hash table, the status flag bits of the two logs with the same information digest are continuously compared in this step: extracting a state identification bit from an msg field of a currently input alarm log, comparing the state identification bit with the state identification bit of the log with the same information abstract, and comparing whether the state identification bit is changed and is changed from low to high or from fault to receiver; if the change occurs, the following step S505 is executed to store the currently input log into the first cache queue, and update the status flag of the log with the same information digest in the first hash table according to the status flag of the currently input log.

Step S505: storing the currently input log into a first cache queue, and updating a state identification bit of the corresponding log in a first hash table according to the state identification bit of the currently input log;

in this step, the currently input log is stored in a first cache queue to be put in storage, and the state identification bit of the log with the information abstract same as that of the currently input log in the first hash table is updated according to the state identification bit of the currently input log.

Step S506: further comparing the timestamps of the two logs; if the difference between the two timestamps is greater than the set value, executing step S507 to store the log into the first cache queue, and updating the timestamp of the corresponding log in the first hash table according to the timestamp of the currently input log; otherwise, go to step S508;

specifically, if a log with the same information abstract and unchanged identification bit as the information abstract of the currently input alarm log exists in the hash table, comparing the time stamp of the log with the time stamp of the currently input alarm log; if the difference between the two timestamps is greater than the set value, that is, the insertion time of the two timestamps exceeds the time length of the timer, step S507 is executed to store the log into the first cache queue, and the timestamp of the corresponding log in the first hash table is updated according to the timestamp of the currently input log.

Step S507: storing the currently input log into a first cache queue, and updating a timestamp of the corresponding log in a first hash table according to the timestamp of the currently input log;

in the step, the currently input log is stored in a first cache queue to prepare for storage, and the timestamp of the log which is the same as the summary of the log information and has unchanged state identification bits in a first hash table is updated according to the timestamp of the currently input log.

Step S508: the currently entered log is discarded.

In the step, merging processing of repeated logs is carried out; through the multiple judgments in the above steps, in this step, the logs which have the same information summary and the same state identification bit as the currently input logs and the time stamp difference within the set value range are determined to be stored in the first hash table, and then the currently input logs are discarded as the repeated logs, so that the repeated warehousing operation of multiple logs with the same content is avoided.

Step S509: and storing the log in the first cache queue into an alarm log data table in the database.

The log of the first buffer queue can be put into the above ring buffer queue as a snapshot while the database insertion operation is executed, and the principle is the same as that of the service log.

Therefore, by comparing the information abstract and the state identification bits, the similar alarm logs can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.

Corresponding to the method for processing the alarm log by the alarm log processing program, the alarmlog processing module 203 provided in the second embodiment of the present application is specifically configured to compare the information digest of the currently input log with the information digests of the logs stored in the first hash table; if the comparison result is inconsistent, storing the log into a first hash table and a first cache queue; otherwise: further comparing the status identification bits of the two logs with the same information abstract; if the status identification bits of the log and the hash table are different, storing the log into a first cache queue, and updating the status identification bit of the corresponding log in a first hash table according to the currently input status identification bit of the log; if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a first cache queue, and updating the timestamp of the corresponding log in a first hash table according to the timestamp of the currently input log; and storing the log in the first cache queue into an alarm log data table in the database.

In the technical scheme of the second embodiment of the application, because the information abstract is compared with the state identification bits, the similar alarm logs can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.

In addition, the alarm type logs are stored in the alarm log data table in the database and are distinguished from the data tables stored in the service logs and the kernel logs, so that the readability and the operability of the whole log system are improved.

EXAMPLE III

Fig. 6 schematically shows a flowchart of a method for processing a log by a kernel log handler according to the second embodiment of the present application.

The processing method of the kernel log is similar to the alarm log, and as shown in fig. 6, the method for processing the kernel log by the kernel log processing program in the third embodiment of the present application may include the following steps:

step S601, the regular matching technology is utilized to screen out the logs with correct formats for the logs with the kernel types.

In this step, the kernel log processing program screens out the logs of the kernel type by using the regular matching technology, and the method for screening out the dirty logs may be the same as the method for screening out the logs in step S301 in fig. 3, and is not described here again.

Further, aiming at the screened kernel log with the correct format, processing is carried out according to the following steps:

step S602: the kernel log processing program compares the protocol information of the currently input log with the correct format with the protocol information of each log stored in the second hash table; if the comparison result is inconsistent, executing the following step S603 to store the currently input log into a second hash table and a second cache queue; otherwise, the following step S604 is executed;

step S603: and storing the currently input log into a second hash table and a second buffer queue.

In this step, the currently input log is stored in the second hash table, and the currently input log is also stored in the second buffer queue to be ready for storage.

Step S604: further comparing the status identification bits of the two logs with the same information abstract; if the status flag bits of the two are different, step S605 is executed to store the currently input log into the second cache queue, and the status flag bit of the corresponding log in the second hash table is updated according to the status flag bit of the currently input log; if the status flag bits are the same, step S606 is executed.

Step S605: storing the currently input log into a second cache queue, and updating a state identification bit of the corresponding log in a second hash table according to the state identification bit of the currently input log;

in this step, the currently input log is stored in a second cache queue to be put in storage, and the state identification bit of the log with the information abstract same as that of the currently input log in the second hash table is updated according to the state identification bit of the currently input log.

Step S606: further comparing the timestamps of the two logs; if the difference between the two timestamps is greater than the set value, step S607 is executed to store the log into the second cache queue, and the timestamp of the corresponding log in the second hash table is updated according to the timestamp of the currently input log; otherwise, go to step S608;

step S607: storing the currently input log into a second cache queue, and updating the time stamp of the corresponding log in a second hash table according to the time stamp of the currently input log;

step S608: the currently entered log is discarded.

In this step, the logs with the same protocol information and the same state identification bit as the currently input logs and the timestamp difference within the set value range are determined to be stored in the second hash table, and then the currently input logs are discarded as the repeated logs, so that repeated warehousing operation of a plurality of logs with the same content is avoided.

Step S609: and storing the log in the second cache queue to an alarm log data table in the database.

The log of the second cache queue can be put into a ring buffer as a snapshot to be displayed while the database insertion operation is executed, and the principle of the method is the same as that of the service log.

Therefore, by comparing the protocol information with the state identification bits, the similar kernel logs can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.

Corresponding to the method for processing the kernel log by the kernel log processing program, a kernellog processing module 204 provided in the third embodiment of the present application is specifically configured to compare protocol information of a currently input log with protocol information of each log stored in a second hash table; if the comparison result is inconsistent, storing the log into a second hash table and a second cache queue; otherwise: further comparing the status identification bits of two logs with the same protocol information; if the status identification bits of the log and the hash table are different, storing the log into a second cache queue, and updating the status identification bit of the corresponding log in a second hash table according to the currently input status identification bit of the log; if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a second cache queue, and updating the timestamp of the corresponding log in a second hash table according to the timestamp of the currently input log; and storing the logs in the cache queue to a kernel log data table in the database.

In the third technical solution of the embodiment of the present application, because of the comparison between the protocol information and the state identification bits, the kernel logs of the same type can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.

In addition, the logs of the kernel type are stored in the kernel log data table in the database and are distinguished from the data tables stored by the service logs and the alarm logs, so the readability and the operability of the whole log system are improved.

Example four

Fig. 7 schematically shows a hardware architecture diagram of a computer device 1000 adapted to implement the classified storage method of the log according to the fourth embodiment of the present application. In an exemplary embodiment of the present application, the computer device 1000 may be a device capable of automatically performing numerical calculation and/or information processing according to instructions set or stored in advance. For example, the server may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers), a gateway, and the like. As shown in fig. 7, the computer device 1000 includes at least, but is not limited to: the memory 1010, processor 1020, and network interface 1030 may be communicatively linked to each other via a system bus. Wherein:

the memory 1010 includes at least one type of computer-readable storage medium including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 1010 may be an internal storage module of the computer device 1000, such as a hard disk or a memory of the computer device 1000. In other embodiments, the memory 1010 may be an external storage device of the computer device 1000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 1000. Of course, the memory 1010 may also include both internal and external memory modules of the computer device 1000. In this embodiment, the memory 1010 is generally used for storing an operating system installed in the computer apparatus 1000 and various types of application software, such as program codes of a method for identifying a behavior subject of the software. In addition, the memory 1010 may also be used to temporarily store various types of data that have been output or are to be output.

Processor 1020 may be, in some embodiments, a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data Processing chip. The processor 1020 is generally configured to control the overall operation of the computer device 1000, such as performing control and processing related to data interaction or communication with the computer device 1000. In this embodiment, the processor 1020 is configured to execute program codes stored in the memory 1010 or process data.

The network interface 1030 may comprise a wireless network interface or a wired network interface, with the network interface 1030 typically being used to establish communications links between the computer device 1000 and other computer devices. For example, the network interface 1030 is used to connect the computer apparatus 1000 to an external terminal via a network, establish a data transmission channel and a communication link between the computer apparatus 1000 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), or Wi-Fi.

It should be noted that FIG. 7 only shows a computer device having components 1010 and 1030, but it should be understood that not all of the shown components are required and that more or fewer components may be implemented instead.

In this embodiment, the method for identifying the behavior entity of the software stored in the memory 1010 can be further divided into one or more program modules and executed by one or more processors (in this embodiment, the processor 1020) to implement the embodiments of the present application.

EXAMPLE five

The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of identifying a subject of a software behavior in embodiments.

In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device. Of course, the computer-readable storage medium may also include both internal and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various types of application software installed in a computer device, for example, the program code of the method for identifying the behavior body of the software in the embodiment, and the like. Further, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

It will be apparent to those skilled in the art that the modules or steps of the embodiments of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different from that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.

The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.

Claims (11)

Translated fromChinese

1.一种日志的分类存储方法，其特征在于，包括：1. a classification storage method of log, is characterized in that, comprises:将收集的日志，根据日志中的关键字段进行分类；Classify the collected logs according to key fields in the logs;将分类出的不同类型的日志分别发送到对应的日志处理程序并行处理后，存入数据库。The classified logs of different types are respectively sent to the corresponding log processing programs for parallel processing, and then stored in the database.2.根据权利要求1所述的方法，其特征在于，所述不同类型的日志，具体包括：业务类型的日志、告警类型的日志以及内核类型的日志中的至少一种；以及2. The method according to claim 1, wherein the different types of logs specifically include: at least one of a service type log, an alarm type log, and a kernel type log; and所述数据库包括至少如下一种数据表：用于存储业务类型的日志的业务日志数据表、用于存储告警类型的日志的告警日志数据表、用于存储内核类型的日志的内核日志数据表。The database includes at least one of the following data tables: a service log data table for storing logs of a service type, an alarm log data table for storing logs of an alarm type, and a kernel log data table for storing logs of a kernel type.3.根据权利要求2所述的方法，其特征在于，当日志为业务类型的日志时，将其发送至业务日志处理程序进行处理后存入所述业务日志数据表；其中，所述业务类型的日志划分为多种子类别，以及所述业务日志数据表为多个，分别对应业务类型的日志的各子类别；3. The method according to claim 2, wherein when the log is a business type log, it is sent to a business log processing program for processing and stored in the business log data table; wherein, the business type The log is divided into multiple subcategories, and the business log data table is multiple, corresponding to each subcategory of the log of the business type;其中，所述业务日志处理程序处理日志的方法包括：利用正则匹配技术筛选出格式正确的日志后，将筛选出的日志插入到预先构建的环形缓冲队列中；Wherein, the method for processing logs by the business log processing program includes: after using regular matching technology to filter out logs in a correct format, inserting the filtered logs into a pre-built ring buffer queue;从所述环形缓冲队列中逐条读取日志；Read logs one by one from the ring buffer queue;针对每条当前读取的日志，识别该日志的子类别，并根据识别的子类别，将该日志缓存于对应于该子类别的日志入库队列中；For each currently read log, identify the subcategory of the log, and according to the identified subcategory, cache the log in the log storage queue corresponding to the subcategory;针对每个子类别，当对应该子类别的日志入库队列中，日志的数量达到设定数量指标，则将该日志入库队列中的日志批量插入到所述数据库中的、对应于该子类别的业务日志数据表，并清空该日志入库队列。For each sub-category, when the number of logs in the log storage queue corresponding to the sub-category reaches the set quantity index, the logs in the log storage queue are inserted into the database in batches corresponding to the sub-category. the business log data table, and clear the log storage queue.4.根据权利要求3所述的方法，其特征在于，所述利用正则匹配技术筛选出格式正确的日志后，将筛选出的日志插入到预先构建的环形缓冲队列，具体包括：4. method according to claim 3, is characterized in that, after described utilizing regular matching technology to screen out the log of correct format, insert the log that screened out into the pre-built ring buffer queue, specifically comprises:通过块解析器获取设定大小的日志数据块，并对获取的日志数据块进行报文格式检查，将所述日志数据块中不合规的日志信息丢弃后，将所述日志数据块拆分成以行为最小信息长度的日志报文信息；Obtain log data blocks of a set size through a block parser, check the packet format of the obtained log data blocks, discard the non-compliant log information in the log data blocks, and then split the log data blocks The log message information of the minimum message length can be generated;通过所述块解析器将拆分的各行日志报文信息分发给多个并行运行的行解析器，由行解析器进行进一步规则检查：Through the block parser, the split log message information of each line is distributed to multiple line parsers running in parallel, and the line parsers perform further rule checking:通过所述行解析器，检查输入的每行日志报文信息中的若干必要字段的信息是否合规，进而将不合规的日志报文信息丢弃，将合规的日志报文信息插入到所述环形缓冲队列。Through the line parser, check whether the information of several necessary fields in the input log message information of each line is compliant, and then discard the non-compliant log message information, and insert the compliant log message information into all log message information. The ring buffer queue described above.5.根据权利要求4所述的方法，其特征在于，所述通过所述行解析器，检查输入的每行日志报文信息中的若干必要字段的信息是否合规，具体包括：5. The method according to claim 4, wherein, by the line parser, checking whether the information of several necessary fields in the input log message information of each line is compliant, specifically comprising:通过所述行解析器将一段时间内输入的日志报文信息进行子类别的划分；针对划分的每个子类别的批量的日志报文信息，使用与该子类别对应的SQL语句块对该子类别的日志报文信息进行批量规则检查。The log message information input in a period of time is divided into sub-categories by the line parser; for the batch log message information of each sub-category, the SQL statement block corresponding to the sub-category is used for the sub-category. The log packet information is checked by batch rules.6.根据权利要求2所述的方法，其特征在于，当日志为告警类型的日志时，将其发送至告警日志处理程序进行处理后存入所述告警日志数据表；其中，所述告警日志处理程序处理日志的方法包括：6. The method according to claim 2, wherein when the log is an alarm type log, the log is sent to an alarm log processing program for processing and then stored in the alarm log data table; wherein, the alarm log The handlers for processing logs include:将当前输入的日志的信息摘要，与第一哈希表中所存的各日志的信息摘要进行对比；若对比结果为不一致，则将当前输入的日志存入第一哈希表，以及第一缓存队列；否则：Compare the information digest of the currently input log with the information digest of each log stored in the first hash table; if the comparison result is inconsistent, store the currently input log in the first hash table and the first cache queue; otherwise:进一步对比信息摘要相同的两个日志的状态标识位；若两者的状态标识位不同，则将当前输入的日志存入第一缓存队列，并根据当前输入的日志的状态标识位更新第一哈希表中对应日志的状态标识位；Further compare the status flags of the two logs with the same information digest; if the status flags of the two are different, the currently input log is stored in the first cache queue, and the first log is updated according to the status flag of the currently input log. The status flag of the corresponding log in the table;若两者的状态标识位相同，则进一步对比所述两个日志的时间戳；若两者的时间戳之差大于设定值，则将该日志存入第一缓存队列，并根据当前输入的日志的时间戳更新第一哈希表中对应日志的时间戳；If the status flags of the two are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the two is greater than the set value, the log is stored in the first cache queue, and the log is stored according to the current input The timestamp of the log updates the timestamp of the corresponding log in the first hash table;将第一缓存队列中的日志存储到所述数据库中的告警日志数据表。The logs in the first cache queue are stored in the alarm log data table in the database.7.根据权利要求2所述的方法，其特征在于，当日志为内核类型的日志时，将其发送至内核日志处理程序进行处理后存入所述内核日志数据表；其中，所述内核日志处理程序处理日志的方法包括：7. The method according to claim 2, wherein when the log is a kernel type log, it is sent to a kernel log processing program for processing and stored in the kernel log data table; wherein, the kernel log The handlers for processing logs include:将当前输入的日志的协议信息，与第二哈希表中所存的各日志的协议信息进行对比；若对比结果为不一致，则将该日志存入第二哈希表，以及第二缓存队列；否则：Compare the protocol information of the currently input log with the protocol information of each log stored in the second hash table; if the comparison result is inconsistent, store the log in the second hash table and the second cache queue; otherwise:进一步对比协议信息相同的两个日志的状态标识位；若两者的状态标识位不同，则将该日志存入第二缓存队列，并根据当前输入的日志的状态标识位更新第二哈希表中对应日志的状态标识位；Further compare the status identification bits of the two logs with the same protocol information; if the status identification bits of the two are different, the log is stored in the second cache queue, and the second hash table is updated according to the status identification bits of the currently input log. The status flag of the corresponding log in ;若两者的状态标识位相同，则进一步对比所述两个日志的时间戳；若两者的时间戳之差大于设定值，则将该日志存入第二缓存队列，并根据当前输入的日志的时间戳更新第二哈希表中对应日志的时间戳；If the status flags of the two are the same, the time stamps of the two logs are further compared; if the difference between the time stamps of the two is greater than the set value, the log is stored in the second cache queue, and according to the current input The timestamp of the log updates the timestamp of the corresponding log in the second hash table;将第二缓存队列中的日志存储到所述数据库中的内核日志数据表。The logs in the second cache queue are stored in the kernel log data table in the database.8.一种日志的分类存储装置，其特征在于，包括：8. A classification storage device for logs, characterized in that, comprising:日志收集模块，用于收集日志，并将收集的日志，根据日志中的关键字段进行分类；The log collection module is used to collect logs and classify the collected logs according to the key fields in the logs;多个日志处理模块，分别对应分类出的各不同类型的日志，用于将对应类型的日志进行处理后存入数据库。A plurality of log processing modules respectively correspond to the classified logs of different types, and are used to process the logs of the corresponding types and store them in the database.9.根据权利要求8所述的装置，其特征在于，所述不同类型的日志，具体包括：业务类型的日志、告警类型的日志以及内核类型的日志中的至少一种；以及9. The apparatus according to claim 8, wherein the different types of logs specifically include: at least one of a service type log, an alarm type log, and a kernel type log; and所述数据库包括至少如下一种数据表：用于存储业务类型的日志的业务日志数据表、用于存储告警类型的日志的告警日志数据表、用于存储内核类型的日志的内核日志数据表。The database includes at least one of the following data tables: a service log data table for storing logs of a service type, an alarm log data table for storing logs of an alarm type, and a kernel log data table for storing logs of a kernel type.10.一种计算机设备，包括存储器、处理器以及存储在存储器上并可在处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时用于实现权利要求1～7中任一项所述的日志的分类存储方法的步骤。10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement claims 1 to 7 when the processor executes the computer program The steps of the method for classifying and storing logs according to any one of the above.11.一种计算机可读存储介质，其特征在于，所述计算机可读存储介质内存储有计算机程序，所述计算机程序可被至少一个处理器所执行，以使所述至少一个处理器执行权利要求1～7中任一项所述的日志的分类存储方法的步骤。11. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program can be executed by at least one processor, so that the at least one processor executes the rights The steps of the method for classifying and storing logs according to any one of requirements 1 to 7 are required.

Images