Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements that are expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a log data processing method provided in an embodiment of the present invention, where the present embodiment is applicable to a case of differentially storing a storm log and a non-storm log in a hierarchical manner, and the method may be performed by a log data processing apparatus, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in an electronic device, where the electronic device may be a terminal device or a server device, so long as the log data processing method can be executed, and the embodiment of the present invention does not limit a specific device type of the electronic device. Accordingly, as shown in fig. 1, the method includes the following operations:
S110, acquiring a log to be stored.
The log to be stored can be collected by a log management platform and is required to be stored.
In the embodiment of the invention, the data processing and management flow of the log can be realized through the log management platform. Optionally, the log management platform may configure the collection module, and configure and collect log data of each device through the collection module, so as to obtain a log to be stored. Optionally, the log management platform may perform log management on one or more different devices, and the embodiment of the present invention does not limit the type of device that the log management platform interfaces with.
And S120, storing the log to be stored according to a text file under the condition that the log to be stored is determined to be a storm log.
The storm log may be a log type generated when a log storm problem occurs. Log storm refers to the phenomenon of continuous large amount of log generated under abnormal condition of the system. It will be appreciated that storm logs are typically large in data volume.
Correspondingly, after the log to be stored is obtained, the log management platform can judge the type of the log to be stored. If the log to be stored is determined to be a storm log, the storm log can be stored as a text file. The storage is carried out according to text files, namely, the journal to be stored is directly used for storing the original text of the storm journal at the operating system level in a text file mode. Because the storage compression ratio of the text file to the log data is higher and is usually 1:20, and the storage compression ratio of the common database to the log data is lower and is usually 1:2, storing storm logs with larger data volume in the text file can save the storage space of the log data and improve the writing data speed of the log data.
And S130, storing the log to be stored into a target database under the condition that the log to be stored is determined to be a non-storm log.
The non-storm log may be a log of a smaller amount of data that the device normally generates. The target database may be a database for storing log data, so long as log data can be stored, and the database type of the target database is not limited in the embodiment of the present invention.
Therefore, when the journal to be stored is identified as the storm journal, the journal to be stored is stored according to the text file, and when the journal to be stored is identified as the non-storm journal, the target database is used for carrying out conventional storage, so that the differentiated hierarchical management capability of the journal data is realized, diversified journal management requirements can be met by distinguishing the journal sources, distinguishing the characteristics in the journal sources and distinguishing the journal storage objects, and the storm journal and the non-storm journal can be effectively stored while the storage cost and the maintenance cost are reduced, so that the integrity of the normal journal data is ensured.
According to the embodiment of the invention, when the obtained log to be stored is determined to be the storm log, the log to be stored is stored according to the text file, and when the obtained log to be stored is determined to be the non-storm log, the log to be stored is stored to the target database, so that the problems of high log data maintenance cost, poor log management stability and the like in the existing storm log processing mode are solved, the log management stability can be improved, and the log maintenance cost is reduced.
Example two
Fig. 2 is a flowchart of a log data processing method according to a second embodiment of the present invention, where the present embodiment is implemented based on the foregoing embodiment, and in the present embodiment, various specific optional implementations for determining whether a log to be stored is a storm log are provided. Accordingly, as shown in fig. 2, the method of this embodiment may include:
S210, acquiring a log to be stored, and calculating quantization index data of the log to be stored.
The quantization index data may be data for quantizing a correlation index of the log to be stored.
Fig. 3 is a schematic flow chart of log processing by a log management platform according to a second embodiment of the present invention. In a specific example, as shown in fig. 3, after the log management platform collects the log to be stored through the collection module, the log to be stored may be sent to the preprocessing module. The preprocessing module can be used for extracting key information in the log to be stored, assigning corresponding management attributes and the like, and is used for subsequent log showing, storing, calculating and the like.
In the embodiment of the invention, the preprocessing module in the log management platform is mainly responsible for analyzing, labeling and enriching the processing content of other data of the log to be stored, so that the log to be stored completes related pre-processing before warehousing. The labeling process may be to distinguish and label storm logs. Other data enrichment processes may include, but are not limited to, supplementing the log to be stored with some additional attribute management fields, such as the manager of the IP (Internet Protocol, protocol interconnecting between networks) address, etc.
Fig. 4 is a schematic diagram of an internal processing flow of a preprocessing module in a log management platform according to a second embodiment of the present invention. In a specific example, as shown in fig. 4, the preprocessing module may complete the preprocessing process of the log to be stored through a Source thread, a Task queue, a Worker thread, a Sink queue, and a Sink thread. Specifically, source threads may be used to grab and push logs to be stored. The acquisition module in the log management platform can be configured with a plurality of collectors, and the collectors are responsible for acquiring logs. The Source thread can then dock each collector to obtain log data to be stored collected by each collector. Alternatively, one collector may configure multiple Source threads for log collection. For example, 10 Source threads may collect log data sent by one collector, and so on. The Task queue can be used for aggregating logs to be stored, which are collected by each Source thread. For example, assuming that one collector performs log collection corresponding to 10 Source threads, log data of one collector collected by a plurality of Source threads may be summarized into one Task queue. A Worker thread may then be a thread used in multi-threaded programming to perform background tasks or process parallel computations. In the embodiment of the present invention, an EPS (Event per Second) calculation process may be performed by a Worker thread in the preprocessing module to calculate quantization index data of a log to be stored. The EPS calculation may be calculating the data amount of the log number processed by the Worker thread per second. In addition, the Worker thread may perform summary content extraction operations to extract important character content in the log to be stored. Meanwhile, the workbench thread can also execute a subsequent storm word bank matching process so as to complete a storm log judging process. It will be appreciated that the workbench thread may also have other general processing functions, such as resolving IP addresses, log wrapping processing, and the like. Sink queues may be a component used for log data transmission and processing, the main function being to write log data to the corresponding system. For example, sink queues may be used to determine the storage path of the log to be stored, i.e. whether it is normally stored to a database or stored as a text file, etc. Correspondingly, the Sink thread can execute operations such as writing a database or writing a file, and the like, so as to finish the final storage process of the log to be stored.
S220, judging whether the quantization index data of the log to be stored meets the log storm judging condition, if so, executing S240, otherwise, executing S230.
The log storm judgment condition may be a condition for judging whether the log to be stored is a storm log according to the quantization index data of the log to be stored. Alternatively, the log storm judgment condition may include at least one judgment condition.
In the embodiment of the invention, the preprocessing module in the log management platform can judge whether the quantized index data of the log to be stored meets the log storm judgment condition according to the calculated quantized index data of the log to be stored so as to quickly identify whether the log storm phenomenon occurs.
In an optional embodiment of the present invention, the quantization index data may include a log obtaining number quantization index, a log processing time-consuming quantization index, a log processing writing quantization index, and a log average time-consuming index, and the determining that the quantization index data of the log data to be stored satisfies a log storm judgment condition may include determining that the quantization index data of the log data to be stored satisfies a log storm judgment condition in a case where it is determined that a ring-up rate of the log obtaining number quantization index exceeds a first threshold, a ring-up rate of the log processing time-consuming quantization index and the log processing writing quantization index exceeds a second threshold, and a ring-up rate of the log average time-consuming index exceeds a third threshold.
The log obtaining number quantization index may be an index of obtaining a number average value of log data per second, which is limited in a time window range. The log processing time-consuming quantization index may be a time-consuming mean value of preprocessing each log defined within a time window. The log processing writing quantization index may be defined within a time window, and the average time consumed for writing the processed log to the corresponding queue. The log average time consumption index may be a time window, and the average time consumption of each log when storing the log. The first threshold, the second threshold and the third threshold may be specific thresholds configured according to the log storm judgment requirement, and the values of the specific thresholds may be the same or different.
For example, as shown in fig. 4, the log obtaining number quantization index may be a number average of log data obtained from the task queue every second within a 10 second window, and may be referred to as 10s_avg_input (eps). The log processing time consumption quantization index may be a time consumption average value of preprocessing each log by the worker thread within a 10 second window, and the unit is microsecond/event, and may be called as 10s_avg_cost_queue (us/e). The log processing writing quantization index may be an average time spent by the worker thread in finishing writing to the sink queue within a 10 second window, and may be referred to as 10s_avg_cost_sink_queue (us/e). The slow time of sink queue processing becomes more time consuming and the value becomes larger, which is used to indicate that the congestion probability increases. The log average time consumption index may be the average time consumption of each log in the sink thread within a 10 second window, and may be referred to as 10s_avg_cost_sink (us/e). Optionally, the preprocessing module may collect and cache corresponding index data every 30 seconds in two processing links of the worker thread and the sink queue.
Correspondingly, after each piece of quantized index data is obtained, whether the quantized index data of the log data to be stored meets the log storm judgment condition can be further judged. Specifically, it may be determined whether the upward rate of change of the log obtaining number quantization index ring ratio exceeds a first threshold, and determine whether the upward rate of change of the log processing time-consuming quantization index ring ratio and the log processing writing quantization index ring ratio exceeds a second threshold, and simultaneously determine whether the upward rate of change of the log average time-consuming index ring ratio exceeds a third threshold. If the judgment conditions are met at the same time, namely, the upward change rate of the log acquisition number quantization index ring ratio exceeds a first threshold, the log processing time-consuming quantization index and the log processing writing quantization index ring ratio exceeds a second threshold, and the upward change rate of the log average time-consuming index ring ratio exceeds a third threshold, it can be determined that the quantization index data of the log data to be stored meets the log storm judgment conditions.
In a specific example, when the quantization index data of the log to be stored simultaneously satisfies the following conditions, it may be determined that a log storm is generated:
the condition 1 is that the number of input logs, namely log acquisition number quantitative indexes 10s_avg_input (eps), is more than 5 times of upward change rate of ring ratios;
The condition 2 is that the processing time consumption of the worker thread is met simultaneously, namely the log processing time consumption quantization index 10s_avg_cost_queue (us/e) and the log processing writing quantization index 10s_avg_cost_sink_queue (us/e) which is output to the target queue time consumption are more than 1 time in the upward change rate;
condition 3. The average log time consumption index of the target queue processing time consumption, 10s_avg_cost_sink (us/e), is more than 1 time the upward rate of change.
S230, determining that the log to be stored is a non-storm log, and storing the log to be stored into a target database.
Illustratively, as shown in FIG. 3, the log management platform may also include a routing module. The routing module can be used for configuring and processing the log data sent by the preprocessing module, and can distribute the log data of different labels to the corresponding storage units for differential storage. The database storage mode can use an unstructured search engine to store normal log data, namely, non-storm logs in a binary file mode.
S240, acquiring a target log field of the log to be stored.
The target log field may be a part of fields screened from the log to be stored, and may be used for referring to determine whether the log to be stored is a storm log.
Correspondingly, when the quantitative index data of the log data to be stored is determined to meet the log storm judgment condition, determining that the log storm occurs at the moment. In order to further screen the storm logs from the logs to be stored, the target log fields of each log to be stored can be obtained to carry out a subsequent storm log judgment flow.
In an optional embodiment of the present invention, the obtaining the target log field of the log to be stored may include determining a target reference field in the log to be stored, counting a total number of characters of the target reference field, and taking a first set number of target reference fields with the maximum total number of characters as the target log field.
The target reference field may be a part of fields screened from the log to be stored, and may be used to determine a target log field of the log to be stored. The first set number may be set according to actual requirements, for example, may be 2, 3, or 5, and the embodiment of the present invention does not limit the specific numerical value of the first set number.
Specifically, in order to obtain the target log field of the log to be stored, the target reference field which can be used as the basis for judging the storm log may be screened from the log to be stored first, and the total number of characters of each target reference field may be counted. It will be appreciated that the target reference fields may be different, and the total number of characters corresponding to the statistics may be the same or different. Further, a set number of target reference fields having the largest total number of characters may be used as the target log field, and for example, the first 3 target reference fields having the largest total number of characters may be used as the target log field.
In an alternative embodiment of the present invention, the determining the target reference field in the log to be stored may include replacing a time-date content of the log to be stored to obtain a fuzzy processing log, performing content segmentation on the fuzzy processing log according to a field to obtain an original word segmentation field, selecting a set number of alternative reference fields for the original word segmentation field, performing pattern recognition on the alternative reference fields, and deleting a pattern field in the alternative reference fields to obtain the target reference field.
The fuzzy processing log may be log data obtained by replacing time and date contents of the log to be stored. The original word segmentation field may be field data obtained by performing content segmentation on the fuzzy processing log according to the field. The second set number may be set according to actual requirements, for example, may be 10or 20, and the embodiment of the present invention does not limit the specific numerical value of the second set number. The alternative reference field may be a partial field screened from the original segmentation field. The mode field may be a field type having a low meaning of judging a storm log.
In the embodiment of the invention, when the target reference field in the log to be stored needs to be determined, the time and date content of the log to be stored can be replaced to obtain the fuzzy processing log. Further, the fuzzy processing log is subjected to content segmentation according to the fields, and each original word segmentation field included in the fuzzy processing log is obtained. After the original word segmentation field is obtained, a part of alternative reference fields with judgment reference values can be further selected from the original word segmentation field, and mode recognition is carried out on the alternative reference fields at the same time so as to delete the mode fields in the alternative reference fields, thereby obtaining the required target reference field.
In a specific example, the log to be stored generated in the log storm time period can be identified according to the matching rule of "time/date", the "time/date" content in each piece of log data to be stored is identified, and replaced by < DT >, and the following examples are:
Assume that a log to be stored is the following:
<190>Oct 0717:08:542809109150008447(root)4424361f Traffic@FLOW:SESSION:10.219.70.195:50630->220.181.7.165:80(TCP),interface ethernet0/2,vr trust-vr,policy 16,user-@-,host-,policy deny.
after the matching processing of the 'time/date' rule, the log to be stored can obtain the following fuzzy processing log:
<190><DT>2809109150008447(root)4424361f Traffic@FLOW:SESSION:10.219.70.195:50630->220.181.7.165:80(TCP),interface ethernet0/2,vr trust-vr,policy 16,user-@-,host-,policy deny.
Further, for each fuzzy processing log obtained by processing, content segmentation can be performed according to marks such as "blank", "<" > ", and the like, so that the content of the fuzzy processing log is segmented into word segmentation fields one by one. Illustratively, the obfuscation log described above may result in a list of original segmentation fields as shown in table 1.
Table 1 list of original segmentation fields
Further, after the "time/date" < DT > field is removed from the original word segmentation field, the first 10 fields are selected as alternative reference fields, so as to obtain an alternative reference field list as shown in table 2.
Table 2 list of alternative reference fields
Further, pattern recognition is carried out on the field content of the selected alternative reference field, the pattern fields such as the serial number, the IP address and the like with lower judging and reference significance on the storm logs are recognized and deleted, and then the target reference field in each storage log can be obtained.
Correspondingly, after the mode field of hit "mode identification" is removed from the alternative reference fields, the total number of characters included in other alternative reference fields with character content is statistically ranked, and the first three fields with the highest ranking are taken as target log fields, wherein the target log fields can violate the field list of "suspected storm content". Table 3 is a target reference field list obtained by performing statistical ranking on the total number of characters after the target reference fields are obtained by screening based on the above table 2. As shown in table 3, the top 2 fields, i.e., field5 and Field6, which are the highest ranking, are taken as target log fields based on the target reference Field list.
Table 3 list of target reference fields
| Sequence number | Field name | Field value | Statistics value |
| 4 | Field4 | 4424361f | 10 |
| 5 | Field5 | Traffic@FLOW: | 1000 |
| 6 | Field6 | SESSION: | 1000 |
| 9 | Field9 | interface | 10 |
| 10 | Field10 | ethernet0/2 | 10 |
| 11 | Field11 | vr | 1 |
S250, judging whether the target log field is matched with a preset storm word bank, if so, executing S260, otherwise, executing S270.
The preset storm word stock can be a pre-constructed word stock and can be used for matching field types in the log so as to judge whether the log is a storm log or not.
S260, determining the log to be stored as the storm log, and storing the log to be stored according to a text file.
S270, storing the target log field into the preset storm word bank, and storing the log to be stored into a target database.
Correspondingly, after the target log field of the log to be stored is obtained, the target log field can be matched with a preset storm word bank. If the target log field hits some fields in the preset storm word bank, the target log field is indicated to be matched with the preset storm word bank, and the target log field is determined to be a log storm keyword. At this time, the corresponding log to be stored may be marked with a storm feature, for example, a storm_flag=y tag is added to the log to be stored, and it is determined that the log is a storm log. Further, the log to be stored, which is determined as the storm log, may be stored as a text file. If the target log field does not hit some fields in the preset storm word bank, the target log field is not matched with the preset storm word bank, the target log field screened at the time can be stored in the preset storm word bank and used as a reference field for judging the storm log next time, but the batch of log data does not mark the label of the storm log and can be normally stored in the target database. It will be appreciated that the logging into the target database will not be adversely affected, since the data volume of the storm log that is first presented and identified will not typically be particularly large.
In a specific example, as shown in fig. 3, the log management platform may further include a routing module. Optionally, the routing module is responsible for configuring and processing the log data to be stored of different labels, so as to ensure that the log data to be stored is correctly distributed to the corresponding storage unit. Specifically, if the log to be stored includes a storm tag, such as storm_flag=y, a storm routing rule may be executed, routed to the operating system "text file" for storage, and stored as text files. If the to-be-stored log does not contain the storm label, the to-be-stored log is directly stored into the target database as a normal log.
In a specific example, as shown in fig. 3, the log management platform may further include a presentation module for presenting stored log data, including both normal log data stored in a conventional database store and storm log data stored in an operating system via text files.
According to the technical scheme, the rapid identification function of storm log content is newly added in the preprocessing module of the log management platform, the automatic discovery capability of the storm log content with high repetition rate is realized, meanwhile, the preset storm word bank is introduced in the preprocessing module, the accuracy rate of identifying the storm log is improved, and meanwhile, the preset storm word bank also has the automatic content updating function, so that the operation and maintenance workload can be reduced. The log management platform is provided with a data routing function, and the differentiated hierarchical management capability of log data is realized, wherein the log management platform comprises source distinguishing, source distinguishing characteristics and target storage object distinguishing. When the log storm occurs, the overall availability of the log management platform and the integrity of the log data can be kept at the normal availability level by adopting the log data processing of the log management platform, so that the processing and storage cost of the storm log is greatly reduced while the integrity of the log data is ensured.
Compared with a pause acquisition method and a pause/discard strategy method executed before warehouse entry provided by the existing log management platform, the embodiment of the invention fully summarizes the characteristic rules of the generation of the log data storm, and can automatically judge whether the log storm exists in the log management platform and which type of log data belongs to the storm log and which type of log data belongs to the normal log data through the combined strategies of the preset storm word bank, the log content overview preprocessing and the differential storage, and reasonably shunt the log data of the storm part, so that the log management platform can realize the differential management of the normal log and the storm log under the condition of no manual intervention, and the continuous and stable operation effect of the platform is achieved. Specifically, through the preprocessing process of the log to be stored, the characteristics and storm contents of the log storm can be automatically identified, then the differential storage of the storm log data is realized through the route, and the original functions of preprocessing, storing and displaying the data stream are still kept for normal log data, so that the resource consumption of the log storm to the log platform is reduced to a controllable level, and the stability of the log management platform for managing the log is improved. Meanwhile, the initial version of the preset storm word bank configured for the log management platform can be created by industry experience values, and can be continuously and automatically updated subsequently. The automatic updating mechanism of the preset storm word bank can be completed through automatic increment accumulation of a preprocessing module on one hand, and can be completed through a Web (World Wide Web) online editing mode provided by a log management platform on the other hand. That is, the routine maintenance of the preset storm word bank does not need to recompile a program or modify a logic code, so that the overall maintenance cost is reduced.
In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the user accord with the regulations of related laws and regulations, and the public order harmony is not violated.
It should be noted that any permutation and combination of the technical features in the above embodiments also belong to the protection scope of the present invention.
Example III
Fig. 5 is a schematic diagram of a log data processing apparatus according to a third embodiment of the present invention, as shown in fig. 5, where the apparatus includes a log obtaining module 310 to be stored, a first log storage module 320, and a second log storage module 330, where:
a log to be stored acquisition module 310, configured to acquire a log to be stored;
the first log storage module 320 is configured to store the log to be stored as a text file if it is determined that the log to be stored is a storm log;
And the second log storage module 330 is configured to store the log to be stored in a target database if it is determined that the log to be stored is a non-storm log.
According to the embodiment of the invention, when the obtained log to be stored is determined to be the storm log, the log to be stored is stored according to the text file, and when the obtained log to be stored is determined to be the non-storm log, the log to be stored is stored to the target database, so that the problems of high log data maintenance cost, poor log management stability and the like in the existing storm log processing mode are solved, the log management stability can be improved, and the log maintenance cost is reduced.
Optionally, the first log storage module 320 is further configured to calculate quantization index data of the log to be stored, obtain a target log field of the log to be stored if the quantization index data of the log to be stored is determined to meet a log storm judgment condition, and determine the log to be stored as the storm log if the target log field is determined to match a preset storm word bank.
Optionally, the quantization index data includes a log obtaining number quantization index, a log processing time-consuming quantization index, a log processing writing quantization index, and a log average time-consuming index, and the first log storage module 320 is further configured to determine that the quantization index data of the log data to be stored satisfies a log storm judgment condition when it is determined that a rate of change of a log obtaining number quantization index ring ratio upwards exceeds a first threshold, a rate of change of the log processing time-consuming quantization index and the log processing writing quantization index ring ratio upwards exceeds a second threshold, and a rate of change of the log average time-consuming index ring ratio upwards exceeds a third threshold.
Optionally, the first log storage module 320 is further configured to determine a target reference field in the log to be stored, count a total number of characters of the target reference field, and use a first set number of target reference fields with the maximum total number of characters as the target log field.
Optionally, the first log storage module 320 is further configured to replace the time and date content of the log to be stored to obtain a fuzzy processing log, segment the fuzzy processing log according to the content of the field to obtain an original word segmentation field, select a second set number of alternative reference fields for the original word segmentation field, perform pattern recognition on the alternative reference fields, and delete a pattern field in the alternative reference fields to obtain the target reference field.
Optionally, the apparatus may further include a target log field storage module, configured to store the target log field to a preset storm word bank if it is determined that the target log field does not match the preset storm word bank.
The log data processing device can execute the log data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. Technical details which are not described in detail in this embodiment can be referred to the log data processing method provided in any embodiment of the present invention.
Since the log data processing apparatus described above is an apparatus capable of executing the log data processing method in the embodiment of the present application, based on the log data processing method described in the embodiment of the present application, a person skilled in the art can understand the specific implementation of the log data processing apparatus in the embodiment of the present application and various modifications thereof, so how the log data processing apparatus implements the log data processing method in the embodiment of the present application will not be described in detail herein. The apparatus used by those skilled in the art to implement the log data processing method according to the embodiments of the present application is within the scope of the present application.
Example IV
Fig. 6 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including an input unit 16, such as a keyboard, mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a magnetic disk, optical disk, etc., and a communication unit 19, such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the respective methods and processes described above, such as a log data processing method.
Optionally, the log data processing method comprises the steps of obtaining a log to be stored, storing the log to be stored according to a text file when the log to be stored is determined to be a storm log, and storing the log to be stored to a target database when the log to be stored is determined to be a non-storm log.
In some embodiments, the log data processing method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the log data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the log data processing method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be a special or general purpose programmable processor, operable to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a Local Area Network (LAN), a Wide Area Network (WAN), a blockchain network, and the Internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.