Disclosure of Invention
The invention provides a plug-in configuration type log forwarding method, device, electronic equipment and storage medium, which are used for solving the defects of low forwarding efficiency and high forwarding cost of log files caused by the fact that a log forwarding system is designed to forward logs in a targeted input format and output format in the prior art, and realizing flexible processing of various log files into effective logs supporting multiple platforms based on plug-in configuration files for outputting.
The invention provides a plug-in configuration type log forwarding method, which comprises the following steps:
Determining a plug-in configuration file based on the forwarding requirement of the log file to be forwarded, wherein the forwarding requirement is determined based on the type of the log file to be forwarded and the type of a target platform, and the plug-in configuration file is used for processing the log file to be forwarded until the data requirement of the target platform is matched with the log file to be forwarded;
Processing the log to be forwarded based on the plug-in configuration file;
And outputting the processed log to be forwarded to the target platform.
According to the plug-in configuration type log forwarding method provided by the invention, the plug-in configuration file comprises decoding plug-in configuration information, and the processing of the log to be forwarded based on the plug-in configuration file comprises the following steps:
Based on the decoding plug-in configuration information, obtaining a decoding plug-in matched with the type of the log file to be forwarded;
And decoding the log to be forwarded by using the decoding plug-in to obtain target decoding information of the log to be forwarded.
According to the plug-in configuration type log forwarding method provided by the invention, the plug-in configuration file comprises filtering plug-in configuration information, and the method further comprises the following steps:
acquiring a filter plug-in matched with the target decoding information based on the filter plug-in configuration information;
And filtering the target decoding information by using the filtering plug-in to obtain target effective information in the log to be forwarded.
According to the plug-in configuration type log forwarding method provided by the invention, the plug-in configuration file comprises coding plug-in configuration information, and the method further comprises the following steps:
Acquiring a coding plug-in matched with the target effective information based on the coding plug-in configuration information;
and using the coding plug-in to code the target effective information to obtain a target effective log.
According to the plug-in configuration log forwarding method provided by the invention, the plug-in configuration file comprises output plug-in configuration information, and the outputting of the processed log to be forwarded to the target platform comprises the following steps:
acquiring a target platform matched with the target effective log based on the output plug-in configuration information;
And outputting the target effective log to the target platform.
According to the plug-in configuration type log forwarding method provided by the invention, the method for outputting the target effective log to the target platform comprises the following steps:
Determining a target idle time range of the target platform based on the platform idle time matching information;
And when the current moment is in the target idle time range, outputting the target effective log to the target platform.
The invention also provides a plug-in configuration type log forwarding device, which comprises:
the system comprises a determining module, a plug-in configuration file, a forwarding module and a forwarding module, wherein the determining module is used for determining the plug-in configuration file based on the forwarding requirement of a log file to be forwarded, the forwarding requirement is determined based on the type of the log file to be forwarded and the type of a target platform, and the plug-in configuration file is used for processing the log file to be forwarded until the data requirement of the target platform is matched with the log file to be forwarded;
the processing module is used for processing the log to be forwarded based on the plug-in configuration file;
And the output module is used for outputting the processed log to be forwarded to the target platform.
The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of any of the plug-in configuration log forwarding methods described above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a plug-in configured log forwarding method as described in any of the above.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements the steps of a plug-in configured log forwarding method as described in any one of the above.
The plug-in configuration type log forwarding method, the device, the electronic equipment and the storage medium provided by the invention have the advantage that the plug-in configuration file is determined by the plug-in configuration type log forwarding method based on the forwarding requirement of the log file to be forwarded. The forwarding requirement is determined based on the type of the log file to be forwarded and the type of the target platform, and the plug-in configuration file is used for processing the log file to be forwarded to be matched with the data requirement of the target platform, so that the determined plug-in configuration file is used for processing the log file to be forwarded, the targeted and flexible effective processing of the log file to be forwarded can be realized, the processed log to be forwarded is further output to the target platform, the purpose of reliably and effectively forwarding various logs to be forwarded to different platforms is realized through the plug-in configuration file, when the forwarding requirements of the log file to be forwarded are different, the internal service logic of a system is not required to be re-developed, and the purpose of flexibly coping with the forwarding requirements of various log files to be forwarded can be realized only by adjusting or modifying the plug-in type in the plug-in configuration file, so that the development cost is reduced, and the forwarding efficiency of the log file is greatly improved.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The plug-in configuration type log forwarding method, the device, the electronic equipment and the storage medium of the invention are described below with reference to fig. 1-3, wherein the plug-in configuration type log forwarding method is applied to a plug-in configuration type log forwarding system, the plug-in configuration type log forwarding system can be provided with a plurality of plug-ins for processing log files to be forwarded based on plug-in custom plug-in functions, and the plug-in custom plug-in functions can provide a series of different custom function plug-ins for the plug-in configuration type log forwarding system.
Fig. 1 is a flow chart of a plug-in configured log forwarding method provided by the invention, as shown in fig. 1, the plug-in configured log forwarding method includes the following steps:
step 110, determining a plug-in configuration file based on the forwarding requirement of the log file to be forwarded.
Specifically, the plug-in configuration type log forwarding system establishes a plurality of plug-ins for processing different log files to be forwarded, in order to ensure the forwarding effectiveness and forwarding flexibility of the log files, the plug-in configuration files can be predetermined based on the forwarding requirements of the log files to be forwarded, and the plug-in configuration files corresponding to the forwarding requirements of the log files to be forwarded carried by the log forwarding instructions can be determined based on the received log forwarding instructions. No specific limitation is made here.
It should be noted that, for the plug-in configuration file, the type of the target platform that receives the log file to be forwarded may be determined based on the type of the log file to be forwarded, for example, when the type of the log file to be forwarded is an http type, the type of the target platform may be determined to be a Redis type. And then, acquiring the data requirement of the target platform based on the type of the target platform, wherein the data requirement of the target platform of the Redis type can comprise effective data after performing operations such as error correction, incomplete data supplementation, effective data discarding and the like on the received http type log file to be forwarded. Therefore, in order to improve the processing efficiency, the plug-in configuration file can be determined based on the data requirement of the target platform, and the plug-in configuration file can include decoding plug-in configuration information, filtering plug-in configuration information, encoding plug-in configuration information and outputting plug-in configuration information so as to quickly and reliably process the log file to be forwarded into the data required by the target platform.
And 120, processing the log to be forwarded based on the plug-in configuration file.
Specifically, because the plug-in configuration file is determined based on the forwarding requirement of the log file to be forwarded, the log to be forwarded is processed based on the plug-in configuration file, and the plug-in with different functions in the plug-in configuration file can be used for processing the log to be forwarded, so that the purpose of processing the log to be forwarded to be matched with the data requirement of the target platform is achieved.
And 130, outputting the processed log to be forwarded to the target platform.
The target platform may include at least one of a high-throughput distributed publish-subscribe message system kafka, a K-V database, an http output platform, a syslog server, and beanstlkd distributed memory queue systems.
Specifically, the plug-in log forwarding system performs an output operation on the processed log file to be forwarded, which may be a log of the log file to be forwarded processed through various plug-in configuration information in the plug-in configuration file, and at this time, the target platform required to be forwarded by the processed log file to be forwarded may be determined and then output based on the forwarding requirement of the log file to be forwarded, so as to ensure the effectiveness and reliability of output.
According to the plug-in configuration type log forwarding method, the plug-in configuration files are determined based on forwarding requirements of the log files to be forwarded, and the plug-in configuration files are used for processing the log files to be forwarded until the data requirements of the target platform are matched with the type of the log files to be forwarded, so that the determined plug-in configuration files are used for processing the log files to be forwarded, the log files to be forwarded can be processed in a targeted and flexible manner, the processed log to be forwarded is further output to the target platform, the purpose that various logs to be forwarded are reliably and effectively forwarded to different platforms is achieved through the plug-in configuration files, and when the forwarding requirements of the log files to be forwarded are different, the internal service logic of a system is not required to be re-developed, only the plug-in types in the plug-in configuration files are required to be adjusted or modified for fast and concise function expansion, the purpose of flexibly coping with the forwarding requirements of the log files to be forwarded of various types is achieved, development cost is reduced, and forwarding efficiency of the log files is greatly improved.
Optionally, the plug-in configuration file includes decoding plug-in configuration information, and the implementation process of step 120 includes:
And decoding the log to be forwarded by using the decoding plug-in to obtain target decoding information of the log to be forwarded.
The configuration information of the decoding plug-in can represent the mapping relationship between the type of the log and the decoding plug-in the standard format.
Specifically, when receiving different log files to be forwarded, the plug-in configuration log forwarding system can match corresponding decoding plug-ins according to the types of the log files to be forwarded based on decoding plug-in configuration information, that is, more than two log files to be forwarded of the same type can be matched to decoding plug-ins of the same standard format, more than two log files to be forwarded of different types can also be matched to decoding plug-ins of another same standard format, for example, more than two log files to be forwarded in http format can be matched to decoding plug-ins of json format, the log files to be forwarded in http format and the log files to be forwarded in file format can be matched to decoding plug-ins of LINESPLITER format, and different log files to be forwarded are firstly matched to decoding plug-ins of different standard formats based on decoding matching information, so that multiple target decoding information corresponding to the log files to be forwarded of different types can be obtained.
It should be noted that, when the plug-in log forwarding system receives a log file to be forwarded, firstly, different log files to be forwarded are converted into standard internal representations based on decoding plug-in configuration information, that is, metadata of different log files to be forwarded are correspondingly decoded into a form capable of being effectively processed, so that target decoding information of each log file to be forwarded is obtained. Wherein each standard format data may be in the form of unstructured data.
In addition, the plug-in configuration type log forwarding system can be connected with a terminal network by using a protocol, and various log files generated by a management system of the terminal are read and analyzed to determine log files to be forwarded which need to be sent to different platforms. Different platforms can comprise a high-throughput distributed publish-subscribe message system kafka, a K-V database, an http output platform, a syslog server and a beanstlkd distributed memory queue system, wherein the K-V database is a stable, reliable and elastically telescopic online database, and protocols which can be used by the plug-in configuration type log forwarding system comprise a syslog protocol, a statsd protocol and an http protocol.
According to the plug-in configuration type log forwarding method, the decoding plug-in matched with the type of the log file to be forwarded is obtained based on the decoding plug-in configuration information, and the decoding plug-in is further used for decoding the log to be forwarded to obtain target decoding information of the log to be forwarded. The purpose of processing the log file to be forwarded into target decoding information is achieved through the decoding plug-in configuration information, and the internal service logic of the system does not need to be redeveloped when the type of the log file to be forwarded changes, so that development cost is reduced, and guarantee is provided for subsequently improving forwarding efficiency.
Optionally, the plug-in configuration file includes filtering plug-in matching information, and the implementation process of step 120 may further include:
And filtering the target decoding information by using the filtering plug-in to obtain target effective information in the log to be forwarded.
The configuration information of the filtering plug-in can represent the mapping relationship between the decoding information in the standard format and the filtering plug-ins with different functions.
Specifically, the plug-in configuration log forwarding system may further match the target decoding information of each standard format to a filtering plug-in with a corresponding function based on the obtained target decoding information of each standard format, that is, the target decoding information of different standard formats is processed by adopting the filtering plug-in matched with the target decoding information of different standard formats, so as to process the target decoding information of each standard format into a structured data form, for example, process the target decoding information of each standard format into a data structure which is perhaps more friendly to digital statistics data, thereby obtaining target effective information in the log file to be forwarded.
For example, in order to improve the processing efficiency of the target decoding information of each standard format, the plug-in configured log forwarding system may first analyze the content to be processed in the target decoding information of each standard format, and for the content to be processed in the target decoding information of each standard format, the matching Filter plug-ins may be classified and matched, for example, when the target decoding information of the json format is matched to 2 or 3 Skylar filters (filters) and 3 SKYLAR FILTER are SKYLAR FILTER1, SKYLAR FILTER2 and SKYLAR FILTER3 respectively, SKYLAR FILTER1 may be set to match the target decoding information of the json format that needs to correct errors, correct logics and complement defective or null values, SKYLAR FILTER2 may be set to match the target decoding information of the json format that needs to delete repeated items, discard data or variables, and SKYLAR FILTER may be set to match the target decoding information of the json format that needs to be unified specification, conversion structure and data compression, so that the target decoding information of each standard format is quickly processed as target effective information in the log to be forwarded.
It should be noted that, the processes of parsing the content to be processed in the target decoding information of each standard format and correcting errors, deleting repeated items, unifying specifications, correcting logic, converting structure, compressing data, complementing incomplete/null values, discarding data/variables for different content to be processed are all common knowledge of those skilled in the art, and therefore, will not be repeated here.
According to the plug-in configuration type log forwarding method provided by the invention, the filter plug-in matched with the target decoding information of the log file to be forwarded is obtained through filtering plug-in configuration information, and the target decoding information is further filtered by using the filter plug-in to obtain the target effective information in the log to be forwarded. The aim of processing the target decoding information of the log file to be forwarded into the target effective information is fulfilled by filtering the plug-in configuration information, and the integrity, the uniqueness, the authority, the legality and the consistency of the target effective information are ensured, so that the data quality problem is solved, and the data has more mining and displaying potential.
Optionally, the plug-in configuration file includes encoded plug-in configuration information, and the implementation process of step 120 may further include:
And carrying out coding processing on the target effective information by using the coding plug-in to obtain a target effective log.
The configuration information of the encoding plug-in may represent a mapping relationship between a format of the effective information and a file type of the encoding plug-in, and the configuration information of the encoding plug-in may correspond to the configuration information of the decoding plug-in.
Specifically, the plug-in configuration log forwarding system can further match each target effective information to a corresponding coding plug-in for coding processing based on the coding plug-in configuration information aiming at each obtained target effective information, so as to obtain a target effective log.
For example, the plug-in configuration log forwarding system may further match each target valid information to a corresponding coding plug-in based on the coding plug-in configuration information for the obtained multiple different target valid information, so as to obtain target valid logs in different file formats. For example, the json-format target effective information 1 obtained by performing error correction, logic correction and incomplete or null value complementation through SKYLAR FILTER, the json-format target effective information 2 obtained by performing repeated item deletion, data discarding or variable deletion through SKYLAR FILTER, and the json-format target effective information 3 obtained by performing unified specification, conversion structure or data compression through SKYLAR FILTER3 are matched to the json-format encoding plugin for encoding processing, so that the json-format target effective log is obtained.
According to the plug-in configuration type log forwarding method, the coding plug-in matched with the target effective information is obtained through coding plug-in configuration information, and the target effective information is further subjected to coding processing by using the coding plug-in, so that the target effective log is obtained. The aim of encoding the effective information of each target of the log file to be forwarded into a file format supported by multiple platforms before outputting is achieved through the configuration information of the encoding plug-in, and reliability and accuracy of processing the effective information of each target into a file format suitable for different platform requirements are improved.
Optionally, the plug-in configuration file includes output plug-in configuration information, and the implementation process of step 130 may further include:
Acquiring a target platform matched with the target effective log based on the output plug-in configuration information; and outputting the target effective log to the target platform.
The output plug-in configuration information can represent a mapping relationship between a file format of the log and a type of the platform.
Specifically, the plug-in configuration log forwarding system can further acquire a target platform corresponding to each target effective log based on the output plug-in configuration information, and further correspondingly output the target effective log, so that data in the target effective log can be written into a database, a time sequence database or a network server for deep processing.
For example, the plug-in configuration log forwarding system may determine, for the obtained target effective logs with different file formats, a log file format required by each target platform based on the output plug-in configuration information, and further match the corresponding target platform based on the file format of each target effective log, where the target effective logs with the same file format may be matched to more than two different target platforms. For example, the json-format target effective log can be matched to 2 Redis databases and 1 http output platform, so that each target effective log can be output to the corresponding target platform for subsequent deep processing.
According to the plug-in configuration type log forwarding method, the plug-in configuration information is output to be matched with the corresponding target platform aiming at a plurality of different target effective logs, so that each target effective log can be flexibly and variously matched with different target platforms, and the forwarding efficiency of log forwarding is effectively improved.
Optionally, the plug-in configuration file includes platform idle time matching information, and the outputting the target valid log to the target platform includes:
And when the current moment is in the target idle time range, outputting the target effective log to the target platform.
Specifically, in order to avoid that a great amount of target effective logs in a certain time cause great pressure to the same target platform when being output to the target platform, platform idle time matching information can be set in the plug-in configuration file, and each platform and a corresponding idle time range thereof can be included in the platform idle time matching information. When obtaining a plurality of target effective logs, the plug-in configuration log forwarding system can firstly carry out log caching, such as caching to a log caching server, then determine a target idle time range corresponding to each target platform based on platform idle time configuration information, further judge whether the current moment is in the target idle time range of the target platform, if the current moment is in the target idle time range of a target platform, output the target effective logs matched with the target platform to the target platform, otherwise, if the current moment is not in the target idle time range of the target platform, not output the corresponding target effective logs and continue log caching until the current moment reaches the target idle time range of the target platform, and execute output operation. Therefore, the purpose that each target platform receives the corresponding target effective log in the respective target idle time range is achieved.
According to the plug-in configuration type log forwarding method, by setting the platform idle time matching information in the plug-in configuration file, the target platforms matched with each target effective log can receive the corresponding target effective log in the respective target idle time range, so that the defect that the load of each target platform is too high to influence the normal operation of other businesses is avoided, and the flexibility and the reliability of receiving the target effective log by each target platform are greatly improved.
The plug-in configured log forwarding device provided by the invention is described below, and the plug-in configured log forwarding device described below and the plug-in configured log forwarding method described above can be referred to correspondingly.
Fig. 2 illustrates a plug-in configured log forwarding device, as shown in fig. 2, the plug-in configured log forwarding device 200 includes:
A determining module 210, configured to determine a plug-in configuration file based on a forwarding requirement of a log file to be forwarded, where the forwarding requirement is determined based on a type of the log file to be forwarded and a type of a target platform, and the plug-in configuration file is configured to process the log file to be forwarded until the plug-in configuration file matches with a data requirement of the target platform;
A processing module 220, configured to process the log to be forwarded based on the add-in configuration file;
And the output module 230 is configured to output the processed log to be forwarded to a target platform.
Optionally, the processing module 220 may be specifically configured to obtain a decoding plug-in matched with the type of the log file to be forwarded based on the decoding plug-in configuration information, and perform decoding processing on the log to be forwarded by using the decoding plug-in to obtain target decoding information of the log to be forwarded.
Optionally, the processing module 220 may be further specifically configured to obtain a filter plug-in matched with the target decoding information based on the filter plug-in configuration information, and perform filtering processing on the target decoding information by using the filter plug-in to obtain target valid information in the log to be forwarded.
Optionally, the processing module 220 may be further specifically configured to obtain a coding plug-in matched with the target valid information based on the configuration information of the coding plug-in, and perform coding processing on the target valid information by using the coding plug-in to obtain a target valid log.
Optionally, the output module 230 may be specifically configured to obtain, based on the output plug-in configuration information, a target platform that matches the target valid log, and output the target valid log to the target platform.
Optionally, the output module 230 may be further configured to determine a target idle time range of the target platform based on the platform idle time matching information, and output the target valid log to the target platform when the current time is determined to be in the target idle time range.
Fig. 3 illustrates a physical schematic diagram of an electronic device, and as shown in fig. 3, the electronic device 300 may include a processor 310, a communication interface (CommunicationsInterface), a memory 330, and a communication bus 340, where the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. Processor 310 may invoke logic instructions in memory 330 to perform a plug-in configured log forwarding method comprising:
Determining a plug-in configuration file based on the forwarding requirement of the log file to be forwarded, wherein the forwarding requirement is determined based on the type of the log file to be forwarded and the type of a target platform, and the plug-in configuration file is used for processing the log file to be forwarded until the data requirement of the target platform is matched with the log file to be forwarded;
Processing the log to be forwarded based on the plug-in configuration file;
And outputting the processed log to be forwarded to a target platform.
Further, the logic instructions in the memory 330 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. The storage medium includes a U disk, a removable hard disk, a Read-only memory (ROM), a random access memory (RAM, randomAccessMemory), a magnetic disk, an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute a plug-in configured log forwarding method provided by the above methods, and the method includes:
Determining a plug-in configuration file based on the forwarding requirement of the log file to be forwarded, wherein the forwarding requirement is determined based on the type of the log file to be forwarded and the type of a target platform, and the plug-in configuration file is used for processing the log file to be forwarded until the data requirement of the target platform is matched with the log file to be forwarded;
Processing the log to be forwarded based on the plug-in configuration file;
And outputting the processed log to be forwarded to a target platform.
In yet another aspect, the present invention further provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the plug-in configured log forwarding method provided by the above methods, the method comprising:
Determining a plug-in configuration file based on the forwarding requirement of the log file to be forwarded, wherein the forwarding requirement is determined based on the type of the log file to be forwarded and the type of a target platform, and the plug-in configuration file is used for processing the log file to be forwarded until the data requirement of the target platform is matched with the log file to be forwarded;
Processing the log to be forwarded based on the plug-in configuration file;
And outputting the processed log to be forwarded to a target platform.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
It should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention, and not for limiting the same, and although the present invention has been described in detail with reference to the above-mentioned embodiments, it should be understood by those skilled in the art that the technical solution described in the above-mentioned embodiments may be modified or some technical features may be equivalently replaced, and these modifications or substitutions do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solution of the embodiments of the present invention.