Log processing method and system, electronic device and storage mediumTechnical Field
The present invention relates to the field of log processing, and in particular, to a log processing method and system, an electronic device, and a storage medium.
Background
A log (log) refers to a collection of certain operations of objects specified by the system and their results in time order. The log file is a log file, records interactive information between a system and a user of the system, and is a data collection method for automatically capturing the interactive type, content or time between a person and a system terminal; for web searches, a log is an electronic record of a user searching for information using a web search engine interacting with the web search engine during the search for information.
Along with the increasing complexity and the increasing strength of the functions of the mobile phone, the log quantity generated by the mobile phone system is increased, and due to the difference in the function realization of each subsystem in the chip, the log format of each subsystem is inevitably different greatly, so that it is difficult to achieve a better log compression ratio and compression overhead by using a uniform data compression algorithm. Each subsystem manages the log independently, so that log entries finally stored in an external memory are scattered, and later system-level log data analysis and mining are inconvenient.
For the mobile phone software test version, the log quantity generated in the operation process is much larger than that of the formal version, and the conventional external memory such as an eMMC (embedded memory) or a TF-Card (TF storage Card), even the conventional PC (Personal Computer) hard disk, cannot completely meet the requirement of continuously storing a large quantity of logs in the external field test, so that the logs are easily lost or covered.
Disclosure of Invention
The invention provides a log processing method and system, electronic equipment and a storage medium, aiming at overcoming the defect that a unified data compression algorithm is difficult to achieve a better log compression ratio and compression overhead in the prior art.
The invention solves the technical problems through the following technical scheme:
a first aspect of the present invention provides a log processing method applied to an electronic device, where the electronic device includes a plurality of subsystems that implement different functions, and the log processing method includes:
aiming at each subsystem, coding the content of the non-attribute value in the log by adopting a uniform coding format;
respectively compressing the logs in each subsystem, and summarizing the compressed logs of each subsystem according to the synchronous signals;
and packaging the summarized logs and transmitting the packaged logs to at least one external memory.
Preferably, each subsystem includes a group of ping-pong buffers, and the step of compressing the logs in each subsystem respectively specifically includes:
and compressing the logs in the ping-pong buffer which does not execute the operation of storing the logs currently aiming at each subsystem.
Preferably, the synchronization signal is generated at intervals, or after transmission of the packed log is completed.
Preferably, the external memory is a board level memory installed in the electronic device, a memory of other electronic devices, and/or a memory on a server.
A second aspect of the present invention provides an electronic device comprising a memory, a processor, a computer program stored on the memory and executable on the processor, and a plurality of subsystems implementing different functions, wherein the processor implements the log processing method of the first aspect when executing the computer program.
A third aspect of the present invention provides a log processing method, including the steps of:
collecting a log of the electronic device of the second aspect stored in at least one external memory;
unpacking and decompressing the collected logs;
combining the decompressed logs to obtain a complete log of the electronic equipment;
and decoding the complete log to obtain an original log.
Preferably, after unpacking the collected log and before decompressing the collected log, the method further comprises the following steps:
and screening the unpacked logs according to a screening principle.
Preferably, the screening principles belong to the same time period or the same subsystem.
Preferably, the external memory is a board level memory installed in the electronic device, a memory of other electronic devices, and/or a memory on a server.
A fourth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the log processing method of the third aspect when executing the computer program.
A fifth aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the log processing method according to the first or third aspect.
A sixth aspect of the invention provides a log processing system comprising the electronic device of the second aspect and the electronic device of the fourth aspect.
The positive progress effects of the invention are as follows: the contents of the non-attribute values in the logs of each subsystem are coded by adopting a uniform coding format, namely the formats of the logs in each subsystem are unified, so that the number of the logs and the transmission cost of the logs are greatly reduced after the logs are compressed, and the compression ratio of the logs is improved.
Drawings
Fig. 1 is a flowchart of a log processing method according to embodiment 1 of the present invention.
Fig. 2 is a block diagram showing a connection relationship between an electronic device and an external memory.
Fig. 3 is a schematic structural diagram of an electronic device according to embodiment 2 of the present invention.
Fig. 4 is a flowchart of a log processing method according toembodiment 3 of the present invention.
Fig. 5 is a block flow diagram illustrating a process of collecting and restoring a Modem log in an electronic device by using the log processing method provided inembodiment 3 of the present invention.
Detailed Description
The invention is further illustrated by the following examples, which are not intended to limit the scope of the invention.
Example 1
The embodiment provides a log processing method, which is applied to electronic equipment, wherein the electronic equipment comprises a plurality of subsystems for realizing different functions. According to the log processing method, the contents of the non-attribute values in the logs of the subsystems are encoded in a uniform encoding format, so that the number of logs, the cost for frequently preparing log character strings and the cost for log transmission are greatly reduced after the logs are compressed, and the compression ratio of the logs is improved.
In some optional embodiments, the electronic device includes a plurality of cellular communication subsystems such as a Modem for implementing a cellular communication function, a Wireless connection processor subsystem such as a WCN (Wireless Connectivity Network) for processing Wireless services such as Wi-Fi, a MultiMedia processor subsystem such as a MultiMedia, application processor ap (application processor) subsystem for processing MultiMedia related services, and a system processor sp (system processor) subsystem for processing system level services.
The log processing method provided by this embodiment specifically includes the following steps, as shown in fig. 1:
and S101, encoding the contents of the non-attribute values in the log by adopting a uniform encoding format for each subsystem. Wherein, the attribute value is a specific feature or parameter defining the attribute, and the content other than the attribute value is the content except the attribute value.
In step S101, the original content is replaced with the encoded result for the content having no attribute value. In a specific example, a log includes attribute values and non-attribute values, and the format of the log is changed to "# M2$0.5$ 3" after the non-attribute values are encoded. Wherein, 0.5 and 3 are attribute values, and the rest contents are the contents of encoded non-attribute values.
It should be noted that the coding format adopted in step S101 for each subsystem may be defined by itself, or may use the prior art. In this embodiment, in order to subsequently compress the log by using an algorithm that is suitable for all the subsystems and has a high compression rate, the contents of the non-attribute values in each subsystem are uniformly encoded, that is, the format of the log in each subsystem is unified.
And S102, respectively compressing the logs in each subsystem, and summarizing the compressed logs of each subsystem according to the synchronous signals.
In an alternative embodiment of step S102, each subsystem includes a group of ping-pong buffers, and for each subsystem, the logs in the ping-pong buffers that do not currently perform the log storing operation are compressed. In specific implementation, each subsystem includes a log management module LogManager, the LogManager switches ping-pong buffers for temporarily storing logs back and forth, and compresses logs temporarily stored last time, that is, logs stored in ping-pong buffers not currently executing log storage operation, specifically using hardware compression or software compression. In one example, for a multimedia processor subsystem, including ping-pong buffer1 and ping-pong buffer2, a LogManager of the multimedia processor subsystem is enabled to perform ping-pong buffer switching based on a synchronization signal and compress a log in the ping-pong buffer before switching. The specific process is as follows: if the current ping-pong buffer1 is executing the operation of storing the log, the LogManager is notified to switch to using the ping-pong buffer2 to store the log, and after the switching of the ping-pong buffer is completed, the log in the ping-pong buffer1 is compressed. In another example, the LogManager in the subsystem switches the ping-pong buffer for storing the log according to the full signal of the ping-pong buffer, and compresses the log in another ping-pong buffer which does not execute the storing operation.
In a specific implementation, the synchronization signal is sent out by a log management module SysLogManager of the system. In an optional implementation manner, each subsystem uploads the compressed log to a log management module SysLogManager of the system to implement log aggregation.
In an alternative embodiment, the synchronization signal is generated at intervals. In this embodiment, the compressed logs of each subsystem are collected at regular intervals. In an alternative embodiment, the synchronization signal is generated for completing transmission of the packed log. In this embodiment, the operation of summarizing the compressed logs of the subsystems is executed after the packed logs are transmitted.
It should be noted that, in an implementation, the period of the synchronization signal is controlled according to the log throughput of each subsystem and the size of the ping-pong buffer in each subsystem, so as to avoid the ping-pong buffer overflow or the log loss.
In order to facilitate subsequent analysis of the log, in an optional implementation manner of step S102, the log after compression of each subsystem is summarized according to the sequence of the timestamps.
And step S103, packaging the collected logs and transmitting the packaged logs to at least one external memory.
In a specific implementation of step S103, the assembled log is packaged by the log packaging processing module LogPackage, and the packaged log is transmitted to the external memory by the log output module LogOut. In an alternative embodiment, the LogPackage packs the summarized logs according to a uniform format, for example, packs the summarized logs in a format of { timestamp, subsystem to which the summarized log belongs, sub-function to which the summarized log belongs }. Wherein, the sub-function may be power consumption.
In the specific implementation of step S103, LogOut may transmit the packaged log to one external memory, or may transmit the packaged log to a plurality of external memories, so as to meet the requirement of storing a large amount of logs. In a specific implementation, the selection of which external storage or storages to transmit the packed log may be based on a test scenario, a real-time transmission overhead condition, or a system limitation.
The external memory may be a board-level memory installed in the electronic device, such as a TF-Card, an eMMC, a UFS (UNIX file system) or a Flash (memory chip), which may meet the requirement of storing test logs in most laboratories; the memory can also be a memory of other electronic equipment, for example, a hard disk of a PC (personal computer) connected with the electronic equipment in a wired manner, and the memory can meet the log storage requirement of a common external field pressure test; the storage on the server can also be used, for example, a hard disk on the server wirelessly connected with the electronic device, which can meet the log storage requirement of a plurality of electronic device terminal tests. It should be noted that the external memory is not limited to the above example, and may be any memory other than the electronic device.
The electronic device in this embodiment may be a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), and the like. Fig. 2 is a block diagram for illustrating a connection relationship between an electronic apparatus and an external memory. In the example shown in fig. 2, the electronic device includes five subsystems, i.e., a Modem, a WCN, a SP, a Multimedia, and an AP, and is connected to the TF-Card through a board interface, a PC through a wired connection, and a server through a wireless connection.
The following is an example of how the mobile phone selects the external memory to store the log.
In one example, the handset decides whether to transfer the log to the TF-Card based on whether the presence of the TF-Card is detected. Specifically, if the mobile phone detects that a TF-Card exists, the log is transmitted to the TF-Card, and if the mobile phone does not detect the TF-Card, the log is transmitted to other external storage, such as a hard disk of a PC or a hard disk on a server.
In another example, the mobile phone determines whether to transmit the log to the memories of other electronic devices according to whether the USB interface has established a data connection and whether the electronic device connected with the mobile phone through the USB has an open storage space. Specifically, if the mobile phone detects that the USB interface has established a data connection and the electronic device connected to the mobile phone via the USB has opened a storage space, the log is transmitted to the electronic device, otherwise, the log is transmitted to another external storage, such as a hard disk on a server.
In another example, the mobile phone is connected to a plurality of external memories, and the mobile phone determines to which external memory the log is transmitted according to the size of the log. For example, if the size of the log exceeds a first threshold, the log is transmitted to a TF-Card installed in the mobile phone; if the size of the log exceeds a second threshold value, transmitting the log to a PC (personal computer) connected with the mobile phone through a wire; and if the size of the log exceeds a third threshold value, transmitting the log to a server in wireless connection with the mobile phone. In this example, the third threshold is greater than the second threshold, which is greater than the first threshold.
In another example, the mobile phone determines which external memory to transfer the log to according to the remaining storage space of the external memory, in addition to considering the log size.
Example 2
Fig. 3 is a schematic structural diagram of an electronic device provided in this embodiment. The electronic device includes a memory, a processor, a computer program stored on the memory and executable on the processor, and a plurality of subsystems implementing different functions, and the processor implements the log processing method of embodiment 1 when executing the program. Theelectronic device 3 shown in fig. 3 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.
The components of theelectronic device 3 may include, but are not limited to: the at least one processor 4, the at least onememory 5, and abus 6 connecting the various system components (including thememory 5 and the processor 4).
Thebus 6 includes a data bus, an address bus, and a control bus.
Thememory 5 may include volatile memory, such as Random Access Memory (RAM)51 and/orcache memory 52, and may further include Read Only Memory (ROM) 53.
Thememory 5 may also include a program/utility 55 having a set (at least one) ofprogram modules 54,such program modules 54 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The processor 4 executes various functional applications and data processing, such as the log processing method of embodiment 1 of the present invention, by running the computer program stored in thememory 5.
Theelectronic device 3 may also communicate with one or moreexternal devices 7, such as a keyboard, pointing device, etc. Such communication may be via an input/output (I/O)interface 8. Also, the model-generatingdevice 3 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) through anetwork adapter 9. As shown in fig. 3, thenetwork adapter 9 communicates with the other modules of the model-generateddevice 3 via thebus 6. It should be appreciated that although not shown in FIG. 3, other hardware and/or software modules may be used in conjunction with the model-generateddevice 3, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
It should be noted that although in the above detailed description several units/modules or sub-units/modules of the electronic device are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module according to embodiments of the invention. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
Example 3
The present embodiment provides a log processing method, which collects and restores the log of the electronic device described in embodiment 2 from an external storage, so as to facilitate a developer to analyze the log. The log processing method provided by the embodiment is generally applied to electronic devices such as a PC, and developers can analyze the restored logs by using the electronic devices such as the PC.
The log processing method of the embodiment includes the following steps, as shown in fig. 4:
step S301, collecting logs of the electronic device as described in embodiment 2 stored in at least one external memory. The external memory is specific to the electronic device described in embodiment 2, and is specifically a memory outside the electronic device, and is used to store a log of the electronic device.
In a specific implementation, the external memory may be a board-level memory installed in the electronic device, such as TF-Card, eMMC, UFS, or Flash, a memory of another electronic device, such as a hard disk of a PC connected to the electronic device by wire, or a memory on a server, such as a hard disk on a server connected to the electronic device wirelessly.
And step S302, unpacking and decompressing the collected logs.
If a plurality of logs stored in the external memory are collected in step S301, the collected logs need to be unpacked and decompressed in step S302.
In an optional implementation manner of step S302, after unpacking the collected log and before decompressing the collected log, the method further includes the following steps: and screening the unpacked logs according to a screening principle.
In an alternative embodiment, the screening criteria are the same time period. For example, all logs belonging to a certain time period are filtered according to the time stamp. In an alternative embodiment, the screening principles are the same subsystem. For example screening all logs belonging to the cellular communication subsystem.
Note that the algorithm for decompression in step S302 corresponds to the compression method in step S102 in embodiment 1.
And step S303, combining the decompressed logs to obtain a complete log of the electronic equipment.
In an optional implementation manner of step S303, the decompressed logs are combined according to the precedence order of the timestamps.
And S304, decoding the complete log to obtain an original log.
The decoding algorithm in step S304 corresponds to the encoding algorithm in step S101 in embodiment 1.
Fig. 5 is a block flow diagram for illustrating a process of collecting and restoring a Modem log in an electronic device by using the log processing method of the present embodiment. In the example shown in fig. 5, logs of the electronic device according to embodiment 2 are collected from the TF-Card, the PC, and the server, respectively, and a Modem log of three parts is screened out therefrom, the three parts are combined to obtain a Modem complete log, and the Modem original log is obtained after decoding.
Example 4
The present embodiment provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the log processing method ofembodiment 3 when executing the program.
Components of the electronic device may include, but are not limited to: the at least one processor, the at least one memory, and a bus connecting the various system components (including the memory and the processor).
Example 5
The present embodiment provides a computer-readable storage medium on which a computer program is stored, the program implementing the steps of the log processing method ofembodiment 1 or 3 when executed by a processor.
More specific examples, among others, that the readable storage medium may employ may include, but are not limited to: a portable disk, a hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible implementation, the present invention can also be implemented in the form of a program product including program code for causing a terminal device to perform the steps of implementing the log processing method ofembodiment 1 or 3 when the program product is run on the terminal device.
Where program code for carrying out the invention is written in any combination of one or more programming languages, the program code may be executed entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on a remote device or entirely on the remote device.
Example 6
The present embodiment provides a log processing system including the electronic device provided in embodiment 2 and the electronic device provided in embodiment 4. In this embodiment, the logs in the electronic device provided in embodiment 2 are re-encoded, compressed, summarized, and packed, and are transmitted to at least one external memory, and the electronic device provided in embodiment 4 is used to collect, unpack, decompress, combine, and decode, so as to obtain the original logs of the electronic device provided in embodiment 2.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.