CN110928853A

Movatterモバイル変換

Info

Publication number: CN110928853A
Application number: CN201811073225.4A
Authority: CN
Inventors: 李继永
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2018-09-14
Filing date: 2018-09-14
Publication date: 2020-03-27

Abstract

The invention discloses a method and a device for identifying a log, and relates to the technical field of computers. One embodiment of the method comprises: when a service request for a target service is detected, intercepting the service request, and acquiring a request parameter and service data in the service request; determining a service identifier corresponding to the request parameter, and generating a log file corresponding to the target service by combining service data; and when receiving a log access request corresponding to the service identifier, determining that the log file corresponds to the log access request and outputting the log file. According to the embodiment, the service identification is carried out on the service request, and the generated log file is labeled by the service identification, so that a worker can quickly and accurately position the log file according to the service identification, and a basis is provided for checking log problems.

Description

Method and device for identifying log

Technical Field

The invention relates to the technical field of computers, in particular to a method and a device for identifying a log.

Background

In internet applications, a large number of logs (logs) are generated every day, each log records a description of the relevant operation such as date, time, user and the like, and the logs are required to be stored for query and analysis.

The log is the main basis for checking the running state of the system and troubleshooting problems, and for mutual calling among a plurality of systems, the log can locate the problem occurrence point and search the performance bottleneck to realize link calling monitoring.

In the process of implementing the invention, the inventor finds that the prior art has at least the following problems:

1) usually, a system is developed and maintained by a plurality of people for a long time, so that the style of the log is uneven, the log file is often searched, and particularly, the effective log file is difficult to search when the problem is solved; or a plurality of log files are found, the information amount is huge, and the time and the labor are consumed for separating effective logs;

2) in an actual scene, log files are often scattered in each storage area, and the arrangement is disordered; and the processing period of each log file is different, which results in larger memory occupation and higher hardware cost in the storage area.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for identifying a log, which can at least solve the problem of a large operation amount in the prior art for checking and monitoring the log.

To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method of identifying a log, including:

when a service request for a target service is detected, intercepting the service request, and acquiring a request parameter and service data in the service request;

determining a service identifier corresponding to the request parameter, and generating a log file corresponding to the target service by combining the service data;

and when receiving a log access request corresponding to the service identifier, determining that the log file corresponds to the log access request and outputting the log file.

Optionally, the determining the service identifier corresponding to the request parameter further includes:

traversing preset service identifier configuration, and determining the inquired service identifier as the service identifier corresponding to the request parameter when the service identifier corresponding to the request parameter exists in inquiry; or

And when the service identifier corresponding to the request parameter does not exist in the inquiry, determining the preset service identifier as the service identifier corresponding to the request parameter.

Optionally, when receiving a log access request corresponding to the service identifier, determining that the log file corresponds to the log access request and outputting the log file includes:

when the log access request is received, intercepting the log access request, and acquiring an access request parameter in the log access request;

and determining a service identifier corresponding to the access request parameter, and when the determined service identifier is the same as the service identifier of the request parameter, determining that the log file corresponds to the log access request and outputting the log file.

Optionally, the service identifier includes a service ticket number;

when receiving a log access request corresponding to the service identifier, determining that the log file corresponds to the log access request and outputting the log file, including:

and when the service identification corresponding to the log access request comprises the service single number, determining that the log file corresponds to the log access request and outputting the log file.

Optionally, the service identifier further includes a time identifier, where the time identifier corresponds to at least one of a time for receiving the service request, a time for intercepting the service request, and a time for generating the log file;

and when the number of the determined log files is multiple, sequencing the determined log files according to the time identification, generating a log chain corresponding to the sequenced log files and outputting the log chain.

To achieve the above object, according to another aspect of the embodiments of the present invention, there is provided an apparatus for identifying a log, including:

the request intercepting module is used for intercepting a service request of a target service when the service request is detected, and acquiring request parameters and service data in the service request;

an identifier determining module, configured to determine a service identifier corresponding to the request parameter, and generate a log file corresponding to the target service in combination with the service data;

and the log determining module is used for determining that the log file corresponds to the log access request and outputting the log file when the log access request corresponding to the service identifier is received.

Optionally, the identifier determining module is configured to:

Optionally, the log determining module is configured to:

Optionally, the service identifier includes a service ticket number;

the log determination module is configured to:

To achieve the above object, according to still another aspect of embodiments of the present invention, there is provided an electronic device for identifying a log.

The electronic device of the embodiment of the invention comprises: one or more processors; a storage device, configured to store one or more programs, which when executed by the one or more processors, cause the one or more processors to implement any of the above methods for identifying a log.

To achieve the above object, according to a further aspect of the embodiments of the present invention, there is provided a computer readable medium having a computer program stored thereon, the computer program, when executed by a processor, implementing any of the above methods for identifying a log.

According to the scheme provided by the invention, one embodiment of the invention has the following advantages or beneficial effects: by carrying out service identification on the service request, the generated log file is labeled by the service identification, so that a worker can quickly and accurately position the log file according to the service identification, and a basis is provided for log problem investigation.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of a main flow of a method for identifying a log according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of the main modules of an apparatus for identifying logs according to an embodiment of the present invention;

FIG. 3 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

FIG. 4 is a schematic block diagram of a computer system suitable for use with a mobile device or server implementing an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The words to which the invention relates are to be construed as follows:

ES: for the acronym of elastic search, ES is a highly extended, open source full text retrieval and analysis engine that can rapidly store, search, and analyze large volumes of data in near real time.

log4j 2: is an open source code project of Apache, and by using Log4j, the destinations of Log information transmission can be controlled to be consoles, files, GUI components, even socket servers, event recorders of NT, UNIX Syslog daemon, and the like; the output format of each log can be controlled; by defining the level of each piece of log information, the generation process of the log can be controlled more finely.

Referring to fig. 1, a main flowchart of a method for identifying a log according to an embodiment of the present invention is shown, including the following steps:

s101: when a service request for a target service is detected, intercepting the service request, and acquiring a request parameter and service data in the service request;

s102: determining a service identifier corresponding to the request parameter, and generating a log file corresponding to the target service by combining the service data;

s103: and when receiving a log access request corresponding to the service identifier, determining that the log file corresponds to the log access request and outputting the log file.

The service in the invention can be order service in an e-commerce platform, and also can be other service needing to record state. For the log record of the business, the data in the business is mainly printed for later query, and currently, log4j/slf4j/log4j2 and the like are mainly available.

The invention takes the order in the E-commerce platform as an example, and completes the whole life cycle of the business order from the time when a user places the order (business request) to the time when the user inquires and takes the order out of the warehouse.

In the above embodiment, for step S101, the present invention intercepts a service request (usually, HTTP/HTTPs request) at each request entry by means of AOP (Aspect oriented programming). The purpose of interception is to append a specific unique identifier to each request, since the original request does not contain the identifier.

For example, after a user purchases a commodity and places an order, the identification information in the order information is extracted through the invention, the service identification of the order is generated and added to the order placing request, and the service identification is transmitted in the whole request process.

It should be noted that the identifier may be user-defined. Each request has its own thread context, the obtained service identifier is added to its own thread local, and each log printing in the subsequent request takes the service identifier from the current thread local first, and then adds the service identifier to each log printing to generate a log file.

In the prior art, the service identification aims at all logs, and the function of only recording the interest of a user cannot be realized.

For step S102, in a specific implementation, the determining method for the service identifier may include:

the method comprises the following steps: acquiring request parameters in the log access request according to a preset reflection mode;

step two: and traversing the preset service identifier configuration, and determining the service identifier corresponding to the request parameter.

For step one: the reflection technique here can be the way in the existing JAVA architecture. The request parameters are not limited in type and format, such as user name, password, identity information, address information, contact information, and service order number (order number, logistics freight order number, etc.) in the order.

For step two: matching the customized service identifier configurations one by one, for example: the user defines an identifier of "name", and parameters requested include "{ age ═ xxx, name ═ xxx, address ═ xxx }", and after analysis and matching, the name identifier name ═ xxx is obtained and stored temporarily.

The service identification configuration may be obtained by staff counting historical request records, for example, 90% of requests sent by users carry names, but only 1% of requests carry names, and only names are added to the service identification configuration.

See, for example, table 1:

TABLE 1

Name of field	Type (B)	Whether or not it is empty
			Service identification	VARchar	N
Whether to intercept	INT	N
			Priority level	INT	N

Wherein, VARCHAR represents that a character string with variable length can be saved. INT is a data type in a database, while, as a function, an INT function refers to a "floor function" in a commonly used function in a database. The portion may be defined by a configuration file or a database.

If the matching fails, however, a fixed service identifier (e.g., identifier "UNUSE") is added to the request, or not. For example: the user only defines the identifiers of "name" and "age", but the request parameters are { home ═ xxx and address ═ xxx }, since the parameters are not in the range defined by the user, the matching is not achieved, and at this time, the default identifier is added.

It should be noted that, for the request parameter-request that fails to match in the service identification configuration, the corresponding log is not of interest to the user. Even if recorded, the amount of access may be small, resulting in a waste of storage resources. In addition, some marks and the like are possibly interesting to the user although being cold, and the marks can be matched.

The invention is therefore mainly directed to querying for existing requests in a service identity arrangement, i.e. sending logs of interest to the user to other systems. Logs that are not of interest to the user identification are also passed and appear to be rather confusing.

And if the number of the inquired and obtained service identifiers is only one, directly adding the service identifiers to the request. But there may be more than one. For example, the user defines identifiers of "name" and "age", and the request parameters are "{ age ═ xxx, name ═ xxx, and address ═ xxx }", there are two service identifiers: "name" and "age".

For the case of multiple service identifiers, there may be a certain management manner, specifically:

1) priority management: the requests may be sorted by their priority (see table 1 for details). The priority order is customized by staff, for example, the user defines two interesting service identifications of "name and age" at the same time, and the user is most interested in the name, then the priority of the two service identifications is the name > age.

To facilitate management of multiple service identities in a request, a processing identity may be generated, including a prioritized service identity. For example, if the service identifier "name and age" are both present in the same request, then the name is placed in bit 1 and the age is placed in bit 2 when the process identifier is generated.

2) And (3) separator management: in order to facilitate the index establishment of the subsequent ES, the service identifier in the processing identifier may be separated by a separator. For example, the request parameter is { age ═ xx, name ═ xxx, address ═ xxx }, the user defines name and age identifier, the delimiter is "-", and the ideal format after processing is name: xxx-age: xx.

It should be noted that the separator may also be other symbols having a separating function, such as "/", "; "and the like. However, if the separator is not used, all the service identifiers may be combined together to form a long string, so that the difference between the identifiers and the data cannot be distinguished, and the subsequent query system cannot be subjected to splitting and word segmentation.

The two management modes may be managed in combination or may be managed by using them individually, which is not limited herein.

Furthermore, in order to improve the accuracy of the service log, the service identifier can be processed, so that subsequent query, management and screening are facilitated.

For example, a prefix, e.g., LOG _ TRACE _ TAG _, may be added before the service identification. A time identifier may also be added after the service identifier, where the time identifier is generally the time of receiving the request, the time of intercepting the request, or the time of generating a log file, and the invention is not limited; and the time identifier may be in the millisecond level, or in other levels, and the invention is not limited as such.

When the log is analyzed subsequently, only the log with the prefix identifier as the beginning can be analyzed, and other logs without the identifier can be discarded, so that the storage capacity of the log is reduced.

It should be noted that, for the added objects of the prefix identifier and the time identifier, the added objects are requests of "non-default service identifiers", that is, the defined parameters matched are queried in the service identifier configuration, and if not matched, the added objects of the prefix identifier and the time identifier will not be added, because the present invention mainly solves the identification request in which the user is interested.

The invention can use the log printing component to record the log. In addition to the local file record, the print component will automatically send the log to the messaging middleware. For example, kafka (a high throughput distributed publish-subscribe messaging system), but could also be other middleware.

As described with reference to the previous step S101, for the printing of the log, the service identifier may be called from the request and performed in conjunction with the service data. The business data may be commodity information (quantity, fragility, valuables, etc.) in the order, urgency, etc.

Typically, the log is stored in the local memory of the server in the form of a file. The invention has two logs, one original log comprises all logs with marks and logs without marks; and a log which is self-defined by the staff is saved.

It should be noted that, for the printing of the log, the generation of the log file, and whether the log has the service identifier or not is not processed here, so as to avoid affecting the performance of the production system.

After receiving the message (log file) of kafka, the analysis component judges whether the log contains a specific service identifier, and if the service identifier exists, the log is stored in an ES (electronic storage) for facilitating subsequent query and use; but if the service identifier does not exist, the service identifier is directly discarded and is not stored.

The index structure of ES here is as follows:

the index structure field trace _ key represents a service identifier, and the trace _ log represents log content.

For step S103, during subsequent log query, the log access request may also be extracted to generate a service identifier, specifically:

the method comprises the following steps: when the log access request is received, intercepting the log access request, and acquiring an access request parameter in the log access request;

step two: determining a service identifier corresponding to the access request parameter;

step three: and when the determined service identification is the same as the service identification of the request parameter, determining that the log file corresponds to the log access request and outputting the log file.

Taking the service order number in the service identifier as an example, after the user/worker/customer service inputs the service order number, in the query mechanism of the ES, the column "trace _ key" -service order number is queried, and the corresponding log is obtained and extracted.

Taking the user name in the service identifier as an example, the column "trace _ key" -user name may also be queried in the query mechanism of the ES, so as to obtain and extract the corresponding log.

However, one user may purchase goods in the platform many times, and in order to improve the screening accuracy, multiple service identifiers are usually screened at the same time. For an order, the processes of ordering, scheduling, delivering and the like are all in the log file, so that the query result may obtain a plurality of log files.

In order to facilitate subsequent viewing, the logs can be arranged according to the generation time and the like of the logs and the reverse description/sequence of the time, so that the logs are simply and clearly displayed, a log chain can be generated for viewing, and the feeling of 'sea fishing needle' is avoided.

The purpose of the log storage is for the person in need to query, so there must be manual action. Currently, the screened logs are mainly compared in a manual mode.

Through the mode, the original inquiry which is troublesome and laboursome can be simplified into the input condition, the output result can be conveniently obtained, the cost is reduced, and the time spent on searching is shortened.

According to the method provided by the embodiment of the invention, the service identification is carried out on the service request, and the generated log file is labeled by the service identification, so that a worker can quickly and accurately position the log file according to the service identification, and a basis is provided for checking log problems.

Referring to fig. 2, a schematic diagram of main modules of anapparatus 200 for identifying a log according to an embodiment of the present invention is shown, including:

arequest intercepting module 201, configured to, when a service request for a target service is detected, intercept the service request, and obtain a request parameter and service data in the service request;

anidentifier determining module 202, configured to determine a service identifier corresponding to the request parameter, and generate a log file corresponding to the target service in combination with the service data;

and thelog determining module 203 is configured to determine that the log file corresponds to the log access request and output the log file when the log access request corresponding to the service identifier is received.

In the device for implementing the present invention, theidentifier determining module 202 is configured to:

In the device for implementing the present invention, thelog determining module 203 is configured to:

In the implementation device of the invention, the service identifier comprises a service single number;

thelog determining module 203 is configured to:

In the implementation device of the present invention, the service identifier further includes a time identifier, and the time identifier corresponds to at least one of a time for receiving the service request, a time for intercepting the service request, and a time for generating the log file;

thelog determining module 203 is configured to:

In addition, the specific implementation content of the log identification device in the embodiment of the present invention has been described in detail in the above log identification method, so that repeated content is not described herein.

According to the device provided by the embodiment of the invention, the service identification is carried out on the service request, and the generated log file is labeled by the service identification, so that a worker can quickly and accurately position the log file according to the service identification, and a basis is provided for checking log problems.

Fig. 3 illustrates anexemplary system architecture 300 to which the log identifying method or log identifying apparatus of embodiments of the present invention may be applied.

As shown in fig. 3, thesystem architecture 300 may include

terminal devices

301, 302, 303, anetwork 304, and a server 305 (by way of example only). Thenetwork 304 serves as a medium for providing communication links between the

terminal devices

301, 302, 303 and theserver 305.Network 304 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal device

301, 302, 303 to interact with theserver 305 via thenetwork 304 to receive or send messages or the like. Various communication client applications may be installed on the

terminal devices

301, 302, 303.

The

terminal devices

301, 302, 303 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

Theserver 305 may be a server providing various services, such as a background management server (for example only) providing support for shopping-like websites browsed by users using the

terminal devices

301, 302, 303. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.

It should be noted that the identification log method provided by the embodiment of the present invention is generally executed by theserver 305, and accordingly, the identification log apparatus is generally disposed in theserver 305.

It should be understood that the number of terminal devices, networks, and servers in fig. 3 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 4, a block diagram of acomputer system 400 suitable for use with a terminal device implementing an embodiment of the invention is shown. The terminal device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 4, thecomputer system 400 includes a Central Processing Unit (CPU)401 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)402 or a program loaded from astorage section 408 into a Random Access Memory (RAM) 403. In theRAM 403, various programs and data necessary for the operation of thesystem 400 are also stored. TheCPU 401,ROM 402, andRAM 403 are connected to each other via abus 404. An input/output (I/O)interface 405 is also connected tobus 404.

The following components are connected to the I/O interface 405: aninput section 406 including a keyboard, a mouse, and the like; anoutput section 407 including a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; astorage section 408 including a hard disk and the like; and acommunication section 409 including a network interface card such as a LAN card, a modem, or the like. Thecommunication section 409 performs communication processing via a network such as the internet. Adriver 410 is also connected to the I/O interface 405 as needed. Aremovable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on thedrive 410 as necessary, so that a computer program read out therefrom is mounted into thestorage section 408 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through thecommunication section 409, and/or installed from theremovable medium 411. The computer program performs the above-described functions defined in the system of the present invention when executed by a Central Processing Unit (CPU) 401.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a request interception module, an identification determination module, and a log determination module. Where the names of these modules do not in some cases constitute a limitation on the module itself, for example, the identity determination module may also be described as "determining the identity module corresponding to the request parameter".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:

According to the technical scheme of the embodiment of the invention, the service identification is carried out on the service request, and the generated log file is labeled by the service identification, so that a worker can quickly and accurately position the log file according to the service identification, and a basis is provided for checking log problems.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.