Disclosure of Invention
The embodiment of the invention provides a network data processing method based on DPDK, which aims to solve the problems that massive data is required to be screened and the data analysis efficiency is low when network data is tracked.
Correspondingly, the embodiment of the invention also provides a network data processing device based on DPDK, which is used for ensuring the realization and application of the method.
In order to solve the above problems, an embodiment of the present invention discloses a network data processing method based on DPDK, including:
network data of a preset structure body is sent through the DPDK, wherein the preset structure body is an Mbuf data structure body;
acquiring a preset functional area in the Mbuf data structure body, wherein the functional area is provided with a tracking control node and a plurality of acquisition nodes, and the tracking control node is provided with a state identifier;
when the tracking control node is in a starting state, generating log data according to the network data acquired by the acquisition node;
and generating a processing path for the network data according to the log data.
Optionally, when the tracking control node is in a start state, generating log data according to the network data collected by the collection node includes:
acquiring domain positions of the network data processing nodes;
generating setting information by the acquisition node by adopting the domain position;
and when the network data is intersected from the functional area, the set information is adopted by the tracking control node to generate the log data.
Optionally, the functional area further has a presentation control node, where the presentation control node has a status identifier, and the method further includes:
when the display control node is in a starting state, trace information matched with the set information is acquired through the display control node, and the trace information is displayed, wherein the trace information comprises at least one of source IP, destination IP, protocol number and length.
Optionally, the generating a processing path for the network data according to the log data includes:
acquiring a mapping relation between the acquisition nodes and the functional area, wherein the mapping relation comprises the sequence of the acquisition nodes;
and generating path information aiming at the network data by adopting the log data and the mapping relation.
The embodiment of the invention also discloses a network data processing device based on DPDK, which comprises:
a data transmitting module, configured to transmit network data of a preset structure body through the DPDK, where the preset structure body is an Mbuf data structure body;
the function area acquisition module is used for acquiring a preset function area in the Mbuf data structure body, wherein the function area is provided with a tracking control node and a plurality of acquisition nodes, and the tracking control node is provided with a state identifier;
the log data generation module is used for generating log data according to the network data acquired by the acquisition node when the tracking control node is in a starting state;
and the data processing module is used for generating a processing path aiming at the network data according to the log data.
Optionally, the log data generating module includes:
a domain position obtaining sub-module, configured to obtain a domain position of the network data processing node;
the setting information generation sub-module is used for generating setting information by adopting the domain position through the acquisition node;
and the log generation sub-module is used for generating the log data by adopting the setting information through the tracking control node when the network data is intersected from the functional area.
Optionally, the functional area further has a presentation control node, where the presentation control node has a status identifier, and the apparatus further includes:
and the data display module is used for acquiring trace information matched with the set information through the display control node when the display control node is in a starting state, and displaying the trace information, wherein the trace information comprises at least one of source IP, destination IP, protocol number and length.
Optionally, the data processing module includes:
the mapping relation acquisition sub-module is used for acquiring a mapping relation between the acquisition nodes and the functional area, wherein the mapping relation comprises the sequence of the acquisition nodes;
and the path information generation sub-module is used for generating path information aiming at the network data by adopting the log data and the mapping relation.
The embodiment of the invention also discloses a device, which comprises:
one or more processors; and
one or more machine-readable media having instructions stored thereon, which when executed by the one or more processors, cause the apparatus to perform one or more methods as described above.
One or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause the processors to perform one or more of the methods described above are also disclosed.
The embodiment of the invention has the following advantages:
in the embodiment of the invention, network data of a preset structure body is sent through a DPDK, the preset structure body is an Mbuf data structure body, then a preset functional area in the Mbuf data structure body is obtained, wherein the functional area is provided with a tracking control node and a plurality of acquisition nodes, the tracking control node is provided with a state identifier, when the tracking control node is in a starting state, log data are generated according to the network data acquired by the acquisition nodes, and then a processing path for the network data is generated according to the log data, so that the data processing path is recorded in the functional area of the preset structure body by combining the characteristics of the preset structure body, the data can be tracked, the analysis process of the transmitted data is greatly simplified, and the processing efficiency of the network data problem is improved.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Referring to fig. 1, a flowchart illustrating steps of an embodiment of a DPDK-based network data processing method according to the present invention may specifically include the following steps:
step 101, network data of a preset structure body is sent through a DPDK, wherein the preset structure body is an Mbuf data structure body;
for Network devices, such as routers, switches, media gateways, SBCs, PS gateways, etc., a large number of packets are often required to be sent and received instantaneously, so that conventional Network devices are configured with a dedicated NP (Network processing) processor, or FPGA (Field-Programmable Gate Array, field programmable gate array), or ASIC (Application Specific Integrated Circuit ), etc. These specialized devices forward messages efficiently through built-in hardware circuits (or hardware circuits formed by programming), and require CPU intervention when deep processing of the message is required.
For application scenarios such as public cloud, NFV (Network Function Virtualization ), etc., the infrastructure uses cpu as an operation core, often does not have a dedicated NP processor, and the operating system also uses general Linux as a main component. In the virtualized environment, the transceiving path of the network data is longer, and because of the switching between the kernel mode and the user mode and the multiple memory copies of the packet processing task, the system consumption becomes large, and the system taking the CPU as the core has a large processing bottleneck. Therefore, intel proposed DPDK techniques for servicing the IA (Intel Architecture) system in order to increase packet processing efficiency at the general purpose server.
DPDK Date Plane Development Kit is a software library for packet data processing acceleration. Compared with the traditional data packet processing, the DPDK has the following characteristics: 1. polling: avoiding the overhead of interrupting the context switch while processing the packet; 2. user state driving: unnecessary memory copying and system calling are avoided, and rapid iterative optimization is facilitated; 3. affinity and monopolization: the specific task can be appointed to work only on a certain core, so that the frequent switching of threads among different cores is avoided, and more cache hits are ensured; 4. and (3) reducing access overhead: the TLB miss is reduced by using the memory big page, and the effective bandwidth of memory access is improved by using memory multi-channel staggered access; 5. software tuning: cache line alignment, prefetching data, and multi-metadata batch operation.
In the embodiment of the invention, a DPDK framework is adopted to output the network data processing process. The data network frame content is encapsulated in a preset structure body so as to utilize DPDK to transmit network data.
In a specific implementation, the preset structure is an Mbuf (struct rte_mbuf) structure, and for efficient access of data, the DPDK may encapsulate the network data frame within the Mbuf structure. Referring to fig. 2, a schematic diagram of an Mbuf structure provided by a DPDK-based network data processing method according to the present invention is shown, in which an Mbuf structure header is carefully designed, and only 1 Cache Line is originally adjusted to 2 Cache lines to adapt to the situation that information carried by an Mbuf header increases, in practical application, basic, frequently accessed data may be placed in one Cache Line byte, and functionally extended data may be placed in a second Cache Line byte. The Mbuf header contains all the data required for packet processing, and for a single Mbuf, there is also a pointer to the next Mbuf structure to form a Frame linked list structure for a Jumbo Frame (Jumbo Frame) that is not being dropped.
In a specific implementation, network data of the Mbuf data structure body is sent through the DPDK, after the network data frame is received, the network data frame can be placed into an abstract Mbuf structure data memory pool, and in the effective lifetime, only the pointer pointing content is modified in a limited way, and large-block data memory movement is not performed, so that CPU resources can be concentrated to process data of a message.
Step 102, obtaining a preset functional area in an Mbuf data structure body;
in the embodiment of the invention, a functional area can be configured in a head (tail) room in the Mbuf data structure body as a trace record of a network data processing process, so that data tracking by using DPDK is realized.
In a specific implementation, a usage pointer of the head room can be obtained by adding a sizeof (struct rte_mbuf) offset to a head pointer address of the current data Mbuf, starting with a base address of the pointer, and defining a usage range of a record, so that a functional area is scratched out as a trace record of a message data processing process, wherein the functional area can be provided with a tracking control node, a display control node and a plurality of acquisition nodes, and each node is provided with a corresponding state identifier.
In an example of the embodiment of the present invention, a 32Bit functional area may be defined, where each Bit has a corresponding state identifier, for example, a start state is 1, and an off state is 0. Defining high 2Bit as enabling of function control, for other Bit, each Bit represents processing trace of a node, when network data processing is performed on the mapped function node, setting 1 corresponding Bit domain, and the specific table is as follows:
the flags log represents tracking control nodes, and when the Bit value is 0, the tracking log is started at the moment, and each acquisition node can acquire network data in the processing process; when the Bit value is 1, it indicates that the data is trace data, and the trace record may be written into a log or reported. Flag. Dump represents a presentation control node, and when the Bit value is 0, the presentation control node can jump to stored information; when the Bit value is 1, the recorded log data can be printed and displayed. And (3) marking the acquisition node, and tracking and marking the network data processing process when the Bit value is 1.
The valid meaning may be defined for Bit of each acquisition node in advance, and each node corresponds to a step in the network data processing process, for example:
#define Network_TraceNode_Step1 0x0000 0001
#define Network_TraceNode_Step2 0x0000 0002
#define Network_TraceNode_Step3 0x0000 0004
#define Network_TraceNode_Step4 0x0000 0008
#define Network_TraceNode_Step5 0x0000 0010
……
#define Network_TraceNode_Step29 0x2000 0000
in a specific implementation, the processing procedure of the network data may be divided into a path and an endpoint, the path may correspond to a plurality of collection nodes, and on each collection node, setting may be performed according to the mapped unique Bit identifier, and a flag. Dump flag may be checked to determine whether to output the log print. At the endpoint, for the output endpoint, it may be decided whether to write the trace record into the log by obtaining the status identification of the flags.
It should be noted that, in the present invention, the functional area length defined with 32 Bits is exemplified, and it is understood that those skilled in the art may increase the Bit area length of the functional area according to the number of nodes, for example, to 64Bits, 128Bits, etc., which is not limited in this aspect of the present invention.
Step 103, when the tracking control node is in a starting state, generating log data according to the network data acquired by the acquisition node;
in the embodiment of the invention, when the Bit value of the tracking control node is 0, the tracking control node indicates that the state is default, and the tracking control node controls all the acquisition nodes to only acquire the network data processing flow, when the Bit value is 1, the tracking control node indicates that the tracking control node is in a starting state, so that all the acquisition nodes can be controlled to acquire the network data processing flow, and corresponding acquisition log data is generated according to the network data acquired by the acquisition nodes.
In a specific implementation, in the network data processing process, mbuf always penetrates through the process, the domain position of the network data processing node is acquired through the acquisition node, then the corresponding set information is generated by adopting the domain position, trace of the network data in the processing process is marked on Mbuf, and when the network data is intersected from the functional area, log data aiming at the network data processing process can be generated by adopting the set information through the tracking control node, so that the real-time tracking and marking of the network data processing node is realized, the inquiry of the network data abnormal flow is facilitated, and the network problem positioning period is shortened.
In an example of the embodiment of the present invention, at an entry of a node function process of each process, a domain location, for example, an IPv4 message, related to a current processing node may be used, and if a network_tracenode_step5 is used, a 5 th Bit value of a 32Bit region of a functional area corresponding to an Mbuf of data is set to 1 (an initial default value is 0), and a process of the node is identified. This allows traces of the network data during processing to be identified on the Mbuf.
At an output endpoint of a data processing path of the network, the identification of the 31 st Bit position in the 32Bit region of the functional region corresponding to Mbuf of the data can be judged, when the Bit value corresponding to the flag. Log Bit is 1, the data is tracking data at the moment, setting information generated by adopting the acquisition node can be used for generating log data aiming at the network data, and thus the tracking record of the tracking data is written into the log.
For a commonly input endpoint, a judging condition of one-step trace can be added in a preprocessing stage of data input, for example, in an external network port receiving processing function, a preprocessing process is added: this judgment condition may be a global data trace enable switch (configured to be turned on by Shell command issuing); or some specific information of the data, such as the data L3 protocol number or the destination IP of the data. When the judging condition is true, outputting the corresponding Bit position 1 of the flag log, and outputting the corresponding Bit record trace after processing the output end from the received data, thereby realizing the control Bit identification configuration of the data.
In addition, the identification of the tracking control node can be configured to be set when the flow table item is established and the data searching flow hits. For an endpoint inserted by an application, the control node can be tracked directly according to the configuration of the input parameters.
In another example of the embodiment of the present invention, in the processing process of the network data, the flag/dump Bit may be determined to determine whether to print trace information, where the flag/dump Bit is located on the 30 th Bit of the Field Bit Field, and when the corresponding Bits value is 1, a print function is called to obtain trace information adapted to the set information, and the trace information of the current node is displayed. The trace information may include a source IP, a destination IP, a protocol number, a length, and the like, among others.
If the flag is 1, the program checks that the key information such as source IP, destination IP, protocol number, and length of the corresponding data is printed and displayed.
Step 104, generating a processing path for the network data according to the log data.
In the embodiment of the invention, a functional area with fixed Bit is divided in an Mbuf data structure body, a plurality of Bit bits are used as the enabling of functional control, and the rest Bit bits are used as acquisition nodes for acquiring a network data processing process, wherein the effective meaning is predefined for the Bit of each acquisition node in advance, so that the sequence of each acquisition node in the functional area is fixed.
In a specific implementation, the mapping relation between the acquisition node and the functional area can be acquired, and then the log data generated by the tracking control node and the mapping relation are adopted to generate the processing path information for the network data, so that the tracking function of the data service processing flow is realized by combining the design of setting, recording and the like in the data processing flow through the function expansion of Mbuf under the DPDK network data processing frame, the skill requirement of maintenance manpower is reduced, and the problem positioning period is shortened.
In one example of the embodiment of the invention, the processing path of the network data can be analyzed by receiving the network data of the preset structure body sent based on the DPDK, analyzing the set information of Bits in the functional area of the preset structure body and combining the mapping relation between the acquisition node and the functional area, so that the realization process of data tracking can be optimized without additional consumption of global variables and without considering recorded memory management, the inquiry of abnormal processing flows is facilitated, and the flow test and the transmission problem positioning are simpler.
In the embodiment of the invention, network data of a preset structure body is sent through a DPDK, the preset structure body is an Mbuf data structure body, then a preset functional area in the Mbuf data structure body is obtained, wherein the functional area is provided with a tracking control node and a plurality of acquisition nodes, the tracking control node is provided with a state identifier, when the tracking control node is in a starting state, log data are generated according to the network data acquired by the acquisition nodes, and then a processing path for the network data is generated according to the log data, so that the data processing path is recorded in the functional area of the preset structure body by combining the characteristics of the preset structure body, the data can be tracked, the analysis process of the transmitted data is greatly simplified, and the processing efficiency of the network data problem is improved.
It should be noted that, for simplicity of description, the method embodiments are shown as a series of acts, but it should be understood by those skilled in the art that the embodiments are not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are presently preferred embodiments, and that the acts are not necessarily required by the embodiments of the invention.
Referring to fig. 3, a block diagram illustrating an embodiment of a DPDK based network data processing apparatus according to the present invention may include the following modules:
a data sending module 301, configured to send network data of a preset structure body through the DPDK, where the preset structure body is an Mbuf data structure body;
the function area obtaining module 302 is configured to obtain a function area preset in the Mbuf data structure body, where the function area has a tracking control node and a plurality of acquisition nodes, and the tracking control node has a status identifier;
the log data generating module 303 is configured to generate log data according to the network data collected by the collecting node when the tracking control node is in a start state;
and the data processing module 304 is configured to generate a processing path for the network data according to the log data.
In an alternative embodiment of the present invention, the log data generating module may include:
a domain position obtaining sub-module, configured to obtain a domain position of the network data processing node;
the setting information generation sub-module is used for generating setting information by adopting the domain position through the acquisition node;
and the log generation sub-module is used for generating the log data by adopting the setting information through the tracking control node when the network data is intersected from the functional area.
In an optional embodiment of the present invention, the functional area further has a presentation control node, where the presentation control node has a status identifier, and the apparatus may further include:
and the data display module is used for acquiring trace information matched with the set information through the display control node when the display control node is in a starting state, and displaying the trace information, wherein the trace information comprises at least one of source IP, destination IP, protocol number and length.
In an alternative embodiment of the present invention, the data processing module may include:
the mapping relation acquisition sub-module is used for acquiring a mapping relation between the acquisition nodes and the functional area, wherein the mapping relation comprises the sequence of the acquisition nodes;
and the path information generation sub-module is used for generating path information aiming at the network data by adopting the log data and the mapping relation.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.
The embodiment of the invention also provides a device, which comprises:
one or more processors; and
one or more machine-readable media having instructions stored thereon, which when executed by the one or more processors, cause the apparatus to perform the method described by the embodiments of the present invention.
Embodiments of the invention also provide one or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause the processors to perform the methods described in embodiments of the invention.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.
It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the invention.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The invention has been described in detail with reference to a DPDK-based network data processing method and a DPDK-based network data processing apparatus, and specific examples are applied to illustrate the principles and embodiments of the invention, and the above examples are only used to help understand the method and core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.