Disclosure of Invention
The application provides a data processing method capable of being applied to multi-core equipment and improving the basic forwarding and security service processing efficiency, and electronic equipment applying the method.
In order to solve the above technical problems, an embodiment of the present application provides a data processing method, which is applied to a multi-core device, and the method includes:
the method comprises the steps of establishing a basic forwarding process group, wherein the basic forwarding process group comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing various processes and resources, and the working processes are respectively affinitive with a processor core;
establishing a security service processing process group, which comprises a plurality of security service processing processes, wherein the security service processing processes are respectively compatible with a processor core and are used for completing the security service processing of the message;
establishing a first channel between the plurality of security service processing processes and the main process, so that the plurality of security service processing processes finish registration to the main process through the first channel;
and establishing a second channel between the plurality of working processes and the plurality of security business processing processes so that the plurality of security business processing processes acquire the message through the second channel.
Optionally, each session data created by the plurality of working processes is shared, and the master process is shared with global data variables among the plurality of working processes;
and the multiple security business processing processes are independent in memory and support independent restarting, and the multiple security business processing processes only access the data in the shared memory area created by the main process.
Optionally, the establishing a first channel between the plurality of security service processing processes and the main process includes:
each security service processing process establishes a first channel with the main process when being started, wherein the first channel comprises a data packet channel;
the establishing a second channel between the plurality of working processes and the plurality of security service processing processes includes:
and establishing a second channel between each security service processing process and each working process, wherein the second channel comprises a DPDK lock-free queue.
Optionally, the service data recorded when the security service processing process processes a session message is private data corresponding to the session message and is associated with the corresponding session message;
the method further comprises the steps of:
the security service processing process determines a data management mode based on the memory occupation condition;
and the security service processing process independently maintains the private data and the session message associated with the private data based on the determined data management mode.
Optionally, the data management manner includes:
applying for a memory pool when each security service processing process is initialized;
determining an ID address of the session message and a private data memory pool head address;
storing private data corresponding to the session message into the corresponding memory pool, and establishing a quick index based on the ID address and the first address;
and verifying the validity of the private data based on the session age of the session message.
Optionally, the data management manner includes:
storing private data corresponding to the session message into a target memory corresponding to the secure data processing process;
establishing a hash chain table for the target memory;
determining the ID address and the session age of the session message;
and calculating a hash index value at least based on the ID address of the session message and the session age.
Optionally, the method further comprises:
and the security service processing process obtains notification information sent by the working process after deleting the session message, and releases the corresponding private data based on the notification information.
Optionally, the method further comprises:
the working process locally establishes a schedule for recording ages, and the ages of the schedule are consistent with the ages of the state tables of the security processing business processes recorded in the main process;
the working process carries out security service processing policy matching on the newly built session message according to the policy issued by the command line of the main process;
the working process determines a target security business processing process based on the matching result and the scheduling table;
the working process records that the ID address of the target security service processing process is in the new session message;
when the target security service processing process finishes the processing of the new session message, the processed new session message is transferred back to the working process;
and the working process executes route forwarding on the received newly-built session message.
Optionally, the method further comprises:
and the working process determines whether the current operation scene is a target scene, if so, forwards the message to the security service processing process, directly executes the routing forwarding of the message, wherein the target scene comprises restarting of the security service processing process, reaching of the configured threshold value of the total number of the target messages, and reaching of the full load of the second channel.
Another embodiment of the present application also provides an electronic device, including:
the first establishing module is used for establishing a basic forwarding process group and comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing each process and resource, and the working processes are respectively affinitive with a processor core;
the second establishing module is used for establishing a safety service processing process group, and comprises a plurality of safety service processing processes, wherein the plurality of safety service processing processes are respectively compatible with one processor core and used for completing the safety service processing of the message;
a third establishing module, configured to establish a first channel between the plurality of security service processing processes and the main process, so that the plurality of security service processing processes complete registration with the main process through the first channel;
and the fourth establishing module is used for establishing a second channel between the plurality of working processes and the plurality of security service processing processes so that the plurality of security service processing processes acquire the message through the second channel.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
The technical scheme of the application is further described in detail through the drawings and the embodiments.
Detailed Description
Hereinafter, specific embodiments of the present application will be described in detail with reference to the accompanying drawings, but not limiting the application.
It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be taken as limiting, but merely as exemplification of the embodiments. Other modifications within the scope and spirit of this disclosure will occur to persons of ordinary skill in the art.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and, together with a general description of the disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the disclosure.
These and other characteristics of the application will become apparent from the following description of a preferred form of embodiment, given as a non-limiting example, with reference to the accompanying drawings.
It is also to be understood that, although the application has been described with reference to some specific examples, a person skilled in the art will certainly be able to achieve many other equivalent forms of the application, having the characteristics as set forth in the claims and hence all coming within the field of protection defined thereby.
The above and other aspects, features and advantages of the present disclosure will become more apparent in light of the following detailed description when taken in conjunction with the accompanying drawings.
Specific embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings; however, it is to be understood that the disclosed embodiments are merely examples of the disclosure, which may be embodied in various forms. Well-known and/or repeated functions and constructions are not described in detail to avoid obscuring the disclosure in unnecessary or unnecessary detail. Therefore, specific structural and functional details disclosed herein are not intended to be limiting, but merely serve as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure.
The specification may use the word "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," which may each refer to one or more of the same or different embodiments in accordance with the disclosure.
Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings.
As shown in fig. 1 and fig. 2, an embodiment of the present application provides a data processing method, which is applied to a multi-core device, and the method includes:
the method comprises the steps of establishing a basic forwarding process group, wherein the basic forwarding process group comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing various processes and resources, and the working processes are respectively affinitive with a processor core;
establishing a safety service processing process group, which comprises a plurality of safety service processing processes, wherein the plurality of safety service processing processes are respectively compatible with a processor core and are used for completing the safety service processing of the message;
establishing a first channel between a plurality of security service processing processes and a main process, so that the plurality of security service processing processes register to the main process through the first channel;
and establishing a second channel between the plurality of working processes and the plurality of security service processing processes so that the plurality of security service processing processes acquire messages through the second channel.
The method in the embodiment can be applied to multi-core equipment, for example, more than 64 cores, and the method based on the embodiment can fully utilize all processor cores, such as CPU cores, so that the overall performance is effectively improved. In addition, the working processes in the basic forwarding process group and the security service processing process group in the embodiment realize data transmission through different first channels and second channels respectively, and the processes cannot be influenced, so that the operation of other processes cannot be influenced even if a certain process fails, the basic forwarding process group (engine) and the security service processing process group (engine) can operate cooperatively and efficiently and stably, and code coupling cannot occur.
Specifically, the security service processing process group in this embodiment may be a security service processing engine, which includes a plurality of SE processes, where each SE process is affinity to a CPU core, and may be one-to-one or many-to-one. The basic forwarding process group may be referred to as a basic forwarding engine, which includes a master host process and a plurality of work processes, which may also be referred to as worker processes. The master main process runs on the 0-core CPU and is responsible for resource initialization such as DPDK large page memory, interface drive and other processes, including management of a working process and a security service processing process (SE process), and the work process and the SE process need to register with the master after being started. Each worker process and one CPU core are compatible, which may be one-to-one or many-to-one. Each worker process is bound with a network card receiving and transmitting queue at the same time and is used for receiving messages from a network card driver and then carrying out routing forwarding. When the method is applied, the basic forwarding process group and the security service processing process group in the embodiment can be realized based on a DPDK second process.
Further, in this embodiment, each session data created by a plurality of work processes is shared, and a master process is shared with global data variables among the plurality of work processes;
the memory of the plurality of security business processing processes is independent, independent restarting is supported, and the plurality of security business processing processes only access the data in the shared memory area created by the main process.
For example, in implementation, the process type running on each CPU core in the multi-core device may be set through a configuration file, and then the DPDK running parameters may be set through a script. The master and the worker processes can be compiled into a binary program, and when the binary program runs, the master process is started to read the number of the workers and the bound CPU core information, and then all the worker processes are pulled up by the master. The session data structures created among the various worker processes are shared, and other data structures can be included for sharing. Global data variable sharing between master and worker processes. The worker process and the master process in this embodiment do not support separate restarts.
Further, the SE process is compiled into a binary program independently, the master process reads the configuration file through the script to obtain the starting number of the SE processes and the CPU core binding information, then all the SE processes are pulled up, resources among the SE processes are independent, the memory is independent, and the SE processes support independent restarting, so that when other SE processes are damaged, only the damaged SE processes are required to be restarted, and the whole security service processing engine is not required to be restarted. Meanwhile, each SE process can also perform independent memory management based on the dlma library. Each SE process accesses the shared memory area by means of the shared memory area created by the read-only mapping master process, namely, only data such as session (message) and the like can be read-only accessed, so that the bad writing of the shared memory is prevented from affecting the operation of other processes.
Further, in this embodiment, establishing a first channel between a plurality of security service processing processes and a main process includes:
each security service processing process establishes a first channel with a main process when being started, wherein the first channel comprises a data packet channel;
establishing a second channel between the plurality of work processes and the plurality of security service processing processes, comprising:
and a second channel is established between each security service processing process and each working process, and comprises a DPDK lock-free queue.
For example, a channel established between each SE process and the master is a local socket channel, and is used for implementing that the SE process transmits registration information and heartbeat keep-alive information to the master process. The first number of channels may be, for example, M, which is the same as the number of SE processes. The second channel established between each SE process and the worker process may optionally use a DPDK lock-free queue rte _ring for transmitting messages and messages. Each SE process can also receive the message forwarded by the worker process through the DPDK lock-free queue to process the security service. The second channel number may be, for example, n+m, where N is the number of started worker processes and M is the number of started SE processes.
Specifically, in this embodiment, all the SE processes are managed by the master process, a local socket channel is established with the master process after each SE process is started, information such as a process type and the like is registered to the master process, heartbeat is sent to the master process at regular time, and SE state data is maintained by the master process. And for the worker process, the worker process acquires the state information of the SE process on the master process through the shared memory to generate a local scheduling table for later determining the SE process of the user processing message. As described above, the memory of each SE process is independent, the processes do not share data, and the SE process can be restarted independently after a problem occurs, so that master, worker and other SE processes are not affected. In the restarting process, the worker process updates a local scheduling table, and the old session flow corresponding to the damaged SE process directly passes and newly establishes session flow to schedule to other SE processes of the same type; after restarting, the SE process registers with the master process again, so that the worker schedule is added to participate in the flow processing.
Further, the session is composed of a source IP, a destination IP, a source port, a destination port, and five tuples represented by four-layer protocol numbers, and represents a message flow, which is used for caching a routing query result and a security service processing result. The service data recorded when each security service processing process processes the same session message is called the private data of the session (message). session is created by the worker process, private data is spawned by the SE process, and private data needs to be associated one-to-one with session.
The method in this embodiment further includes:
the security service processing process determines a data management mode based on the memory occupation condition;
the security service processing process independently maintains private data and session messages associated with the private data based on the determined data management mode.
Specifically, the SE process performs read-only mapping on the session memory, independently maintains private data information and a corresponding relation with the session, and guarantees the validity of the private data through the session age. The SE process provides two session private data management modes of a private data memory pool and a hash chain table, specifically selects which mode can be selected according to the actual memory occupation condition, if the memory occupation condition is serious, the hash chain table can be selected preferentially, otherwise, the memory pool can be selected.
For example, embodiment one:
the data management mode comprises the following steps:
applying for a memory pool when each security service processing process is initialized;
determining an ID address of the session message and a first address of a private data memory pool;
storing private data of the corresponding session message into a corresponding memory pool, and establishing a quick index based on the ID address and the first address;
and verifying the validity of the private data based on the session age of the session message.
For example, as shown in fig. 3, when each SE process is initialized, a memory pool (the total number of sessions is equal to the size of the private data management header) is applied, and according to the session ID, the private data management header address can be quickly indexed and corresponding session private data information is maintained. The private data management mode has the advantages of high index speed, no overhead released by memory application, independent process of each SE, no inter-process mutual exclusion and no memory boundary crossing access risk. The dse_private_global in the figure is the first address of the private data memory pool, and the offset session_id is just the private data management structure body. And the Age is the session Age of the session message, and is used for checking the validity of the private data, and the 8-byte private_data array stores the private data address.
Embodiment two:
the data management mode comprises the following steps:
storing private data corresponding to the session message into a target memory corresponding to the security data processing process;
establishing a hash chain table for a target memory;
determining an ID address and a session age of the session message;
the hash index value is calculated based at least on the ID address of the session message and the session age.
For example, as shown in fig. 4, private data may be managed by using a hash chain table (hash chain table), and the hash value calculated according to the session_id, the session_age, and the hash_lines is indexed to the corresponding hash chain table header, so that when the private data is added, the application for the session private data management structure memory is required, and the corresponding chain table is inserted, and when the private data is deleted, the corresponding chain table is required to be deleted, and the private data management structure memory is released. In the figure, dse_private_hash_global is the initial address of the memory pool of the hash chain table head after initialization, and the hash values calculated according to the session_id and the session_age are shifted to obtain the chain table head. The private data management structure body needs to apply for release, the age guarantees the validity of the private data, session_id is used for hash conflict detection, and 8-byte private_data stores a private data address for an array.
Further, the method in this embodiment further includes:
the security service processing process obtains the notification information sent after the working process deletes the session message, and releases the corresponding private data based on the notification information.
For example, as shown in fig. 5, the release of session private data in the SE process is performed by notifying the SE process through an msg message after the worker process deletes the session message. Considering the influence of frequent issuing of msg messages on performance under a newly built flow model, when a worker process can cumulatively delete a specific number (for example, 32) of session messages, encapsulating one msg message and sending the msg message to an SE process, and deleting private data msg messages by using session id: and carrying out encapsulation transmission in the format of the session message. The SE process receives the msg message and then processes the session id in batch: the session is indexed to the corresponding private data structure memory, and the validity of the private data is judged according to the session. Releasing the effective private data according to the destructors set by different modules; and for the condition that the private data release callback does not exist, the private data address is put into a se_private_free_global resource recycling annular queue, the timer is used for timing, and the queue automatically releases the private data after the time is out.
Further, the method in this embodiment further includes:
the working process locally establishes a schedule for recording ages, and the ages of the schedule are consistent with the ages of the state tables of the security processing business processes recorded in the main process;
the working process carries out security service processing policy matching on the newly built session message according to the policy issued by the command line of the main process;
the working process determines a target security business processing process based on the matching result and the scheduling table;
the working process records that the ID address of the target security service processing process is in the newly built session message;
when the target security service processing process finishes the processing of the new session message, the processed new session message is transferred back to the working process;
and the working process executes route forwarding on the received newly-built session message.
For example, when packet scheduling is performed on the worker processes, in order to avoid the performance being affected by locking, as described above, each worker process generates a schedule locally, records the state table age for comparison with the master process, and can quickly update the local schedule of the worker process when the state table age of the SE process monitored by the master process changes. And when the message is processed by one SE process, the message is converted back to the worker process, the worker process continues to match the next strategy, the id of the target SE determined by matching is recorded in sequence until all strategy processing is completed, and then the routing forwarding is continuously executed. The subsequent message of the session directly carries out message forwarding according to the stored target SE id information, the survival state of the target SE process and the state of a message channel, namely the second channel state, are obtained before forwarding, and if the state is abnormal, the flow bypass is directly executed.
For example, as shown in FIG. 6, the WAF is run on the SE1 process and the SE3 process, and the SE2 process and the SE4 process run the IPS. After the Worker process matches WAF and IPS strategies, a SE1 process and a SE2 process are selected according to a local scheduling table, corresponding SE ids are recorded, messages are sequentially forwarded to the SE1 process, the SE1 process returns the messages and carried processing results to the Worker after processing, the messages are then scheduled to the SE2 process for processing, and finally the Worker process forwards the messages to the next stage from the interface of the equipment according to the message output interface obtained by searching the route. Wherein the parts numbered 1 and 2 in the figure represent the first schedule of the worker process and the SE process, and the parts numbered 3 and 4 represent the second schedule.
Further, the method in this embodiment further includes:
the working process determines whether the current operation scene is a target scene, if so, the message is forwarded to the security service processing process, the routing forwarding of the message is directly executed, the target scene comprises restarting of the security service processing process, the total number of the target messages reaches a configured threshold value, and the second channel reaches full load.
For example, the worker process executes the traffic bypass, i.e., directly routes forwarding without performing security traffic processing, in a scenario comprising:
1) The cross-core queue rte _ring is full;
2) The total number of the cross-core messages reaches a configured threshold value;
3) The SE process is restarted.
The data processing method described in the above embodiment can make full use of all CPU cores on multi-core security devices, such as devices above 64 cores, to improve overall performance; the method can realize the single-core restarting of the security service engine and the flow bypass on the basic forwarding engine on the premise of ensuring the performance, and improve the running stability of the equipment, thereby achieving the aims of improving the cost performance of the equipment and reducing the later maintenance cost. In addition, the embodiment can realize a basic forwarding process and a safe service process based on a DPDK second process, the basic forwarding process is used for binding a limited interface queue to transmit and receive messages, the safe service processing process only carries out service processing, the two processes are independently compiled and operated, zero-copy transmission of messages between the two processes is realized through a lock-free queue, and the data processing efficiency is further improved.
As shown in fig. 7, another embodiment of the present application also provides an electronic device, including:
the first establishing module is used for establishing a basic forwarding process group and comprises a plurality of working processes for receiving and forwarding messages and a main process which runs on a processor and is used for managing each process and resource, and the working processes are respectively affinitive with a processor core;
the second establishing module is used for establishing a safety service processing process group, comprising a plurality of safety service processing processes, wherein the plurality of safety service processing processes are respectively compatible with a processor core and are used for completing the safety service processing of the message;
the third establishing module is used for establishing a first channel between the plurality of security service processing processes and the main process so that the plurality of security service processing processes can register with the main process through the first channel;
and the fourth establishing module is used for establishing a second channel between the plurality of working processes and the plurality of security service processing processes so that the plurality of security service processing processes acquire messages through the second channel.
Optionally, each session data created by the plurality of working processes is shared, and the master process is shared with global data variables among the plurality of working processes;
the memory of the plurality of security business processing processes is independent, independent restarting is supported, and the plurality of security business processing processes only access the data in the shared memory area created by the main process.
Optionally, establishing a first channel between a plurality of security service processing processes and a main process includes:
each security service processing process establishes a first channel with a main process when being started, wherein the first channel comprises a data packet channel;
establishing a second channel between the plurality of work processes and the plurality of security service processing processes, comprising:
and a second channel is established between each security service processing process and each working process, and comprises a DPDK lock-free queue.
Optionally, the service data recorded when the security service processing process processes a session message is private data of the corresponding session message and is associated with the corresponding session message;
the electronic device further includes:
the security service processing module is used for determining a data management mode based on the memory occupation condition and independently maintaining private data and session messages associated with the private data based on the determined data management mode.
Optionally, the data management manner includes:
applying for a memory pool when each security service processing process is initialized;
determining an ID address of the session message and a first address of a private data memory pool;
storing private data of the corresponding session message into a corresponding memory pool, and establishing a quick index based on the ID address and the first address;
and verifying the validity of the private data based on the session age of the session message.
Optionally, the data management manner includes:
storing private data corresponding to the session message into a target memory corresponding to the security data processing process;
establishing a hash chain table for a target memory;
determining an ID address and a session age of the session message;
the hash index value is calculated based at least on the ID address of the session message and the session age.
Optionally, the security service processing module is further configured to:
and obtaining notification information sent after the working process deletes the session message, and releasing corresponding private data based on the notification information.
Optionally, the electronic device further includes a working module:
the method comprises the steps of establishing a scheduling table for recording ages locally, keeping the ages of the scheduling table consistent with the ages of a state table of a security processing service process recorded in a main process, and performing security service processing policy matching on a newly-built session message according to a policy issued by a command line of the main process; determining a target security business processing process based on the matching result and the scheduling table; recording the ID address of the target security service processing process in the newly built session message; when the target security service processing process finishes the processing of the new session message, the processed new session message is transferred back to the working process; and executing route forwarding on the received newly-built session message.
Optionally, the working module is further configured to:
and determining whether the current operation scene is a target scene, if so, forwarding the message to a security service processing process, and directly executing the routing forwarding of the message, wherein the target scene comprises restarting of the security service processing process, reaching of the configured threshold value by the total number of the target messages, and reaching of the full load by the second channel.
Another embodiment of the present application also provides an electronic device, including:
one or more processors;
a memory configured to store one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the above-described methods of processing.
An embodiment of the present application also provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the processing method as described above. It should be understood that each solution in this embodiment has a corresponding technical effect in the foregoing method embodiment, which is not described herein.
Embodiments of the present application also provide a computer program product tangibly stored on a computer-readable medium and comprising computer-readable instructions that, when executed, cause at least one processor to perform a processing method such as in the embodiments described above. It should be understood that each solution in this embodiment has a corresponding technical effect in the foregoing method embodiment, which is not described herein.
The computer storage medium of the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage media element, a magnetic storage media element, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, antenna, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
It should be understood that although the present application has been described in terms of various embodiments, not every embodiment is provided with a single embodiment, and the description is provided for clarity only, and those skilled in the art will recognize that the embodiments may be suitably combined to form other embodiments as would be understood by those skilled in the art.
The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application, the scope of which is defined by the claims. Various modifications and equivalent arrangements of this application will occur to those skilled in the art, and are intended to be within the spirit and scope of the application.