Disclosure of Invention
The application provides a threat detection method and device in a virtual detection environment, which are used for solving the problems in the related art.
In a first aspect, the present invention provides a threat detection method in a virtual detection environment, including extracting internal traffic of the virtual environment from a virtual switch to a probe node; after the probe nodes are processed according to preset rules, the processed data are sent to a remote distributed analysis node, wherein whether threat exists or not is identified at the distributed analysis node; if the threat exists, a control command is issued to the corresponding probe node through the management control node, so that an off-line instruction is issued to the corresponding virtual switch by the corresponding probe node.
Optionally, processing at the probe node according to a preset rule includes: filtering the internal traffic of the virtual environment at the probe node based on a preset filtering strategy; performing session recombination on the filtered flow and packaging; and carrying out elephant flow identification on the packaged conversation package, and processing the conversation package with the identified elephant flow by adopting different processing modes.
Optionally, performing session reorganization and encapsulation on the filtered traffic includes: performing session reorganization according to the network quintuple; and encapsulating the source address and the destination address according to the three-layer IP forwarding agreed by ERSPAN, wherein if the tenant ID is configured, the tenant ID is encapsulated in a reserved field of ERSPAN II and III protocols, and three bytes of the reserved field are used.
Optionally, processing the session packet identifying the elephant flow in a different processing manner includes: discarding the session packet if a preset discarding rule is satisfied; for the non-discarded session packets, determining east-west session traffic data similar to the non-discarded session packets; and importing similar east-west session traffic data to the probe node idle CPU core.
Optionally, after processing by adopting different processing modes, the session identifier and the session sequence number of the session packet to be sent are also identified, and the identifier is put into the tail part of the session packet and encapsulated by ERSPAN protocol.
Optionally, the identifying, at the distributed analysis node, whether a threat exists includes: executing identification tenant ID identification and removing ERSPAN protocol header actions aiming at the session packet obtained by processing; carrying out recombination of protocol layers in sequence; threat identification is performed based on the reorganized data.
In a second aspect, the present invention provides a threat detection apparatus in a virtual detection environment, including a traffic detection unit configured to draw traffic inside the virtual environment from a virtual switch to a probe node; the probe node processing unit is configured to process the probe nodes according to preset rules, then send the processed data to a remote distributed analysis node, and the distributed analysis node processing unit is configured to identify whether a threat exists at the distributed analysis node; if the threat exists, a control command is issued to the corresponding probe node through the management control node, so that an off-line instruction is issued to the corresponding virtual switch by the corresponding probe node.
Optionally, processing at the probe node according to a preset rule includes: filtering the internal traffic of the virtual environment at the probe node based on a preset filtering strategy; performing session recombination on the filtered flow and packaging; and carrying out elephant flow identification on the packaged conversation package, and processing the conversation package with the identified elephant flow by adopting different processing modes.
In a third aspect, the present invention provides a computer readable storage medium storing a computer program which when executed by a processor implements the method of any of the first aspects described above.
In a fourth aspect, the present invention provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method provided in the first aspect when executing the program.
The invention discloses a threat detection method and a threat detection device under a virtual detection environment, wherein the method comprises the steps of leading out internal traffic of the virtual environment from a virtual switch to a probe node; after the probe nodes are processed according to preset rules, the processed data are sent to a remote distributed analysis node, wherein whether threat exists or not is identified at the distributed analysis node; if the threat exists, a control command is issued to the corresponding probe node through the management control node, so that an off-line instruction is issued to the corresponding virtual switch by the corresponding probe node. By using the three-layer structure of the probe, the distributed analysis node and the centralized control node, east-west traffic in a virtualized environment is mainly analyzed, and the defect that threat of east-west network traffic cannot be detected and processed in the related technology is overcome.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
An exemplary method for flattening a live-action three-dimensional model is described below with reference to fig. 1. The method comprises the following steps:
step 101: and leading out the internal traffic of the virtual environment from the virtual switch to the probe node.
In this embodiment, a virtual switch related port is configured to draw the internal traffic of the virtual environment to a related probe node, which is also typically a virtual host, which can sniff network traffic from the related port (virtual network card). The manner of configuring the ports related to the virtual switch is not limited herein, and a virtual switch, a distributed switch manner, or a neutron in OpenStack may be used. The internal traffic of the virtual environment may include east-west network traffic, north-south traffic.
Step 102: and after the probe nodes are processed according to preset rules, the processed data are sent to a remote distributed analysis node, wherein whether threat exists or not is identified at the distributed analysis node.
In this embodiment, the probe node may set different tenant identities by using ERSPAN II and III protocol reservation bits, and may analyze and isolate relevant data of the tenants in the distributed analysis node, so as to support detection and treatment of network threats in a complex environment.
The collected east-west flow can be automatically and uniformly distributed to different distributed analysis nodes by using a specific flow balancing algorithm, and the elephant flow can be automatically identified and intelligently processed or discarded; the probe node sends the data packets after the sequential recombination to the distributed analysis node, so that the recombination efficiency of the distributed analysis node can be accelerated.
Step 103: if the threat exists, a control command is issued to the corresponding probe node through the management control node, so that an off-line instruction is issued to the corresponding virtual switch by the corresponding probe node.
In this embodiment, the three-layer structure of the probe node, the distributed analysis node and the centralized control node is used to analyze the east-west traffic in the virtualized environment: the probe node realizes east-west network traffic collection, data packet filtering, traffic balance sending mechanism, ERSPAN protocol encapsulation capability and specific virtual machine offline capability; the distributed analysis nodes have distributed threat analysis and storage capacity; the centralized management and control node can uniformly manage various strategies, centralize visual processing and offline virtual machines judged to be threatened.
As an optional implementation manner of this embodiment, processing at the probe node according to a preset rule includes: filtering the internal traffic of the virtual environment at the probe node based on a preset filtering strategy; performing session recombination on the filtered flow and packaging; and carrying out elephant flow identification on the packaged conversation package, and processing the conversation package with the identified elephant flow by adopting different processing modes.
In the optional implementation manner, the flow to be filtered is configured at the probe node, and the filtering strategy adopts a BPF or eBPF compatible format and mainly comprises addresses and protocols below a transmission layer; and supports filtering of the application layer protocol, but filtering action for the application layer may be after the probe recognizes the protocol, so there is a problem that the processing performance is affected.
As an optional implementation manner of this embodiment, session reorganization and encapsulation of the filtered traffic includes: performing session reorganization according to the network quintuple; and encapsulating the source address and the destination address according to the three-layer IP forwarding agreed by ERSPAN, wherein if the tenant ID is configured, the tenant ID is encapsulated in a reserved field of ERSPAN II and III protocols, and three bytes of the reserved field are used.
In this optional implementation manner, session reassembly is performed on the filtered network traffic, that is, session assembling is performed according to network five-tuple (source destination address, source destination port, transport layer protocol), and the source address and destination address are encapsulated according to the three-layer IP forwarding agreed by ERSPAN, if a configuration Tenant ID (Tenant ID) is provided, the configuration Tenant ID is encapsulated in a reserved field of ERSPAN II and III protocols, and three bytes of the reserved field are used.
In order to be able to adequately process the relevant east-west traffic data, the probe node needs to be able to identify and perform certain processing for the elephant flow (Elephant Flow).
As an alternative implementation manner of this embodiment, processing the session packet in which the elephant flow is identified by adopting different processing manners includes: discarding the session packet if a preset discarding rule is satisfied; for the non-discarded session packets, determining east-west session traffic data similar to the non-discarded session packets; and importing similar east-west session traffic data to the probe node idle CPU core.
In this alternative implementation, for the case of discard: the method includes the steps of counting and identifying received session data packets by utilizing a data packet counting means, directly guiding a discardable elephant flow to a data packet releasing part, and not processing the data, wherein the counting method is to comprehensively consider the flow rate and the session duration of a certain session unit time, and discard the data if the flow rate and the session duration exceed a certain threshold value:
BPS>th1 and Duration>th2 and TotalBytes>th3
In the above equation, th1、th2、th3 is three thresholds, respectively, and if all are met, they are discarded, BPS is the transmission rate, duration is the Duration of the session in bytes, totalBytes is the cumulative number of bytes transmitted for the session.
For the case of no discard: for the elephant flow which needs analysis and cannot be discarded, reserving 1 to a plurality of CPU cores in the probe node, and after the elephant flow is identified, importing the subsequent elephant flow data into the reserved CPU cores so as to support the continuous processing of the related network data; considering the commonality of the framework, the implementation of the probes and the limitation of different virtualization environments, the processing of the patent is different from the general process of processing the image stream by using the SDN, and a method of re-issuing a flow table is used or NitroSketch is used as INTEL DPDK (depending on hardware, and in order to ensure compatibility, an af-packet mode is used for capturing packets by a probe node).
In the above processing method without discarding, the probe node learns the data packet metadata (i.e. network quintuple and application layer protocol) and partial load content which are determined to be elephant stream by historical correlation, and once east-west session traffic data with similarity exceeding a certain threshold value exists, the data packet metadata is imported to an idle CPU core for processing as soon as possible, namely, is processed by a dynamic CPU affinity (affinity) method, so as not to cause packet loss.
As an optional implementation manner of this embodiment, after processing in different processing manners, the session packet to be sent further identifies the session identifier and the session sequence number, and the identifier is placed at the tail of the session packet and encapsulated by using ERSPAN protocol.
In this alternative implementation, the probe node uses a Hash function (Hash) on the quintuple information after the session reorganization according to the configuration, and before sending these data packets, identifies the session number and the sequence number in the session of the forwarded session data packet with an 8-byte integer data, where the first four bytes are session identifications, and the last four bytes are sequence numbers of the data packets in the session, where the identifications are placed at the tail of each data packet, encapsulated with ERSPAN protocol, and then they are transmitted to the remote distributed analysis node.
As an alternative implementation manner of this embodiment, identifying, at the distributed analysis node, whether there is a threat includes: executing identification tenant ID identification and removing ERSPAN protocol header actions aiming at the session packet obtained by processing; carrying out recombination of protocol layers in sequence; threat identification is performed based on the reorganized data.
In this alternative implementation, after the distributed analysis node receives the data of the related session, it does not need to reorganize the data through the network five-tuple, but directly reorganizes the data through the tail session of each data packet, before that, it needs to execute the actions of identifying the tenant ID and removing ERSPAN the protocol header for the data packets, and then reorganizes the protocol layers for the sessions in sequence, and identifies the possible threat in the session.
For the virtual machine identified as the threat, the management control node directly transmits the virtual machine to the relevant probe node according to the position information of the data transmitted by the probe, and the probe transmits a virtual machine off-line instruction (namely, executes an off-line command, and can transmit the off-line command through an API or a command line) to the relevant virtual switch according to the area where the probe is located.
The embodiment can filter the relevant east-west flow more flexibly, so that the flow is analyzed more selectively, and the flow can be directly filtered at the bottom layer aiming at backup flow, video flow and the like, thereby effectively improving the overall processing efficiency; by using a layered structure, different flows can be effectively shared by using a software method (not hardware), so that the throughput efficiency and the elasticity of the whole system are greatly improved; for the possible multi-tenant situation, the reserved bits of the protocol are utilized for the possible overlapping network IP, so that the processing can be performed in the acquisition part of the virtualized traffic, and the complex virtual environment can be dealt with.
The embodiment overcomes the problem of threat detection for east-west traffic by providing a so-called ERSPAN (Encapsulated Remote Switch Port Analyzer) method by a general network manufacturer, which can send the network traffic of each network port to a remote switch through encapsulation in a tunnel method, but is not applicable to a certain extent for east-west threat detection in a virtual environment or cloud environment, although VMWare can partially solve the problem by using NSX or configuration distributed virtual switch (VDS), there are a plurality of problems, which are expressed in the following aspects:
it is not possible to flexibly process or filter which network traffic is needed, particularly when detecting east-west network traffic, it is necessary to discard network traffic that occupies too much bandwidth and is basically useless, such as some video traffic or known file backup communications, or remove unwanted protocol packets, such as some broadcast packets (Broadcasting) or multicast (Multicasting) packets;
The network data packets cannot be processed flexibly, for example, only the initial partial bytes of the data packets are reserved or only the initial several data packets of each network session are reserved without all but threat detection; in a virtual environment or a cloud environment, if multiple tenants exist, the overlapping IP address segments cannot be distinguished, and the VxLAN cannot be used to package the original data packet (the user deployment environment does not support), but there is a need to distinguish the different tenants; the distributed detection is needed for east-west traffic, and in the case of large traffic, a hierarchical architecture is used to balance the threat detection calculation force, but the consistency of network sessions needs to be ensured (i.e. different data packets of the same session cannot be scattered to different nodes for analysis, otherwise, wrong results or missing report results can be caused); the method can operate in a virtual environment lacking centralized management, such as no VCenter unified management on a plurality of VMWARE ESXi, namely, a remote port of a distributed virtual switch cannot be configured for sniffing traffic, or no virtual computing management platform such as OpenStack and the like is provided; there is a need for convenient management and control of problematic virtual machines, at least to provide offline operations for the virtual machine level.
The application also provides an embodiment of the threat detection apparatus in the virtual detection environment, comprising a flow detection unit configured to draw out the internal flow of the virtual environment from the virtual switch to the probe node; the probe node processing unit is configured to process the probe nodes according to preset rules, then send the processed data to a remote distributed analysis node, and the distributed analysis node processing unit is configured to identify whether a threat exists at the distributed analysis node; if the threat exists, a control command is issued to the corresponding probe node through the management control node, so that an off-line instruction is issued to the corresponding virtual switch by the corresponding probe node.
As an optional implementation manner of this embodiment, processing at the probe node according to a preset rule includes: filtering the internal traffic of the virtual environment at the probe node based on a preset filtering strategy; performing session recombination on the filtered flow and packaging; and carrying out elephant flow identification on the packaged conversation package, and processing the conversation package with the identified elephant flow by adopting different processing modes.
As an optional implementation manner of this embodiment, session reorganization and encapsulation of the filtered traffic includes: performing session reorganization according to the network quintuple; and encapsulating the source address and the destination address according to the three-layer IP forwarding agreed by ERSPAN, wherein if the tenant ID is configured, the tenant ID is encapsulated in a reserved field of ERSPAN II and III protocols, and three bytes of the reserved field are used.
As an alternative implementation manner of this embodiment, processing the session packet in which the elephant flow is identified by adopting different processing manners includes: discarding the session packet if a preset discarding rule is satisfied; for the non-discarded session packets, determining east-west session traffic data similar to the non-discarded session packets; and importing similar east-west session traffic data to the probe node idle CPU core.
As an optional implementation manner of this embodiment, after processing in different processing manners, the session packet to be sent further identifies the session identifier and the session sequence number, and the identifier is placed at the tail of the session packet and encapsulated by using ERSPAN protocol.
As an alternative implementation manner of the embodiment, the identifying, at the distributed analysis node, whether the threat exists includes: executing identification tenant ID identification and removing ERSPAN protocol header actions aiming at the session packet obtained by processing; carrying out recombination of protocol layers in sequence; threat identification is performed based on the reorganized data.
Fig. 2 is a schematic diagram of an entity structure of an electronic device according to an embodiment of the present invention, as shown in fig. 2, an electronic device 50 includes: a processor 501 (processor), a memory 502 (memory), and a bus 503; wherein, the processor 501 and the memory 502 complete the communication with each other through the bus 503; the processor 501 is configured to invoke the program instructions in the memory 502 to perform the methods provided by the method embodiments described above.
The present embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the methods provided by the above-described method embodiments.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: various storage media such as ROM, RAM, magnetic or optical disks may store program code.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the embodiments or the methods of some parts of the embodiments.
While the invention has been described in detail in the foregoing general description and specific examples, it will be apparent to those skilled in the art that modifications and improvements can be made thereto. Accordingly, such modifications or improvements may be made without departing from the spirit of the invention and are intended to be within the scope of the invention as claimed.