Background
With the continued development of the internet, the security threat of computer networks has increased dramatically over the last decade, and the importance of network security has become a major issue. Meanwhile, in a complex network environment, the attack behavior which is beneficial to obtain the privacy data is layered, the destructiveness is more and more powerful, and thus an intrusion detection system (Intrusion Detection System, IDS) aiming at various network attacks is generated. A Network-based intrusion detection system (Network-Based Intrusion Detection System, NIDS) is an attack detection system that guarantees the security of the system by continuously monitoring malicious or suspicious behaviors in Network traffic.
Network intrusion detection technology has been researched by many students from the proposal to the application and development. The continuous development and research of intrusion detection technology also becomes an important force for promoting the development of the space security of the new generation network, and the world-wide security operation of the network is better maintained. Zhang Xinglan et al propose an intrusion detection model for a variable fused random attention capsule network, by which the model can better capture data features. However, this method has problems such as data leakage, privacy security, and time-consuming training while achieving a high accuracy (document 1).
Federal learning (FEDERATED LEARNING, FL) is used as a distributed machine learning framework based on privacy protection, so that the problem of 'data island' is solved, the data security can be ensured, the potential value of the data is fully mined, and the network intrusion detection model has high performance and privacy security. Meanwhile, the distributed intelligent training mode can improve the overall performance and robustness of the intrusion detection system, especially when large-scale network data are processed.
Popoola et al propose an FDL, a joint deep learning method, which adopts a federal learning framework, ensures the privacy security of data, and has lower communication overhead, network delay and storage space. The performance similar to that of the centralized method is realized while the data privacy is ensured in the detection of the zero-day botnet attack. Although this method improves system performance and ensures data privacy, the method cannot detect abnormal traffic in a real network environment in real time (document 2).
De Oliveira et al propose F-NIDS, a distributed intrusion detection system that overcomes privacy problems in distributed scenarios by evaluating the robustness of performance and accuracy, precision, and recall indicators. The system is FL-based and aims to provide a distributed architecture that does not require the exchange of client data, as it eliminates the transmission of single client information. Although this method can effectively solve the problem of data privacy, and maintain higher detection accuracy, the deployment and maintenance costs of this method are higher (document 3).
A software defined network (Software Defined Network, SDN) as a new network architecture ameliorates the shortcomings of traditional networks and is considered a technique that can effectively manage the entire network and transform complex network architecture into simple and easy to manage. SDN separates a network control plane and a forwarding plane, and network intelligence is centralized in a controller, so that the SDN controller is responsible for management of the whole network, and underlying infrastructure such as a switch operates based on instructions deployed by the SDN controller. Compared with the traditional network, the SDN has faster strain resistance, so the deployment of the NIDS in combination with the SDN can quickly formulate emergency measures when the network is attacked.
Li Chuanhuang et al propose a DDoS (Distributed Denial of Service, DDoS) attack detection method based on a deep learning hybrid model, which has higher accuracy and can shorten the processing time of classification detection. The detection input features of the method are the flow table features in the SDN switch and the self-constructed flow table features, so that the method belongs to a lightweight detection method and can be directly deployed in an SDN controller. However, the method can only be applied to single DDoS attack detection, and cannot realize the comprehensive detection of abnormal traffic in the network (document 4).
Zainudin et al describe a low complexity intrusion detection and classification method using federal learning in an SDN based environment. In contrast to conventional collaborative learning, which may compromise privacy when distributing local training data to a central server, by performing local training in each SDN controller, an aggregation server may aggregate training parameters from the respective SDN controllers using a joint average algorithm, thereby effectively protecting the privacy of the network data. The deep neural network (Deep Neural Network, DNN) model is used for training on each controller and is used for detecting and classifying abnormal network flows, and the accuracy of detection after the method is trained for many times is greatly improved. This method increases communication overhead because the switch needs to constantly transmit a large amount of complete packet information to the controller for model training (document 5).
Liu Yanhua et al propose a weighted aggregation algorithm based on federal incremental learning, combine LSTM (Long ShortTerm Memory, LSTM) to realize DDoS attack real-time defense decision in SDN environment, and realize real-time issuing of flow rules according to DDoS attack detection results and network entity information, so as to achieve the effect of effectively blocking DDoS attack flow. Although the method well solves the problem of independent and uniform distribution of data caused by data increment, the method does not consider more application scenes and cannot detect more different types of malicious attacks (document 6).
Network data plane solutions in SDN aim to take advantage of hardware speeds and the latest flexibility of programmable switches. However, these solutions are not designed with the limitations of flow table size and memory space in mind, and the channels between the data plane and the control plane often present severe bandwidth and delay bottlenecks, compromising normal SDN network operation and affecting timely response to ongoing network attacks. The recent advent of programmable data planes has introduced a technology that provides a promising solution to the above challenges. The programmable data plane allows the data plane pipeline (referred to as fastpath) to be customized using a domain-specific language (e.g., P4). P4 (Programming Protocol-INDEPENDENT PACKET Processors) is a Domain Specific Language (DSL) for the programmable data layer of a network switch, a language for modifying the packet forwarding behavior of an SDN switch, which is used to program hardware as is the Verilog and VHDL languages, but unlike Verilog and VHDL, one does not need to know the details of the underlying hardware. The P4 open source language brings possibility to the programmability of the SDN data plane, the characteristic of the data plane can lead the data plane to completely get rid of the constraint of the fixed function of the traditional forwarding equipment, and a user can completely customize the forwarding processing flow of the data packet to realize the efficient processing of the data packet.
Doriguzzi-Corin et al propose a P4DDLe framework, which is a framework for efficient packet-level feature extraction and preprocessing on a P4-based programmable data plane. The scheme can meet the requirement of NIDS based on ML (MACHINE LEARNING, ML), ensures accurate detection of DDoS attacks, and reduces the opportunity of missing detection of malicious streams due to lack of memory space, but the detection type of network attacks is limited (document 7). Xavier et al propose a MAP4 framework that relies on the P4 language to deploy a pre-trained ML model into a programmable switch, and the ML model as a classifier can better fit data by using the programmability of the P4 language, resulting in a few packets required to accurately classify flows. However, the method can only offload a simple machine learning model to a data plane to realize stream classification, and detection accuracy is low (document 8).
In summary, at present, there is no solution capable of flexibly deploying and implementing efficient and intelligent detection of abnormal traffic and network attack types in a network environment, and the existing related solutions have the problems of data leakage, high time consumption of model training, high deployment and maintenance costs, single detection type, low accuracy and the like.
Literature 1: zhang Xinglan, yinlin variable fused random attention capsule network intrusion detection model [ J ]. Communication journal, 2020,41 (11): 160-168.
Literature 2：Popoola S I,Ande R,Adebisi B,et al.Federated deep learning for Zero-Day botnet attack detection in IoT-Edge devices[J].IEEE Internet of Things Journal,2022,9(5):3930-3944.
Literature 3：De Oliveira J A,Goncalves V P,Meneguette R I,et al.F-NIDS—A network intrusion detection system based on federated learning[J].Computer Networks,2023,236(1):1-14.
4: Li Chuanhuang, wu Yan, qian Zhengzhe, et al DDoS attack detection and defense based on deep learning hybrid model under SDN [ J ]. Communicator 2018,39 (7): 176-185.
Literature 5：Zainudin A,Akter R,Kim D S,et al.Federated learning inspired low-complexity intrusion detection and classification technique for sdn-based industrial cps[J].IEEE Transactions on Network and Service Management,2023,20(3):2442-2459.
Literature 6: liu Yanhua, fang Wen, guo Wenzhong, et al DDoS attack detection model under SDN environment based on federal incremental learning [ J ]. Computer science report 2024,47 (12): 2852-2866.
Literature 7：Doriguzzi-Corin R,Knob L A D,Mendozzi L,et al.Introducing packet-level analysis in programmabledata planes to advance network intrusion detection[J].Computer Networks,2024,239(1):1-14.
Literature 8：Xavier B M,Guimaraes R S,Comarela G,et al.MAP4:A pragmatic framework for in-network machine learning traffic classification[J].IEEE Transactions on Network and Service Management,2022,19(4):4176-4188.
Disclosure of Invention
The invention provides an intelligent high-efficiency network intrusion detection system, method and product based on federal learning and P4, which are used for solving the technical problems of high data leakage, high time consumption for model training, high deployment and maintenance cost, single network attack detection type, low accuracy and the like in the prior art.
The method adopts the technical scheme that the network intrusion detection system based on federal learning and P4 comprises an abnormal flow detection module and an abnormal flow classification module;
The abnormal flow detection module is used for extracting data packet characteristic information in network flow, calculating a flow reconstruction loss value according to preset parameters, comparing the reconstruction loss value with a benign threshold and an attack threshold to judge whether the flow is abnormal flow or not;
The abnormal flow classification module calculates the energy value of each flow, and compares the energy value of each flow with the energy threshold value of the flow class so as to detect the network attack type.
Preferably, the abnormal flow detection module comprises a feature extraction sub-module, a counting bloom filter, a flow feature processing sub-module, a depth self-encoder, a parameter setting sub-module and a forwarding decision sub-module;
the characteristic extraction submodule analyzes the data packets by using a P4 analyzer, the P4 analyzer performs Ethernet analysis on all the data packets, stores the analyzed header data and metadata, transfers the analysis state to Accept according to the type value of the Ethernet header, and finally enters the Accept state after the data packet analysis is completed;
the flow characteristic processing submodule is used for processing flow characteristics by using a classified EA algorithm, generating a new field through fitting, and adding the new field into the original flow data;
the depth self-encoder is used for generating parameters and thresholds, including benign thresholds and attack thresholds, and consists of an encoder layer, a bottleneck layer and a decoder layer which are sequentially connected, wherein the encoder layer, the bottleneck layer and the decoder layer are all neural networks.
Preferably, the preset parameter, the benign threshold and the attack threshold are all obtained by training the abnormal flow detection module;
The training process comprises the following steps:
(1) Constructing a training data set, extracting data packet characteristic information in network flow by the training data through an abnormal flow detection module, verifying whether the number of data packets belonging to the same flow is smaller than a preset value through a counting bloom filter, and if so, reserving characteristic field information of the data packets;
(2) The flow characteristic processing module processes the flow characteristic by using a binary EA algorithm, generates a new field through fitting, and adds the new field into the original flow data;
(3) The depth self-encoder calculates input data x and output dataThe reconstruction loss value of (1) obtains a threshold value, and benign sample data and attack sample data are respectively reconstructed to obtain a benign threshold value and an attack threshold value;
(4) The optimal parameters and the optimal thresholds are further obtained through training of a federal learning sub-module, wherein the federal learning sub-module is used for obtaining the optimal parameters and the optimal thresholds through FedAvg aggregation algorithm, distributed training and multiple rounds of training and aggregation.
Preferably, the calculated flow reconstruction loss valueWhere N refers to the number of flow-based features associated with each sample xi in the dataset, xi represents the initial value of each feature,Representing the value of the feature xi after reconstruction from the encoder.
Preferably, the abnormal flow classification module comprises a register, a model reasoning sub-module and an abnormal flow classification sub-module;
the model reasoning submodule performs feature extraction and processing on input data, calculates a coupling value and a local field value of the input data respectively, and then obtains an energy value of the input data;
The abnormal flow classification sub-module calculates the energy value of each class of flow according to the model reasoning sub-module, finds the minimum energy value in each class of flow and compares with the trained threshold value in the model reasoning sub-module, each attack flow corresponds to a training threshold value, and judges the attack class to which the attack flow belongs according to the comparison result of the minimum energy value and the threshold value of the class flow.
Preferably, the trained threshold in the model reasoning sub-module takes the nth% potential value of the energy value of each class of flow as the threshold, and N is a preset value.
Preferably, the energy value of each flowWherein,Eij(dki,dkj) represents the coupling value, hi(dki) the local field value, N the number of streams, dkN the energy value of the nth stream, and dki and dkj the values of feature i and feature j, respectively, for the set of all possible flows.
Preferably, the system further comprises an abnormal data flow real-time display module, an abnormal flow monitoring effect display module, a network attack type detection effect display module and a network flow collection effect display module;
The abnormal data flow real-time display module is used for displaying the specific information of the abnormal data flow detected by the abnormal flow detection module;
the abnormal flow monitoring effect display module is used for displaying the abnormal flow detection effect in the network flow;
the network attack type detection effect display module is used for displaying each attack type and each attack quantity detected by the abnormal flow classification module;
the network flow collection effect display module is used for displaying the effect of the abnormal flow detection module for collecting network flow.
The technical scheme adopted by the method is that the network intrusion detection method based on federal learning and P4 comprises the following steps:
When network traffic reaches a programmable switch, an abnormal traffic detection module extracts data packet characteristic information in the network traffic, verifies that the number of data packets belonging to the same flow through a counting bloom filter, and if the number of the data packets is smaller than a preset value, reserves characteristic field information of the data packets;
Step 2, the abnormal flow detection module calculates a flow reconstruction loss value according to preset parameters, compares the reconstruction loss value with a benign threshold and an attack threshold to judge whether the flow is abnormal, if so, forwards the flow, and if so, submits the flow to the abnormal flow classification module;
step3, the abnormal flow classification module writes the characteristic field value of the abnormal flow through a register, and the abnormal flow classification sub-module classifies the abnormal flow and detects the type of network attack;
and 4, displaying abnormal data flow information, abnormal flow detection effect, network attack type detection effect and network flow collection effect in a visual mode.
The network intrusion detection product based on federal learning and P4 comprises computer program instructions which, when run on a computer, cause the computer to execute the network intrusion detection method based on federal learning and P4.
Compared with the prior art, the intelligent high-efficiency network intrusion detection scheme based on federal learning and P4 has the advantages that the intelligent high-efficiency network intrusion detection scheme based on federal learning and P4 is realized, the flow in a network can be efficiently collected by combining the filtering mechanism of a counting bloom filter, the intelligent high-efficiency network intrusion detection scheme is realized by software and is directly deployed on programmable switches and servers in various network environments, the cost is reduced, the maintenance and the upgrading of a system are easier, meanwhile, the generalization capability of the intelligent high-efficiency network intrusion detection scheme is improved by combining an energy flow classifier and a deep self-encoder by utilizing a federal learning frame, the model training time is shortened, the intelligent model aggregation is realized while the data privacy is protected, the comprehensiveness and the accuracy of network intrusion detection are remarkably improved, and a new solution is provided for constructing an intelligent and high-efficiency network protection system.
Detailed Description
The following examples, as well as specific embodiments, are used to further illustrate the technical aspects of the present invention. In addition, in the course of describing the technical solutions, some drawings are also used. Other figures and the intent of the present invention can be derived from these figures without inventive effort for a person skilled in the art.
The technical terms used in this embodiment will be explained and explained first:
Data plane in SDN architecture based on programmable switch, through module of data packet processing logic and forwarding behavior defined by programmable language, dynamic protocol analysis and custom behavior are supported, usually at hardware layer of network device.
Control plane-centralized module that manages network resources, issues rules, and dynamically configures data plane logic, typically at a centralized controller or server.
And the application plane is used as a user interaction interface in the network intrusion detection system and is responsible for collecting information from the data plane and the control plane, displaying specific information of abnormal flow, detection effect, flow trend and the like in a visual mode, and helping a user monitor and analyze network security conditions in real time.
The abnormal flow detection module is one module in the network intrusion detection system and is used for detecting abnormal flow in the network.
The abnormal flow classification module is one module in the network intrusion detection system and is used for classifying the abnormal flow in the network and detecting the network attack type.
The P4run time API is a standardized control plane interface protocol for interacting and managing with network devices supporting the P4 programming language, the control plane interacting with the data plane through remote procedure calls provided by the P4run time API.
Five key attributes of network traffic, including source IP address, destination IP address, transport layer protocol type, source port and destination port.
The counting bloom filter is a probability data structure with high space efficiency, supports the inquiry of the collection members and is used for counting the number of data packets in the network traffic.
Registers, which are state-able memories in programmable switches, whose values can be read and written by operations, can be used more widely to preserve the state of data.
The inverse Potts model is a statistical mechanical model used to study interactions in a multi-spin system, where the energy of the interactions of the spins depends on the angular difference between them. The inverse Potts model works by defining a multi-state system in which each state represents a different class of traffic.
The EFC multi-classifier is an energy-based flow classification algorithm, which is based on inverse statistics, calculates the energy value of each flow sample according to an inferred statistical model to realize the effective classification of network flows.
Coupling value-in network flow analysis, a parameter that describes interactions or associations between different flows. In the energy calculation of network flows, coupling values are typically used to measure the correlation between network flows, which reflects the interplay of different flows in the process.
Local field value-a parameter used to describe the influence of a single network flow by other flows in the network environment. The local field value typically quantifies in some way the local influence of the network state in which the node or stream is located, similar to the local field of the spin system in physics, which represents the external influence or change in local environment to which the point is subjected.
The depth automatic encoder is one of a depth learning system, is a multi-hidden-layer depth neural network structure, and is different from a traditional neural network, and an automatic encoder (Auto-Encoder, AE) contained in the neural network can automatically learn the inherent dependency relationship in data in an unsupervised manner to extract characteristic data. The deep automatic encoder has the advantages that the learning capacity is improved by the layer-by-layer unsupervised pre-training compared with the traditional deep neural network, and satisfactory effects are achieved on a plurality of classification and regression problems (see https:// gitsub.com/rajarsheem/libsdae-autoencoder-tensorflow).
Accuracy rate (Accuracy) represents the ratio of the number of correctly classified normal and abnormal samples to the total number, and is an evaluation index in the classification task of detecting abnormal flow, and is defined as follows:
Wherein the parameters TP, TN, FP and FN are explained as follows:
TP (True Positive) the sample real category is positive, and the prediction result is positive.
TN (True Negative) the sample true category is negative, and the prediction result is negative.
FP (False Positive) the sample real category is negative, and the prediction result is positive.
FN (False Negative) the sample real category is positive, and the prediction result is negative.
Accuracy (Precision) is the proportion of the actual attack record in all the predicted attack records, and is an evaluation index in the classification task of the flow abnormality detection, and is defined as follows:
Recall rate (Recall), which represents the proportion of correct classification in the actual attack record, is also known as True Positive Rate (TPR) or sensitivity, and is an evaluation index in such classification tasks for flow anomaly detection, defined as follows:
f1 Score F1 Score (F1 Score) is a result of comprehensively considering recall and accuracy, and is an evaluation index in such classification tasks for flow anomaly detection, which is defined as follows:
flow collection Rate (Collected Flows Rate) is a measure of the ratio of the amount of traffic stored in the switch memory to the total number of traffic injected into the switch in a given time window, through which the system's ability to capture traffic information in the network can be understood in depth.
Please refer to fig. 1, the network intrusion detection system based on federal learning and P4 provided in this embodiment is applied in an abnormal traffic detection and network attack type detection situation in a network environment, where the situation includes five entities, namely, a control plane, a data plane, an application plane, an abnormal traffic detection module, and an abnormal traffic classification module, when the network traffic reaches a programmable switch, the abnormal traffic detection module processes the network traffic, the data plane extracts characteristic information of traffic in the network, only retains characteristic information of data packets required for training a control plane model through a counting bloom filter, sets parameters to efficiently detect and execute forwarding decisions, trains a control plane construction model and transmits parameters and a threshold to the data plane, the abnormal traffic classification module processes the abnormal traffic, the data plane writes characteristic field values of the abnormal traffic, the control plane detects types of network attacks in an abnormal traffic classification mode, and the control plane and the data plane realize interaction through a P4Runtime API, and the application plane shows abnormal data flow information, an abnormal traffic detection effect, a network attack type detection effect and a network traffic collection effect.
In one embodiment, the abnormal flow detection module comprises a feature extraction sub-module, a counting bloom filter, a flow feature processing sub-module, a depth self-encoder, a parameter setting sub-module and a forwarding decision sub-module;
the characteristic extraction submodule analyzes the data packets by using a P4 analyzer, the P4 analyzer performs Ethernet analysis on all the data packets, stores the analyzed header data and metadata, transfers the analysis state to Accept according to the type value of the Ethernet header, and finally enters the Accept state after the data packet analysis is completed;
the flow characteristic processing submodule is used for processing flow characteristics by using a classified EA algorithm, generating a new field through fitting, and adding the new field into the original flow data;
the depth self-encoder is used for generating parameters and thresholds, including benign thresholds and attack thresholds, and consists of an encoder layer, a bottleneck layer and a decoder layer which are sequentially connected, wherein the encoder layer, the bottleneck layer and the decoder layer are all neural networks.
In one embodiment, the preset parameters, benign threshold, and attack threshold are all obtained by training the abnormal traffic detection module;
The training process comprises the following steps:
(1) Constructing a training data set, extracting data packet characteristic information in network traffic by the training data through an abnormal traffic detection module, verifying whether the number of data packets belonging to the same flow is less than 3 through a counting bloom filter, if so, retaining characteristic field information (including EtherType, protocol, IPv4DF, IPv4MF, tcpDstPort, udpDstPort, packSize, tcpFIN, tcpSYN, tcpRST, tcpPSH, tcpACK and TcpURG) and quintuple information of the data packets, and filtering redundant data packets;
In one embodiment, using Mininet as an SDN network simulation experiment platform, creating a simulated network topology as shown in fig. 2 to simulate an enterprise campus network, selecting BMv as a programmable switch (S1,S2,S3), selecting a server configured as a 36-core Intel (R) Xeon (R) Gold 6240C CPU (2.60 GHz) and 128GB RAM as a controller (C1,C2); host H1、H2 and switch S1 represent research and development in the campus network, host H3、H4 and switch S2 represent human resources in the campus network, host H4、H5 and switch S3 represent administration in the campus network, in this embodiment taking research and development departments as examples, controllers C1 and C2 represent control planes, switches S1、S2 and S3 represent data planes, the control planes and data plane functions being performed on the controllers and switches, respectively, and public dataset CICIDS2017 (see https:// www.unb.ca/cic/datasets/ids-2017. Html) is generated by one of the collaborative projects between the canadian communications security institute and the canadian network security institute, including the latest attacks and benign traffic, providing an environment similar to that of a real network, dataset being a PCAP file generated on a test platform, split into two networks (victims and aggressors), including most types of attacks, using the dataset to simulate traffic of different parts.
In one embodiment, as shown in fig. 3-5, the steps specifically include the sub-steps of:
The data plane extracts the characteristic information of the data packet in the network flow, and the analyzer receives the incoming data packet and extracts the flow header field and the metadata for verification and verification;
In one embodiment, host H1 sends a data packet P in data set CICIDS2017 to host H2, and when P reaches switch S1, the switch extracts the header fields and metadata of the data packet through the parser of the data plane, and if the data packet passes Checksum verification, the components in the ingress pipe of the data plane are responsible for storing traffic characteristics and other metadata in the stateful memory.
The method comprises the steps of 1.2, calculating the number of data packets belonging to the same flow collected in a specific time by a data plane through a counting bloom filter, and checking whether the characteristic information of the data packets is kept in a memory after the probability technology verification based on a hash algorithm (CRC 32 algorithm), wherein a parser acquires the information of the data packets P and extracts quintuple including a source IP address (IPsrc), a destination IP address (IPdst), a transport layer Protocol type (Protocol), a source Port (Portsrc), a destination Port (Portdst) and a quintuple identifier MP＝(IPsrc,IPdst,Protocol,Portsrc,Portdst of the defined data packets, calculating the CRC32 value of each identifier, checking whether the calculated position counter value stored in the filter is smaller than the allowed maximum data packet number P or not by the algorithm, and if the condition is met, storing the quintuple identifier and the corresponding characteristic information of the data packets in the memory;
In one embodiment, the parser of the data plane extracts the five-tuple MP = (10.0.0.1, 10.0.0.2, tcp,0, 1) of the data packet, counts the bloom filter size to 32 bits, and the maximum data packet number p=3, and the five-tuple set m= { x, y, z, w }, wherein x＝(10.0.0.1,10.0.0.2,0,1),y＝(0,1,10.0.0.1,10.0.0.2),z＝(10.0.0.1,10.0.0.2,TCP,0,1),w＝(0,1,10.0.0.1,TCP,10.0.0.2), counts the bloom filter to calculate the CRC32 value of four elements in the five-tuple set, modulo the value of 32 to {10,14,23,18}, then maps the element to the corresponding position and increments the counter by 1, and the counting bloom filter process is shown in fig. 4. If the value of the minimum counter is less than 3, the five-tuple identification of the data packet and the corresponding characteristic information thereof are stored in the memory.
In one embodiment, CICIDS2017 data sets are divided into training and testing sets, accounting for 80% and 20%, respectively. The data set is trained using the EA algorithm, and the predicted results (energy values) obtained on the training set and the test set are added to the original training set and the original test set, respectively, as a new feature field (EA) value. The training set with the EA field added is again partitioned into 90% for training and 10% for verification (during depth self-encoder model training), and the test set remains unchanged in size.
(2) Directing attention to fig. 6, the flow characteristic processing module uses a classified EA algorithm to process flow characteristics, and generates a new field through fitting and adds the new field into the original flow data;
(3) The depth self-encoder calculates input data x and output dataThe reconstruction loss value of (1) obtains a threshold value, and benign sample data and attack sample data are respectively reconstructed to obtain a benign threshold value and an attack threshold value;
In one embodiment, the control plane uses a depth auto-encoder construction model of an unsupervised learning algorithm, and the auto-encoder structure consists of three layers, namely an encoder network, a bottleneck portion (also called a latent space or compressed representation), and a decoder network, the three layers being neural networks, the encoder network being responsible for encoding the input data into a low-dimensional representation that captures the main characteristics of the input data, the bottleneck portion being the space of the low-dimensional representation that is the bridge between the encoder network and the decoder network, and the decoder network being tasked with recovering the original data from the low-dimensional representation, as shown in FIG. 7. The process of training a depth self-encoder aims at minimizing reconstruction lossesI.e. calculate all given inputsTo output toWhere N is the number of flow-based features associated with each sample x in the dataset, the input data is training data composed of data processed by flow-based features based on the EA algorithm during training of the self-encoder, the input data used for minimum distance calculation during training is the same as the input data here, and the mean square error (Mean Squared Error, MSE) is used as a loss function during trainingWherein Xo,i represents an observed value, Xr,i represents a true value, and n represents the number of observations;
training the model constructed by the depth self-encoder to generate two thresholds, namely a benign threshold (TB) and an attack threshold (TA), calculating the distance between the reconstruction loss value of the test sample and the two thresholds, determining the reconstruction loss value as an attack sample if the reconstruction loss value is close to the attack threshold, otherwise, determining the reconstruction loss value as a benign sample, and calculating and reasoning the reconstruction loss value considered in the calculation and reasoning process as an average absolute error (Mean Absolute Error, MAE), wherein the calculation formula is thatBenign samples b= { xm, benign }, attack samples a= { xn, attach }, m represents the benign number of samples, n represents the Attack number of samples, m+n represents the verification lumped number of samples, these thresholds are derived by mean absolute error computation (MAE) of the verification subset and local inference by the self-encoder, i.e.
In one embodiment, the parameters of the depth self-encoder used are shown in the following table,
Given the number of input samples x= {3.2,2.8,4.1,5.5,3.9,6.2,2.5,7.1,4.3,5.8}, reconstruct the dataN=10, and the reconstruction loss value mae=0.14 is calculated.
(4) The optimal parameters and the optimal thresholds are further obtained through training of a federal learning sub-module, wherein the federal learning sub-module is used for obtaining the optimal parameters and the optimal thresholds through FedAvg aggregation algorithm, distributed training and multiple rounds of training and aggregation.
In one embodiment, the control plane provides FedAvg an aggregation algorithm, distributed model training is performed on four clients respectively, each client consists of a data set and a deep self-encoder model architecture, each client acquires the model architecture from a global server in the initialization process, the model architecture has random weights, and a federal average (FEDERATED AVERAGING, FEDAVG) algorithm is used for distributed training on a plurality of clients and generates a new global model by aggregating model updates of the clients. The number of clients K is indexed by K, B is the size of the local small batch data, and E is the round of local training. The global parameters w0 are initialized at the server side, global updates t=1, 2,3 for each round are performed. Where C is a constant, representing participation rate. Then randomly selecting m clients from K clients to form a set St, calling a client update function for each client K epsilon St in parallel, acquiring model parameters Wk+1 of the client K after local training, collecting data quantity nk of the client K, and finally calculating a global modelWherein the method comprises the steps of
In one embodiment, distributed training is performed on four clients, the clients traverse the local data 10 times, each batch contains 128 samples, the data volume of the clients nk = {100,150,200,120}, the model parameters after local trainingUpdate parameter wk+1 of global model=0.55.
In one embodiment, the abnormal flow classification module comprises a register, a model reasoning sub-module and an abnormal flow classification sub-module;
the model reasoning submodule performs feature extraction and processing on input data, calculates a coupling value and a local field value of the input data respectively, and then obtains an energy value of the input data;
The abnormal flow classification submodule calculates the energy value of each class of flow according to the model reasoning submodule, finds the minimum energy value in each class of flow and compares with the trained threshold value in the model reasoning submodule, each attack flow corresponds to a training threshold value, and judges the attack class to which the attack flow belongs according to the result of comparing the minimum energy value and the threshold value of the class flow.
In one embodiment, the model reasoning sub-module uses the 95% bit value of the energy value of each class of flow as the threshold value.
In one embodiment, the energy value of each streamWherein,Eij(dki,dkj) represents the coupling value, hi(dki) the local field value, N the number of streams, dkN the energy value of the nth stream, and dki and dkj the values of feature i and feature j, respectively, for the set of all possible flows.
In one embodiment, the system further comprises an abnormal data flow real-time display module, an abnormal flow monitoring effect display module, a network attack type detection effect display module and a network flow collection effect display module;
The abnormal data flow real-time display module is used for displaying the specific information of the abnormal data flow detected by the abnormal flow detection module;
the abnormal flow monitoring effect display module is used for displaying the abnormal flow detection effect in the network flow;
the network attack type detection effect display module is used for displaying each attack type and each attack quantity detected by the abnormal flow classification module;
the network flow collection effect display module is used for displaying the effect of the abnormal flow detection module for collecting network flow.
The embodiment also provides a network intrusion detection method based on federal learning and P4, which comprises the following steps:
When network traffic reaches a programmable switch, an abnormal traffic detection module extracts data packet characteristic information in the network traffic, verifies that the number of data packets belonging to the same flow through a counting bloom filter, and if the number of the data packets is smaller than a preset value, reserves characteristic field information of the data packets;
Step 2, the abnormal flow detection module calculates a flow reconstruction loss value according to preset parameters, compares the reconstruction loss value with a benign threshold and an attack threshold to judge whether the flow is abnormal, if so, forwards the flow, and if so, submits the flow to the abnormal flow classification module;
In one embodiment, the flow reconstruct loss valueWhere N refers to the number of flow-based features associated with each sample xi in the dataset, xi represents the initial value of each feature,Representing the value of the feature xi after reconstruction from the encoder.
In one embodiment, as shown in fig. 3, step 2 specifically includes the following sub-steps:
Step 2.1, after extracting the data packet characteristics, the data plane calculates a reconstruction loss value according to parameters provided by the control plane;
Step 2.2, comparing the reconstruction loss value with a benign threshold value and an attack threshold value trained by a control plane to judge whether the reconstruction loss value is abnormal traffic;
Step 2.3, if the flow is normal, forwarding, and if the flow is abnormal, submitting the abnormal flow to an abnormal flow classification module;
In one embodiment, benign threshold TB =21.35, attack threshold TA =30.28, and the data plane calculates the reconstruction loss value Sloss =30.2 of the packet, and the packet is detected as an abnormal packet due to |sloss-TB|＞|Sloss-TA |, and the characteristic information is stored in the register.
The data plane of the abnormal flow classification module writes the characteristic field value of the abnormal flow through a register;
in one embodiment, as shown in fig. 8, step 3 specifically includes the following sub-steps:
Step 3.1. The control plane derives an inverse statistical model for abnormal traffic classification to detect the type of network attack based mainly on the coupling values and local field values, the model using a pattern data structure based on lattice spin to preserve the complete characteristic information of abnormal traffic, the main idea comprising extracting the statistical model from benign traffic samples to infer the coupling values and local field values representing such traffic, here mainly by means of the idea of the Potts model, the N-tuple of a given characteristic being denoted (D1…DN), which can be instantiated (Dk1…dkN) for traffic k, where Dk1∈θ1,...dkN∈θN. Each feature dki is encoded with an integer in the set θ= {1,2,..q } i.e. all feature alphabets are identical, of size Q. If a feature can only take on M values and M < Q, then the values from M+1 to Q are considered possible, but not actually observed. For example, assuming that a feature "protocol" only has the possible value { 'TCP', 'UDP', '4', then it maps to { 'TCP', '1', 'UDP', '2', '3', and '4', where the feature values 3 and 4 do not appear, the formula for the coupling value is shown as follows,
Wherein Cij(di,dj)＝fij(di,dj)-fi(di)fj(dj),fij(di,dj) is the combined empirical frequency of the values di and dj of the features i and j, fi(di) is the empirical frequency of the value di on the feature i, fj(dj) is the empirical frequency of the value dj on the feature j, η is the feature set, the calculation formula of the local field values is shown as follows,
Calculating the frequency and paired frequency of each class of stream, and calculating the energy value formula of each stream according to the coupling value and the local field asA set of all possible flows;
In one embodiment, the flow k is instantiated as (d 1, d2, d 3), the feature set θ= {1,2,3}, the single empirical frequency f1(1)＝0.2,f2(2)＝0.5,f3 (3) =0.3, the total empirical frequency f1(Q)＝1,f2(Q)＝1,f3 (Q) =1, the coupling matrixFor each feature, a local field value is calculated to be h1(1)＝-0.5,h2(2)＝-0.3,h3 (3) = -0.1, a coupling value is calculated to be E12(1,2)＝-0.3,e13(1,3)＝-0.1,e23 (2, 3) = -0.2, and an energy value e=0.3 is finally calculated from the coupling value and the local field value.
Step 3.2 using an anomaly traffic based multi-classifier EFC, the energy of a given traffic can be calculated based on its eigenvalue and model parameters, the energy of a traffic being the inverse of the sum of its eigenvalue and local field, the network attack type detection being actually a multi-classification of anomaly traffic, in which case the model deduces a number of distributions, one for each traffic class. And calculating the flow energy in each distribution, comparing the values, returning a classification result, and carrying out model training process of the EFC classifier as shown in figure 9, wherein model inference and threshold calculation are required to be carried out on each flow class in the multi-classification model, namely, a coupling value and a local field value are calculated, and a statistical threshold is designed. The model for each class is then stored for use in the classification stage, FIG. 10 shows an EFC multi-class test procedure, in which each model is induced to calculate its energy in the training stage to generate a model vector for each instance in order to classify an instance from the test set, the energy of a stream in a distribution being a measure of whether the stream is similar to the set used to infer the distribution, the energy vector of the stream actually comprising a value inversely proportional to the probability of the stream belonging to each class. Thus after calculating the energy of the stream, EFC takes the lowest energy value lowest and compares it to the class threshold cutoff. If lowset is less than or equal to cutoff, considering that the flows of the category are from the same category, and distributing corresponding attack tags;
In one embodiment, dataset CICIDS2017 is used, which includes attack categories of Bot, brute Force, DDoS, doS, infiltration, injection, and the energy sample set for each type of attack is calculated during the training phase as:
F1＝{0.2,0.3,0.5,0.7,0.8},F2={0.3,0.4,0.6,0.7,0.9},
F3＝{0.1,0.2,0.4,0.5,0.6},F4={0.1,0.3,0.4,0.5,0.7},
F5＝{0.4,0.5,0.6,0.7,0.5},F6={0.2,0.3,0.4,0.5,0.4}。
And selecting 95% of the energy value set as a threshold value, obtaining a threshold value set of cutoff= {0.8,0.9,0.6,0.7,0.5,0.4}, calculating the energy value of a new flow sample to be 0.65 according to the local field value and the coupling value in the test stage, and judging the flow class as DoS attack flow as 0.6<0.65< 0.7.
And 4, displaying abnormal data flow information, abnormal flow detection effect, network attack type detection effect and network flow collection effect in a visual mode.
The application plane displays the specific information of the detected abnormal data flow, including quintuple information of the data flow and finally identified attack types;
in one embodiment, specific information of the detected abnormal data stream is shown in the following table:
| Source IP address | Destination IP address | Source port number | Destination port number | Protocol type | Attack type |
| 172.17.0.1 | 10.0.0.2 | 0 | 1 | TCP | DDoS |
The application plane displays the abnormal flow detection effect in the network flow, and the evaluation indexes of the abnormal flow detection effect comprise accuracy, precision, recall rate and F1 fraction;
in one embodiment, the abnormal flow detection effect is shown in the following table:
| Data set | Accuracy rate of | Accuracy of | Recall rate of recall | F1 fraction |
| CICIDS | 0.981 | 0.890 | 0.921 | 0.912 |
The application plane shows the effect that the abnormal flow detection module collects network flow;
In one embodiment, the CICIDS2017 dataset is replayed on switch S1 with a detected network flow collection rate of 98.8%.
The application plane displays each attack type and number detected by the abnormal flow classification module;
in one embodiment, the type of each attack flow detected in CICIDS2017 dataset and its number in all attack flows are as shown in the following table:
| Bot | Brute Force | DDoS | DoS | Infiltration | Injection |
| 25.19% | 13.77% | 30.70% | 13.56% | 10.64% | 8.04% |
The embodiment also provides a network intrusion detection product based on federal learning and P4, which comprises computer program instructions, wherein the computer program instructions, when run on a computer, cause the computer to execute the network intrusion detection method based on federal learning and P4.
The invention realizes a network intrusion detection scheme based on federal learning and P4, and uses a counting bloom filter to realize the efficient and comprehensive collection of network traffic on a data plane and reduce the memory load of a switch, processes real-time network traffic data, calculates a reconstruction loss value and compares the reconstruction loss value with a training threshold transmitted by a control plane to realize the efficient detection of abnormal traffic. On a control plane, an unsupervised learning depth self-encoder and a federal learning training model are used, and the distributed intelligent training can improve the overall performance and robustness of the system and effectively protect the network safety and privacy. Meanwhile, on a control plane, an inverse statistical model is built based on an inverse Potts method, complete characteristic information is stored, abnormal traffic is comprehensively and accurately classified by using a multi-classifier EFC, and various types of network attacks are detected.
It should be understood that the embodiments described above are some, but not all, embodiments of the invention. In addition, the technical features of each embodiment or the single embodiment provided by the invention can be combined with each other at will to form a feasible technical scheme, and the combination is not limited by the sequence of steps and/or the structural composition mode, but is necessarily based on the fact that a person of ordinary skill in the art can realize the combination, and when the technical scheme is contradictory or can not realize, the combination of the technical scheme is not considered to exist and is not within the protection scope of the invention claimed.
It should be understood that the foregoing description of the preferred embodiments is not intended to limit the scope of the invention, but rather to limit the scope of the claims, and that those skilled in the art can make substitutions or modifications without departing from the scope of the invention as set forth in the appended claims.