CN111950635A - A Robust Feature Learning Method Based on Hierarchical Feature Alignment - Google Patents
A Robust Feature Learning Method Based on Hierarchical Feature AlignmentDownload PDFInfo
- Publication number
- CN111950635A CN111950635ACN202010809932.6ACN202010809932ACN111950635ACN 111950635 ACN111950635 ACN 111950635ACN 202010809932 ACN202010809932 ACN 202010809932ACN 111950635 ACN111950635 ACN 111950635A
- Authority
- CN
- China
- Prior art keywords
- feature
- samples
- features
- adversarial
- hierarchical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
- G06V10/464—Salient features, e.g. scale invariant feature transforms [SIFT] using a plurality of salient features, e.g. bag-of-words [BoW] representations
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明涉及鲁棒机器学习技术领域，具体涉及一种基于分层特征对齐的鲁棒特征学习方法。The invention relates to the technical field of robust machine learning, in particular to a robust feature learning method based on hierarchical feature alignment.
背景技术Background technique
近几年，深度卷积神经网络在如图像分类、目标检测等众多计算机视觉任务上面都取得了突破。然而，研究人员发现这些深度卷积神经网络容易受到那些经过特殊设计的包含人眼不易察觉的对抗扰动样本的欺骗。这些由对抗攻击方法生成的对抗样本给那些对于安全性、稳定性具有较高要求的系统带来了严峻的挑战，这些系统包括自动驾驶系统、医疗诊断系统和安防系统等。另外，如果一个深度网络模型在给了带有少量扰动的样本作为输入就以很高的置信度改变它的预测结果，那么就可以判断这些模型并没有很好的从头输入样本中学习到任务相关的固有属性，也无法从样本中学习到鲁棒的视觉概念。因此，设计对于对抗扰动具有足够鲁棒性的深度网络模型对于安全可靠的计算机视觉应用来说是至关重要的。In recent years, deep convolutional neural networks have made breakthroughs in many computer vision tasks such as image classification and object detection. However, the researchers found that these deep convolutional neural networks were vulnerable to adversarial perturbations that were specially designed to contain samples that were imperceptible to the human eye. These adversarial samples generated by adversarial attack methods bring severe challenges to systems with high requirements for security and stability, such as autonomous driving systems, medical diagnostic systems, and security systems. In addition, if a deep network model changes its predictions with a high degree of confidence when given samples with a small perturbation as input, then it can be judged that these models do not learn task-related well from the input samples from scratch The inherent properties of , and robust visual concepts cannot be learned from samples. Therefore, designing deep network models that are robust enough against perturbations is crucial for safe and reliable computer vision applications.
在近几年的研究工作中，研究人员提出多种对抗防御机制来克服不同的对抗攻击方法。这些防御机制可以被粗略的分为两种类别。第一种类别的防御方法主要采用在输入图像上进行多种预处理来克服对抗攻击。Dziugaite等人和Das等人把JPEG图像压缩作为对抗防御手段。这些方法在输入图像领域中使用离散傅立叶变换来处理对抗噪声。但是，这些基于JPEG像压缩的方法远未达到成功去除对抗噪声的目的。通过充分利用生成对抗网络强大的表示能力，Defense-GAN方法被Samangouei等人提出来防御多种对抗攻击；该方法通过重新生成和输入图像足够相似的图像来达到去除对抗噪声的目地。Mustafa等人提出把图像超分辨作为一种对抗防御的手段，通过把深度超分辨网络作为一个映射函数，该方法把样本从对抗领域映射到正常领域，从而达到去除对抗噪声的目的，最后把映射后的图像输入到图像识别系统中进行正常的识别。另外一种对抗防御手段主要通过修改训练过程或者网络结构来处理对抗扰动从而来提升模型的对抗鲁棒性。对抗训练是一种有效的提升模型对抗鲁棒性的手段，它通过在训练数据中加上特殊设计的对抗样本来达到此目的。Goodfellow等人在干净样本中加入使用FGSM(Fast Gradient Sign Method,快速梯度符号方法)对抗攻击方法生成的对抗样本来训练网络模型。Madry等人使用Min-Max优化方法进行对抗训练，该方法使用PGD(Project Gradient Descent,映射梯度下降)攻击方法产生对抗样本。集成对抗训练也是一种新颖的对抗防御方法，该方法使用从多种不同的深度网络生成的对抗样本作为训练数据来优化模型参数。另外，为了提升深度模型对于对抗样本的泛化能力，Song等人使用领域自适应的方法进行网络模型的训练。In recent years of research work, researchers have proposed multiple adversarial defense mechanisms to overcome different adversarial attack methods. These defense mechanisms can be roughly divided into two categories. The first category of defense methods mainly employs multiple preprocessing on the input image to overcome adversarial attacks. Dziugaite et al. and Das et al. use JPEG image compression as an adversarial defense. These methods use discrete Fourier transforms in the input image domain to deal with adversarial noise. However, these methods based on JPEG image compression are far from successfully removing adversarial noise. By making full use of the powerful representation ability of generative adversarial networks, the Defense-GAN method was proposed by Samangouei et al. to defend against various adversarial attacks; this method achieves the goal of removing adversarial noise by regenerating images that are sufficiently similar to the input image. Mustafa et al. proposed to use image super-resolution as a means of adversarial defense. By using the deep super-resolution network as a mapping function, the method maps samples from the adversarial field to the normal field, so as to achieve the purpose of removing the adversarial noise. Finally, the mapping The resulting image is input into the image recognition system for normal recognition. Another adversarial defense method mainly improves the adversarial robustness of the model by modifying the training process or network structure to deal with adversarial disturbances. Adversarial training is an effective way to improve the adversarial robustness of a model by adding specially designed adversarial examples to the training data. Goodfellow et al. added the adversarial samples generated by the FGSM (Fast Gradient Sign Method, Fast Gradient Sign Method) adversarial attack method to the clean samples to train the network model. Madry et al. used the Min-Max optimization method for adversarial training, which uses the PGD (Project Gradient Descent) attack method to generate adversarial examples. Ensemble adversarial training is also a novel adversarial defense method that uses adversarial examples generated from multiple different deep networks as training data to optimize model parameters. In addition, in order to improve the generalization ability of the deep model to adversarial samples, Song et al. used a domain adaptation method to train the network model.
尽管上述方法在提升深度卷积神经网络的对抗鲁棒性上取得了不错的进展，然而，对于不同种类的白盒攻击方法，受限于模型较差的泛化性能，它们往往无法取得令人满意的结果。Although the above methods have made good progress in improving the adversarial robustness of deep convolutional neural networks, for different kinds of white-box attack methods, they often fail to achieve impressive generalization performance due to the poor generalization performance of the models. Satisfying result.
发明内容SUMMARY OF THE INVENTION
针对现有技术存在的不足，本发明的目的在于提供一种基于分层特征对齐的鲁棒特征学习方法，该方法使得基于深度卷积神经网络的模型可以通过分层特征对齐的操作学到更鲁棒的图像特征，从而解决现有技术存在的针对不同领域对抗样本模型泛化能力受限的问题，为深度模型系统的部署与应用提供有效的可靠性和安全性保障。In view of the deficiencies in the prior art, the purpose of the present invention is to provide a robust feature learning method based on hierarchical feature alignment, which enables a model based on a deep convolutional neural network to learn more through hierarchical feature alignment operations. Robust image features, so as to solve the problem of limited generalization ability of adversarial sample models in different fields in the existing technology, and provide effective reliability and security guarantees for the deployment and application of deep model systems.
为实现上述目的，本发明提供了如下技术方案：一种基于分层特征对齐的鲁棒特征学习方法，包括以下步骤：To achieve the above object, the present invention provides the following technical solutions: a robust feature learning method based on hierarchical feature alignment, comprising the following steps:
(1)从不同领域的样本中使用深度卷积神经网络提起不同层次的深度特征；(1) Use deep convolutional neural networks to extract different levels of deep features from samples in different fields;
(2)对于提取的分层特征，通过图卷积神经网络给特征的通道和空间关系加以限制，从而使得模型学得更加丰富的特征表示；(2) For the extracted hierarchical features, the channel and spatial relationship of the features are restricted by the graph convolutional neural network, so that the model can learn a richer feature representation;
(3)使用基于最优传输理论的Wasserstein distance来准确的度量不同领域样本特征表示之间的差异；(3) Using the Wasserstein distance based on the optimal transmission theory to accurately measure the difference between the feature representations of samples in different fields;
(4)将从不同领域样本中提取的分层特征之间的差异作为模型损失函数的一部分来帮助模型学习更加鲁棒的特征，从而提升深度神经网络模型的对抗鲁棒性。(4) The difference between hierarchical features extracted from samples from different domains is used as part of the model loss function to help the model learn more robust features, thereby improving the adversarial robustness of the deep neural network model.
作为优选的，所述不同领域的图像样本包括正常领域图像样本和对抗领域图像样本。Preferably, the image samples in different fields include normal field image samples and adversarial field image samples.
作为优选的，步骤(1)，使用ResNet-110网络结构来进行图像的特征提取，分为4个不同的结构层次，输入正常样本或对抗样本后，在网络进行正向推理的时，在4个不同的结构层次处，使用卷积结构提取不同尺度、不同抽象程度的图像特征。Preferably, in step (1), the ResNet-110 network structure is used for image feature extraction, which is divided into 4 different structural levels. After inputting normal samples or adversarial samples, when the network performs forward inference, at 4 At different structural levels, the convolutional structure is used to extract image features of different scales and levels of abstraction.
作为优选的，步骤(2)，使用两个一维卷积进行图卷积的操作，该图卷积操作公式化为如下形式：Preferably, in step (2), two one-dimensional convolutions are used to perform a graph convolution operation, and the graph convolution operation is formulated as follows:
公式(1)：Formula 1):
GCN(f)＝Conv1D[Conv1D(f)]GCN(f)=Conv1D[Conv1D(f)]
其中，在公式(1)中，GCN(﹒)表示图卷积神经网络，f表示经过降维处理的特征向量，f表示图卷积操作的输入；此外，Conv1D(﹒)表示一个一维卷积操作，使用两个方向不同的一维图卷积操作进行特征提取，在经过充分的端到端训练后，该图卷积操作加强对于特征中不同区域之间关系的表示能力。Among them, in formula (1), GCN(﹒) represents the graph convolutional neural network, f represents the feature vector processed by dimensionality reduction, and f represents the input of the graph convolution operation; in addition, Conv1D(﹒) represents a one-dimensional volume The product operation uses two one-dimensional graph convolution operations with different directions for feature extraction. After sufficient end-to-end training, the graph convolution operation enhances the ability to represent the relationship between different regions in the feature.
作为优选的，步骤(3)，采用X表示从正常领域内样本中使用深度神经网络提取的在某一层的特征，Y表示从对抗领域内样本中使用同样的深度神经网络提取的在同一个层处的特征，这两个特征分布X与Y之间的最优传输距离被公式化为如下形式：Preferably, in step (3), X is used to represent the features at a certain layer extracted from samples in the normal field by using a deep neural network, and Y is used to represent the same layer of features extracted from samples in the confrontation field using the same deep neural network. features at layers, the optimal transmission distance between these two feature distributions X and Y is formulated as:
公式(2)：Formula (2):
公式(2)即为Wasserstein distance(推土机距离)的定义。其中，：＝表示这是一个定义，把右边的计算结果定义左边的表示形式，PX和PY分别表示特征X和Y的边缘分布形式，并且P(X～PX，Y～PY)表示特征X和Y的联合分布，c(x，y)是任意的可测误差函数，它度量了X与Y之间的距离；此外，E(X，Y)～Γ表示计算的是联合概率下的数学期望，inf表示这里计算是数学期望的下确界，因此，Wc(PX，PY)就被定义为在可测误差函数c的前提下，以特征X与Y的边缘分布PX和PY为输入，在所有的度量距离方式中，X到Y距离最小的方式被称为最优传输方式，计算出的距离值为需要的最优传输距离。Formula (2) is the definition of Wasserstein distance (bulldozer distance). Among them, := indicates that this is a definition, the calculation result on the right is defined as the representation on the left, PX and PY represent the marginal distribution of features X and Y respectively, and P(X～PX , Y～PY ) Represents the joint distribution of features X and Y, c(x, y) is an arbitrary measurable error function, which measures the distance between X and Y; in addition, E(X, Y) ~ Γ indicates that the joint probability is calculated Under the mathematical expectation, inf indicates that the calculation here is the infimum of the mathematical expectation, therefore, Wc (PX , PY ) is defined as the marginal distribution of features X and Y under the premise of a measurable error function c PX and PY are inputs. Among all the distance measurement methods, the method with the smallest distance from X to Y is called the optimal transmission method, and the calculated distance value is the required optimal transmission distance.
作为优选的，步骤(4)，具体包括从正常领域图像样本和对抗领域图像样本中分层提取的特征表示，在使用图卷积进行处理后，使用Wasserstein distance计算它们之间的差异，将不同层次中对抗样本特征表示和正常样本特征表示的Wasserstein distance加入到优化网络参数使用的最终损失函数中去，通过充分的端到端训练，让网络模型逐渐的利用特征对齐来学习到更加鲁棒的特征表示；Preferably, step (4) specifically includes feature representations extracted hierarchically from normal domain image samples and adversarial domain image samples, and after processing using graph convolution, using Wasserstein distance to calculate the difference between them, The Wasserstein distance of the adversarial sample feature representation and the normal sample feature representation in the hierarchy is added to the final loss function used to optimize the network parameters. Through sufficient end-to-end training, the network model can gradually use feature alignment to learn a more robust model. feature representation;
最终的损失函数如下所示：The final loss function looks like this:
公式(3)：Formula (3):
其中，在公式(3)中，F表示用于图像分类的深度神经网络，θ为该深度神经网络的参数，该参数是在网络端到端训练时进行学习的，LCE表示交叉熵损失函数，同时计算了正常样本和相应的对抗样本的交叉熵损失，使得网络能对正常样本和对抗样本成功分类；xclean表示正常样本，xadv表示对抗样本，ytrue表示数据的正确标签，
本发明的优点是：与现有技术相比，本发明从领域自适应的角度提出了一种新颖的分层特征对齐方法，使得深度卷积神经网络可以从对抗样本中学习到鲁棒的特征表示；在通过渐进式地沿着模型网络结构提升对抗样本特征与正常样本特征相似度时，为了更好的让模型学得鲁棒特征表示，本发明提供了一种基于最优传输理论的Wassersteindistance来度量对抗样本特征与正常样本特征之间的差异。The advantages of the present invention are: compared with the prior art, the present invention proposes a novel hierarchical feature alignment method from the perspective of domain adaptation, so that the deep convolutional neural network can learn robust features from adversarial samples When the similarity between adversarial sample features and normal sample features is gradually improved along the model network structure, in order to better allow the model to learn a robust feature representation, the present invention provides a Wasserstein distance based on optimal transmission theory. to measure the difference between adversarial sample features and normal sample features.
本发明提供的方法可以有效的提升基于深度卷积神经网络的模型对于不同对抗领域样本的泛化能力，即使受到以往方法难以处理的白盒攻击，本发明也给提供有效的防御；The method provided by the present invention can effectively improve the generalization ability of the model based on the deep convolutional neural network for samples in different adversarial fields, and the present invention also provides effective defense even if it suffers from white-box attacks that are difficult to handle by previous methods;
本发明基于深度卷积神经网络的模型可以通过分层特征对齐的操作学到更鲁棒的图像特征，从而解决现有技术存在的针对不同领域对抗样本模型泛化能力受限的问题，为深度模型系统的部署与应用提供有效的可靠性和安全性保障。The model based on the deep convolutional neural network of the present invention can learn more robust image features through the operation of hierarchical feature alignment, so as to solve the problem of limited generalization ability of adversarial sample models in different fields in the prior art. The deployment and application of the model system provide effective reliability and security guarantees.
下面结合说明书附图和具体实施例对本发明作进一步说明。The present invention will be further described below with reference to the accompanying drawings and specific embodiments of the description.
附图说明Description of drawings
图1为本发明实施例的流程图；1 is a flowchart of an embodiment of the present invention;
图2为本发明实施例为对抗防御提出模型结构示意图；2 is a schematic structural diagram of a model proposed for adversarial defense according to an embodiment of the present invention;
图3为本发明实施例在对抗样本上面的鲁棒特征学习过程示意图；3 is a schematic diagram of a robust feature learning process on an adversarial sample according to an embodiment of the present invention;
图4为本发明实施例在三种典型分类数据集上面分别对正常样本和对抗样本决策空间的可视化示意图。FIG. 4 is a schematic diagram of visualization of decision spaces for normal samples and adversarial samples respectively on three typical classification data sets according to an embodiment of the present invention.
具体实施方式Detailed ways
参见图1、图2和图3，本发明公开的一种基于分层特征对齐的鲁棒特征学习方法，包括以下步骤：Referring to Fig. 1, Fig. 2 and Fig. 3, a robust feature learning method based on hierarchical feature alignment disclosed in the present invention includes the following steps:
(1)从不同领域的样本中使用深度卷积神经网络提起不同层次的深度特征；(1) Use deep convolutional neural networks to extract different levels of deep features from samples in different fields;
(2)对于提取的分层特征，通过图卷积神经网络给特征的通道和空间关系加以限制，从而使得模型学得更加丰富的特征表示；(2) For the extracted hierarchical features, the channel and spatial relationship of the features are restricted by the graph convolutional neural network, so that the model can learn a richer feature representation;
(3)使用基于最优传输理论的Wasserstein distance来准确的度量不同领域样本特征表示之间的差异；(3) Using the Wasserstein distance based on the optimal transmission theory to accurately measure the difference between the feature representations of samples in different fields;
(4)将从不同领域样本中提取的分层特征之间的差异作为模型损失函数的一部分来帮助模型学习更加鲁棒的特征，从而提升深度神经网络模型的对抗鲁棒性。(4) The difference between hierarchical features extracted from samples from different domains is used as part of the model loss function to help the model learn more robust features, thereby improving the adversarial robustness of the deep neural network model.
步骤(1)，具体过程为，使用PGD(Project Gradient Descent，映射梯度下降)攻击方法从正常领域内的图像样本产生相对应的带有不同程度扰动的对抗领域图像样本。对于正常领域图像样本和对抗领域图像样本，本方法使用深度卷积神经网络来进行图像的多层次特征提取。为了使得提取的特征更具有代表性，我们将深度网络按照其结构分为多个层次。由于本发明所提出的是一个框架，因此，我们使用不同的“层”来表示这些被划分的用来进行特征提取的结构层次，而不是明确指定是网络的哪一个层，本发明提出的结构层次划分标准如图2所示。Step (1), the specific process is to use the PGD (Project Gradient Descent, mapping gradient descent) attack method to generate corresponding image samples in the confrontation domain with different degrees of disturbance from the image samples in the normal domain. For normal domain image samples and adversarial domain image samples, this method uses deep convolutional neural networks to perform multi-level feature extraction of images. In order to make the extracted features more representative, we divide the deep network into multiple layers according to its structure. Since the present invention proposes a framework, we use different "layers" to represent these divided structural layers for feature extraction, rather than specifying which layer of the network is explicitly. The structure proposed by the present invention The level division standard is shown in Figure 2.
以图2所示结构为例，我们使用ResNet-110(“ResNet-110”此处指具有110个网络层的残差网络)网络结构来进行图像的特征提取，在此处我们将其分为4个不同的结构层次。输入正常样本或对抗样本后，在网络进行正向推理的时，在4种不同的层次处，使用卷积结构提取不同尺度、不同抽象程度的图像特征。Taking the structure shown in Figure 2 as an example, we use the ResNet-110 ("ResNet-110" here refers to a residual network with 110 network layers) network structure for image feature extraction, where we divide it into 4 different structural levels. After inputting normal samples or adversarial samples, when the network performs forward inference, the convolution structure is used to extract image features of different scales and degrees of abstraction at four different levels.
在进行模型训练时，为了更好的进行特征对齐的操作，本发明提出首先在干净样本上面进行模型的训练，然后再共同使用干净样本和相对应的对抗样本进行基于分层特征对齐的鲁棒特征学习过程的模型训练。During model training, in order to perform better feature alignment, the present invention proposes to first train the model on clean samples, and then jointly use clean samples and corresponding adversarial samples to perform robustness based on hierarchical feature alignment. Model training for the feature learning process.
步骤(2)，具体过程为，在使用深度卷积神经网络提取不同层次处的图像特征后，对于这些具有代表性的图像特征，为了使网络学得更加丰富的图像特征表示，我们使用图卷积神经网络来对提取的不同层次的特征进行处理。图卷积可以更好的从全局的角度捕捉深度特征中不同区域之间关系；同时也可以为后续进行特征对齐时给特征加上更强的限制。由于在计算最优传输Wasserstein distance时候，我们计算的是特征向量之间的距离，因此，我们将张量形式的图像特征转为特征向量的形式；并且，为了加快该距离度量的计算，我们将采取了一系列特征选择与降维的操作。Step (2), the specific process is, after using the deep convolutional neural network to extract image features at different levels, for these representative image features, in order to make the network learn more rich image feature representation, we use the graph volume The neural network is used to process the extracted features at different levels. Graph convolution can better capture the relationship between different regions in deep features from a global perspective; it can also impose stronger constraints on features for subsequent feature alignment. Since we calculate the distance between feature vectors when calculating the optimal transmission Wasserstein distance, we convert the image features in the form of tensors into the form of feature vectors; and, in order to speed up the calculation of the distance metric, we will A series of feature selection and dimensionality reduction operations are taken.
为了降低降维的复杂度，我们使用了具有代表性的特征线性组合的方式来对提取的特征从通道以及特征结点两个方面进行处理，In order to reduce the complexity of dimensionality reduction, we use a representative linear combination of features to process the extracted features from two aspects: channels and feature nodes.
经过降维后，我们使用两个一维卷积进行图卷积的操作，该图卷积操作可以公式化为以下形式：After dimensionality reduction, we use two one-dimensional convolutions to perform graph convolution operations, which can be formulated as follows:
公式(1)：Formula 1):
GCN(f)＝Conv1D[Conv1D(f)]GCN(f)=Conv1D[Conv1D(f)]
在公式(1)中，GCN(﹒)表示图卷积神经网络，f表示经过降维处理的特征向量，此处，f表示图卷积操作的输入；此外，Conv1D(﹒)表示一个一维卷积操作，在此处我们使用两个方向不同的一维图卷积操作进行特征提取。在经过充分的端到端训练后，该图卷积操作可以加强对于特征中不同区域之间关系的表示能力。In formula (1), GCN(﹒) represents the graph convolutional neural network, f represents the feature vector after dimensionality reduction processing, where f represents the input of the graph convolution operation; in addition, Conv1D(﹒) represents a one-dimensional Convolution operation, where we use two 1D graph convolution operations with different directions for feature extraction. After sufficient end-to-end training, this graph convolution operation can enhance the representation of the relationships between different regions in the features.
具体细节信息如图2所示。在经过充分的端到端训练后，该图卷积操作可以加强对于特征中不同区域之间关系的表示能力。另外，对于在不同网络结构层次处从正常样本中提取的特征和从对抗样本中提取的特征，在计算它们之间的Wasserstein distance之前，本发明都会使用图卷积对特征进行处理。如图2所示，在使用ResNet-110结构时，我们计算了4个不同位置处的Wasserstein distance来进行不同领域样本特征差异的度量。The specific details are shown in Figure 2. After sufficient end-to-end training, this graph convolution operation can enhance the representation of the relationships between different regions in the features. In addition, for the features extracted from normal samples and the features extracted from adversarial samples at different network structure levels, before calculating the Wasserstein distance between them, the present invention uses graph convolution to process the features. As shown in Figure 2, when using the ResNet-110 structure, we calculated the Wasserstein distance at 4 different locations to measure the difference of sample features in different domains.
步骤(3)，具体过程为，在使用步骤(1)进行层次图像特征提取，并且使用步骤(2)进行特征选择、降维、图卷积操作后，我们使用带有正则化项的最优传输Wassersteindistance来计算不同领域样本之间的差异，这一步是为了进行不同领域样本特征之间的特征对齐操作,将对抗样本的层次特征向正常样本的层次特征进行对齐，从而使得神经模型具有足够的鲁棒性。Step (3), the specific process is, after using step (1) for hierarchical image feature extraction, and using step (2) for feature selection, dimensionality reduction, and graph convolution operations, we use the optimal solution with regularization term. Transfer Wasserstein distance to calculate the difference between samples in different fields. This step is to perform feature alignment operations between sample features in different fields, and align the hierarchical features of adversarial samples to the hierarchical features of normal samples, so that the neural model has sufficient robustness.
在本实施例中，我们使用X与Y表示两个不同分布的特征向量的集合，更具体来说，X表示从正常领域内样本中使用深度神经网络提取的在某一层的特征，Y表示从对抗领域内样本中使用同样的深度神经网络提取的在同一个层处的特征，这两个特征分布X与Y之间的最优传输距离可以被公式化为如下形式：In this embodiment, we use X and Y to represent a set of feature vectors with two different distributions. More specifically, X represents the features at a certain layer extracted from samples in the normal field using a deep neural network, and Y represents For the features at the same layer extracted from the samples in the adversarial domain using the same deep neural network, the optimal transmission distance between the two feature distributions X and Y can be formulated as follows:
公式(2)：Formula (2):
公式(2)即为Wasserstein distance(推土机距离)的定义。其中，符号：＝表示这是一个定义，我们把右边的计算结果定义左边的表示形式。在公式中，PX和PY分别表示特征X和Y的边缘分布形式，并且P(X～PX，Y～PY)表示特征X和Y的联合分布。c(x，y)是任意的可测误差函数，它度量了X与Y之间的距离。此外，在公式中，E(X，Y)～Γ表示计算的是联合概率下的数学期望，inf表示这里计算是数学期望的下确界。因此，Wc(PX，PY)就被定义为在可测误差函数c的前提下，以特征X与Y的边缘分布PX和PY为输入，在所有的度量距离方式中，X到Y距离最小的方式被称为最优传输方式，计算出的距离值即为这里需要的最优传输距离。Formula (2) is the definition of Wasserstein distance (bulldozer distance). Among them, the symbol: = indicates that this is a definition, and we define the calculation result on the right as the representation on the left. In the formula, PX and PY represent the marginal distribution forms of the features X and Y, respectively, and P(X ~ PX , Y ~ PY ) represents the joint distribution of the features X and Y . c(x,y) is an arbitrary measurable error function that measures the distance between X and Y. In addition, in the formula, E(X, Y) ~ Γ indicates that the mathematical expectation under the joint probability is calculated, and inf indicates that the calculation here is the infimum of the mathematical expectation. Therefore, Wc (PX , PY ) is defined as, under the premise of a measurable error function c, taking the marginal distributions PX and PY of features X and Y as input, in all metric distance methods, X The method with the smallest distance to Y is called the optimal transmission method, and the calculated distance value is the optimal transmission distance required here.
在本实施例，使用
在实际应用中，可以将其离散化为如下公式形式：In practical applications, it can be discretized into the following formula:
其中，在公式中<·，·>表示矩阵P与C之间的哈达玛德(Hadamard)乘积，由于P与C都是二维矩阵形式，因此此处表示矩阵P与矩阵C每一个对应位置处元素的乘积之和,min表示这里是一个计算最小值的优化问题。由于方法随着数据量的增大，其计算代价会快速上升，因此，本发明使用了熵正则化的方式来对算法进行改进，并且使用Sinkhorn迭代算法进行优化。关于矩阵P的熵正则化项如下公式所示：Among them, in the formula <·, ·> represents the Hadamard product between the matrices P and C. Since both P and C are in the form of two-dimensional matrices, each corresponding position of the matrix P and the matrix C is represented here. The sum of the products of the elements at, min indicates that this is an optimization problem to calculate the minimum value. Since the calculation cost of the method increases rapidly with the increase of the amount of data, the present invention uses the entropy regularization method to improve the algorithm, and uses the Sinkhorn iterative algorithm to optimize. The entropy regularization term for the matrix P is given by the following formula:
因此，可以得到带有正则化的最优传输Wasserstein distance计算方法：Therefore, the optimal transmission Wasserstein distance calculation method with regularization can be obtained:
其中，在公式中，∈用来平衡该正则化问题与原始问题的逼近程度，当∈趋向于0时，上述正则化问题就转变为原始问题，在本发明中，∈＝0.1。此外，由于该问题是一个凸优化问题，因此它具有唯一的解。另外，在本发明中，Wasserstein distance被用来度量从正常样本和对抗样本中使用深度卷积神经网络提取的中间特征表示之间的差异。Among them, in the formula, ∈ is used to balance the approximation degree of the regularization problem and the original problem. When ∈ tends to 0, the above-mentioned regularization problem becomes the original problem. In the present invention, ∈=0.1. Furthermore, since the problem is a convex optimization problem, it has a unique solution. Additionally, in the present invention, Wasserstein distance is used to measure the difference between intermediate feature representations extracted from normal samples and adversarial samples using deep convolutional neural networks.
另外，在优化该最优传输距离时，我们选择使用Sinkhorn迭代算法。In addition, when optimizing the optimal transmission distance, we choose to use the Sinkhorn iterative algorithm.
步骤(4)，具体过程为，我们将不同层次中对抗样本特征表示和正常样本特征表示的Wasserstein distance加入到优化网络参数使用的最终损失函数中去，通过充分的端到端训练，让网络模型逐渐的利用特征对齐来学习到更加鲁棒的特征表示。Step (4), the specific process is that we add the Wasserstein distance represented by the adversarial sample feature representation and the normal sample feature representation in different levels into the final loss function used to optimize the network parameters, and through sufficient end-to-end training, let the network model. Gradually, feature alignment is used to learn more robust feature representations.
最终的损失函数如下公式所示：The final loss function is as follows:
公式(3)Formula (3)
其中，在公式(3)中，F表示用于图像分类的深度神经网络，θ为该深度神经网络的参数，该参数是在网络端到端训练时进行学习的，LCE表示交叉熵损失函数，同时计算了正常样本和相应的对抗样本的交叉熵损失，使得网络能对正常样本和对抗样本成功分类；xclean表示正常样本，xadv表示对抗样本，ytrue表示数据的正确标签，
本发明实施例，具有以下有益效果：The embodiment of the present invention has the following beneficial effects:
与现有技术相比，本发明从领域自适应的角度提出了一种新颖的分层特征对齐方法，使得深度卷积神经网络可以从对抗样本中学习到鲁棒的特征表示；在通过渐进式地沿着模型网络结构提升对抗样本特征与正常样本特征相似度时，为了更好的让模型学得鲁棒特征表示，本发明提供了一种基于最优传输理论的Wasserstein distance来度量对抗样本特征与正常样本特征之间的差异。Compared with the prior art, the present invention proposes a novel hierarchical feature alignment method from the perspective of domain adaptation, so that the deep convolutional neural network can learn robust feature representations from adversarial samples; When improving the similarity between adversarial sample features and normal sample features along the model network structure, in order to better allow the model to learn robust feature representation, the present invention provides a Wasserstein distance based on optimal transmission theory to measure adversarial sample features difference from normal sample features.
本发明提供的方法可以有效的提升基于深度卷积神经网络的模型对于不同对抗领域样本的泛化能力，即使受到以往方法难以处理的白盒攻击，本发明也给提供有效的防御；The method provided by the present invention can effectively improve the generalization ability of the model based on the deep convolutional neural network for samples in different adversarial fields, and the present invention also provides effective defense even if it suffers from white-box attacks that are difficult to handle by previous methods;
本发明基于深度卷积神经网络的模型可以通过分层特征对齐的操作学到更鲁棒的图像特征，从而解决现有技术存在的针对不同领域对抗样本模型泛化能力受限的问题，为深度模型系统的部署与应用提供有效的可靠性和安全性保障。The model based on the deep convolutional neural network of the present invention can learn more robust image features through the operation of hierarchical feature alignment, so as to solve the problem of limited generalization ability of adversarial sample models in different fields in the prior art. The deployment and application of the model system provide effective reliability and security guarantees.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，所述的程序可以存储于计算机可读取存储介质中，所述的存储介质，如ROM/RAM、磁盘、光盘等。Those of ordinary skill in the art can understand that all or part of the steps in the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium , such as ROM/RAM, magnetic disk, CD, etc.
上述实施例对本发明的具体描述，只用于对本发明进行进一步说明，不能理解为对本发明保护范围的限定，本领域的技术工程师根据上述发明的内容对本发明做出一些非本质的改进和调整均落入本发明的保护范围之内。The specific description of the present invention in the above embodiments is only used to further illustrate the present invention, and should not be construed as a limitation on the protection scope of the present invention. fall within the protection scope of the present invention.
Claims (6)
Translated fromChinese
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010809932.6ACN111950635B (en) | 2020-08-12 | 2020-08-12 | Robust feature learning method based on layered feature alignment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010809932.6ACN111950635B (en) | 2020-08-12 | 2020-08-12 | Robust feature learning method based on layered feature alignment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111950635Atrue CN111950635A (en) | 2020-11-17 |
| CN111950635B CN111950635B (en) | 2023-08-25 |
Family
ID=73331806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010809932.6AActiveCN111950635B (en) | 2020-08-12 | 2020-08-12 | Robust feature learning method based on layered feature alignment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111950635B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112465019A (en)* | 2020-11-26 | 2021-03-09 | 重庆邮电大学 | Countermeasure sample generation and countermeasure defense method based on disturbance |
| CN113436073A (en)* | 2021-06-29 | 2021-09-24 | 中山大学 | Real image super-resolution robust method and device based on frequency domain |
| CN117409217A (en)* | 2023-08-11 | 2024-01-16 | 北京邮电大学 | Inter-domain difference measurement method for image dataset |
| CN119110084A (en)* | 2024-08-30 | 2024-12-10 | 上海浙江大学高等研究院 | A high compression ratio image compression method based on optimal transmission mapping |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018028255A1 (en)* | 2016-08-11 | 2018-02-15 | 深圳市未来媒体技术研究院 | Image saliency detection method based on adversarial network |
| US20190128989A1 (en)* | 2017-11-01 | 2019-05-02 | Siemens Healthcare Gmbh | Motion artifact reduction of magnetic resonance images with an adversarial trained network |
| US20190303720A1 (en)* | 2018-03-30 | 2019-10-03 | Arizona Board Of Regents On Behalf Of Arizona State University | Systems and methods for feature transformation, correction and regeneration for robust sensing, transmission, computer vision, recognition and classification |
| CN110674866A (en)* | 2019-09-23 | 2020-01-10 | 兰州理工大学 | Method for detecting X-ray breast lesion images by using transfer learning characteristic pyramid network |
| CN110728219A (en)* | 2019-09-29 | 2020-01-24 | 天津大学 | 3D face generation method based on multi-column multi-scale graph convolution neural network |
| CN110738622A (en)* | 2019-10-17 | 2020-01-31 | 温州大学 | Lightweight neural network single image defogging method based on multi-scale convolution |
| CN111126258A (en)* | 2019-12-23 | 2020-05-08 | 深圳市华尊科技股份有限公司 | Image recognition method and related device |
| CN111178504A (en)* | 2019-12-17 | 2020-05-19 | 西安电子科技大学 | Information processing method and system of robust compression model based on deep neural network |
| CN111242227A (en)* | 2020-01-16 | 2020-06-05 | 天津师范大学 | A Multimodal Ground-Based Cloud Recognition Method Based on Heterogeneous Depth Features |
| US20200234110A1 (en)* | 2019-01-22 | 2020-07-23 | Adobe Inc. | Generating trained neural networks with increased robustness against adversarial attacks |
| CN111476200A (en)* | 2020-04-27 | 2020-07-31 | 华东师范大学 | Face de-identification generation method based on generation of confrontation network |
- 2020
- 2020-08-12CNCN202010809932.6Apatent/CN111950635B/enactiveActive
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018028255A1 (en)* | 2016-08-11 | 2018-02-15 | 深圳市未来媒体技术研究院 | Image saliency detection method based on adversarial network |
| US20190128989A1 (en)* | 2017-11-01 | 2019-05-02 | Siemens Healthcare Gmbh | Motion artifact reduction of magnetic resonance images with an adversarial trained network |
| US20190303720A1 (en)* | 2018-03-30 | 2019-10-03 | Arizona Board Of Regents On Behalf Of Arizona State University | Systems and methods for feature transformation, correction and regeneration for robust sensing, transmission, computer vision, recognition and classification |
| US20200234110A1 (en)* | 2019-01-22 | 2020-07-23 | Adobe Inc. | Generating trained neural networks with increased robustness against adversarial attacks |
| CN110674866A (en)* | 2019-09-23 | 2020-01-10 | 兰州理工大学 | Method for detecting X-ray breast lesion images by using transfer learning characteristic pyramid network |
| CN110728219A (en)* | 2019-09-29 | 2020-01-24 | 天津大学 | 3D face generation method based on multi-column multi-scale graph convolution neural network |
| CN110738622A (en)* | 2019-10-17 | 2020-01-31 | 温州大学 | Lightweight neural network single image defogging method based on multi-scale convolution |
| CN111178504A (en)* | 2019-12-17 | 2020-05-19 | 西安电子科技大学 | Information processing method and system of robust compression model based on deep neural network |
| CN111126258A (en)* | 2019-12-23 | 2020-05-08 | 深圳市华尊科技股份有限公司 | Image recognition method and related device |
| CN111242227A (en)* | 2020-01-16 | 2020-06-05 | 天津师范大学 | A Multimodal Ground-Based Cloud Recognition Method Based on Heterogeneous Depth Features |
| CN111476200A (en)* | 2020-04-27 | 2020-07-31 | 华东师范大学 | Face de-identification generation method based on generation of confrontation network |
Non-Patent Citations (1)
| Title |
|---|
| 范宝杰，等: "基于卷积对抗网络的多通道图像修复方法", 《计算机应用与软件》, vol. 37, no. 7, pages 176 - 179* |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112465019A (en)* | 2020-11-26 | 2021-03-09 | 重庆邮电大学 | Countermeasure sample generation and countermeasure defense method based on disturbance |
| CN113436073A (en)* | 2021-06-29 | 2021-09-24 | 中山大学 | Real image super-resolution robust method and device based on frequency domain |
| CN117409217A (en)* | 2023-08-11 | 2024-01-16 | 北京邮电大学 | Inter-domain difference measurement method for image dataset |
| CN117409217B (en)* | 2023-08-11 | 2025-03-28 | 北京邮电大学 | A method for measuring inter-domain differences in image datasets |
| CN119110084A (en)* | 2024-08-30 | 2024-12-10 | 上海浙江大学高等研究院 | A high compression ratio image compression method based on optimal transmission mapping |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111950635B (en) | 2023-08-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111950635B (en) | Robust feature learning method based on layered feature alignment | |
| Lin et al. | Ru-net: Regularized unrolling network for scene graph generation | |
| WO2023280065A1 (en) | Image reconstruction method and apparatus for cross-modal communication system | |
| Esmaeili et al. | Fast-at: Fast automatic thumbnail generation using deep neural networks | |
| CN110309835B (en) | A method and device for extracting local features of an image | |
| Zhang et al. | An intrusion detection method based on stacked sparse autoencoder and improved gaussian mixture model | |
| CN109255381B (en) | Image classification method based on second-order VLAD sparse adaptive depth network | |
| CN114565053A (en) | A Deep Heterogeneous Graph Embedding Model Based on Feature Fusion | |
| CN115908908B (en) | Remote sensing image aggregation type target recognition method and device based on graph attention network | |
| CN111178504B (en) | Information processing method and system of robust compression model based on deep neural network | |
| CN114036303B (en) | Remote supervision relation extraction method based on double granularity attention and countermeasure training | |
| CN106228245A (en) | Infer based on variation and the knowledge base complementing method of tensor neutral net | |
| CN110751195A (en) | Fine-grained image classification method based on improved YOLOv3 | |
| CN105894469A (en) | De-noising method based on external block autoencoding learning and internal block clustering | |
| CN107871103A (en) | Face authentication method and device | |
| CN111291760A (en) | Semantic segmentation method and device for image and electronic equipment | |
| CN111400572A (en) | Content safety monitoring system and method for realizing image feature recognition based on convolutional neural network | |
| CN114528971A (en) | Atlas frequent relation mode mining method based on heterogeneous atlas neural network | |
| CN113987203A (en) | A knowledge graph reasoning method and system based on affine transformation and bias modeling | |
| CN117197451B (en) | Remote sensing image semantic segmentation method and device based on domain adaptation | |
| CN114693997A (en) | Image description generation method, device, equipment and medium based on transfer learning | |
| CN112307914B (en) | Open domain image content identification method based on text information guidance | |
| CN117292249A (en) | Underwater sonar image open set classification method, system, equipment and medium | |
| CN118379593A (en) | Semi-supervised semantic segmentation model training method based on multistage label correction and application | |
| CN106529604A (en) | Adaptive image tag robust prediction method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |