CN112434654A

Movatterモバイル変換

Info

Publication number: CN112434654A
Application number: CN202011430914.3A
Authority: CN
Inventors: 张艳; 相旭; 唐俊; 王年; 屈磊
Original assignee: Anhui University
Current assignee: Anhui University
Priority date: 2020-12-07
Filing date: 2020-12-07
Publication date: 2021-03-02
Anticipated expiration: 2040-12-07
Also published as: CN112434654B

Abstract

The invention discloses a cross-modal pedestrian re-identification method based on a symmetric convolutional neural network, which comprises the following steps of: 1, acquiring pedestrian photos under two different modes of visible light and infrared light, constructing a cross-mode pedestrian re-identification data set, and constructing a search library; 2, establishing a symmetrical convolution neural network cross-modal pedestrian re-identification method model by using a neural network; training a cross-modal pedestrian re-identification method model based on a symmetric convolutional neural network by using a data set; and 4, forecasting is realized by utilizing the established model so as to achieve the purpose of cross-modal pedestrian re-identification. The method can greatly relieve the problem that the existing pedestrian re-identification method is inaccurate in detection in the cross-mode state, and still has higher detection precision under the condition of larger modal difference.

Description

Cross-modal pedestrian re-identification method based on symmetric convolutional neural network

Technical Field

The invention belongs to the technical field of computer vision, and particularly relates to a cross-modal pedestrian re-identification method based on a symmetric convolutional neural network.

Background

Cross-modal pedestrian re-identification is an important subject in the field of computer vision. Cross-modal pedestrian re-identification plays a crucial role in target tracking, video monitoring and public security, and is concerned by more and more scholars.

For a traditional single-mode pedestrian re-identification task, data only comprises visible light images, and the difficulty is mainly that the visual angle of a camera is changed, the camera is shielded, the posture of a pedestrian is changed, the illumination is changed, the background is complex, and the like. The cross-mode pedestrian re-identification data includes not only a visible light image but also an infrared light image, and image retrieval needs to be performed in the two modes. At night, the visible camera has difficulty capturing enough pedestrian appearance information due to weak lighting, and the pedestrian appearance information is mainly acquired by the infrared camera or the depth camera. Because the imaging mechanisms of the two cameras are different, two modes are formed, and huge mode difference exists between the two images. The visible light image is different from the infrared light image, and as shown in fig. 1, it can be seen that the visible light image contains more color information than the infrared light image. In addition to the intra-modal differences, inter-modal differences also present another significant problem to be solved for cross-modal pedestrian re-identification.

The inter-modal differences between the visible light modality and the infrared light modality can be subdivided into characteristic differences and appearance differences. To reduce the impact of feature variation, some approaches reduce feature variation by aligning cross-modal features using a uniform embedding space, but this ignores the large apparent difference between the two modalities. Other methods use generation of a countermeasure network (GAN) to effect image conversion between visible and infrared light images in such a way as to reduce the effect of appearance differences. Although the virtual image generated by GAN is similar to the original image, it is not guaranteed that the detail information related to the identity is generated, and the generated information is not guaranteed to be completely reliable.

Disclosure of Invention

The invention provides a cross-modal pedestrian re-identification method based on a symmetric convolutional neural network aiming at the problem of inter-modal and intra-modal differences, so as to reduce the inter-modal and intra-modal differences and improve the re-identification effect and precision.

In order to achieve the purpose, the invention adopts the following technical scheme:

the invention relates to a cross-modal pedestrian re-identification method based on a symmetric convolutional neural network, which is characterized by comprising the following steps of:

step 1, collecting a visible light image set V of N pedestrians, wherein j visible light images of the ith pedestrian are recorded as V_iAnd V is_i＝{V_i¹,V_i²,...,V_i^j}，V_i^jA jth visible light picture representing the ith pedestrian and giving ith identity information y to the ith pedestrian_i；i＝1,2,…,N；

Collecting an infrared light image set T of N pedestrians by using an infrared light camera or a depth camera, wherein m infrared light images of the ith pedestrian are recorded as T_iAnd T is_i＝{T_i¹,T_i²,...,T_i^m}，T_i^mAn m-th infrared light image representing an i-th pedestrian;

constructing a search library by visible light pictures and infrared light images of other pedestrians with known identity information;

step 2, constructing a symmetrical convolutional neural network consisting of a generator and a discriminator;

the generator consists of two independent columns of ResNet50 networks, wherein the ResNet50 network consists of d residual sub-modules, and a column of full connection layers S is added after the d-1 residual sub-module₁Adding a column of fully-connected layers S after the d-th residual sub-module₂；

The discriminator consists of a visible light image classifier and an infrared light image classifier;

initializing network weights for the ResNet50 network;

initializing parameters of the full connection layer and the discriminator by adopting a random initialization mode;

step 3, respectively inputting the visible light image set V and the infrared light image set T of the N pedestrians into two independent ResNet50 networks, and outputting a d-1 group of visible light characteristic information V after a d-1 residual sub-module_d-1And d-1 group infrared light characteristic information t_d-1Respectively inputting the d-th residual error sub-module and outputting a d-th group of visible light characteristic information v_dAnd d group infrared light characteristic information t_d；

Step 4, constructing the d-1 th sample feature space X_d-1；

Selecting visible light characteristic information and infrared light characteristic information of P pedestrians from all the characteristic information output by the d-1 th residual error submodule, wherein the visible light characteristic information v of each pedestrian_i,d-1And infrared light characteristic information t_i,d-1Respectively selecting K pieces of feature information to construct a d-1 th sample feature space X_d-1；

The d-1 th sample feature space X_d-1Are input together to the subsequent fully-connected layer S₁D-1 group visible light feature vector v 'is output'_d-1And infrared light feature vector t'_d-1；

Step 5, constructing the d sample feature space X_d；

Selecting visible light characteristic information and infrared light characteristic information of P pedestrians from all the characteristic information output by the d residual error submodule, wherein the visible light characteristic information v of each pedestrian_i,dAnd infrared light characteristic information t_i,dRespectively selecting K pieces of feature information to construct a d-th sample feature space X_d；

Then the d sample feature space X is used_dAre input together to the subsequent fully-connected layer S₂And outputs a d-th group visible light feature vector v'_dAnd infrared light feature vector t'_d；

Step 6, the d-1 group of visible light feature vectors v'_d-1Inputting the infrared light characteristic vector into the visible light image classifier, outputting an initial probability distribution GV of the visible light, and converting the d-1 th group of infrared light characteristic vectors t'_d-1Inputting the infrared light image classifier and outputting an initial probability distribution GT of infrared light;

construction of identity loss function L using equation (1)_ID：

Step 7, from the d-1 sample feature space X_d-1To select the kth characteristic information of the a-th pedestrian

If the feature vector is recorded as the anchor sample feature vector, the anchor sample feature vector is recorded together with the anchor sample feature vector

The z-th characteristic information of the a-th pedestrian with the same identity information

Is denoted as the z-th positive sample feature vector, and

the c characteristic information of the f pedestrian with different identity information

And (4) establishing a mixed ternary loss function L by using the formula (2) after the c negative sample feature vector is recorded_TRI1(X_d-1)：

In the formula (2), the reaction mixture is,

representing anchor sample feature vectors

And the z-th positive sample feature vector

The Euclidean distance of (a) is,

representing anchor sample feature vector and the c negative sample feature vector of the f pedestrian

Euclidean distance of (p)₁Is a mixed ternary loss function L_TRI1(X_d-1) A predefined minimum interval;

step 8, from the d sample characteristic space X_dIn the method, the s characteristic information of the r pedestrian is selected

B-th characteristic information of r-th pedestrian with same identity information

Is denoted as the b-th positive sample feature vector, and

qth characteristic information of the h pedestrian with different identity information

And (4) establishing a mixed ternary loss function L by using the formula (3) after the q negative sample feature vector is recorded_TRI2(X_d)：

In the formula (3), the reaction mixture is,

representing anchor sample feature vectors

And the b-th positive sample feature vector

The Euclidean distance of (a) is,

representing the anchor sample feature vector and the qth negative sample feature vector of the h pedestrian

Euclidean distance of (p)₂Is a mixed ternary loss function L_TRI2(X_d) A predefined minimum interval;

step 9, establishing a mixed ternary loss function L by using the formula (4)_TRI：

L_TRI＝L_TRI1+L_TRI2 (4)

Establishing a global penalty function L using equation (5)_ALL：

L_ALL＝L_ID+βL_TRI (5)

In the formula (5), β represents a mixed ternary loss function L_TRIThe coefficient of (a);

carrying out optimization solution on the formula (5) by a random gradient descent method, carrying out gradient back propagation, training each parameter of the symmetric convolutional neural network, and obtaining a preliminarily trained symmetric convolutional neural network model;

step 10, the d-1 group visible light feature vector v'_d-1Inputting the visible light image classifier in the preliminarily trained symmetric convolutional neural network model, and outputting visible lightD-1 group infrared light feature vector t'_d-1Inputting the infrared light image classifier in the preliminarily trained symmetric convolutional neural network model, and outputting the probability distribution GT' of infrared light; d-1 group visible light feature vector v'_d-1Inputting the pseudo visible light probability distribution GV 'into an infrared light classifier in the preliminarily trained symmetric convolutional neural network model to obtain a pseudo visible light probability distribution GV';

constructing a divergence loss function L between the pseudo visible light eigenvector GV 'and the visible light probability distribution GV' by using the formula (6)_KL：

L_KL＝KL(GV″,GV′) (6)

In the formula (6), KL (·,) represents the difference value of the probability distributions of the two;

establishing a discriminator loss function L using equation (7)_DIS：

L_DIS＝L_ID-αL_KL (7)

In the formula (7), α represents L_KLThe coefficient of (a);

step 11, establishing a generator loss function L by using the formula (8)_GEN：

L_GEN＝αL_KL+βL_TRI (8)

And 12, sequentially optimizing and solving the formula (5), the formula (7) and the formula (8) by a gradient descent method:

firstly, carrying out optimization solution on the formula (5) and training all parameters of the network;

secondly, carrying out optimization solution on the formula (7), in the gradient back propagation process, only carrying out back propagation on the gradient of the discriminator, and setting the gradient of the generator to zero, thereby freezing the generator parameters and training the discriminator parameters;

finally, carrying out optimization solution on the formula (8), in the gradient back propagation process, only carrying out back propagation on the gradient of the generator, and setting the gradient of the discriminator to zero, thereby freezing the parameters of the discriminator and training the parameters of the generator;

after training in turn, make L_ALL，L_DIS，L_GENConverge to optimum in antagonistic learning when L_DISReach the optimumWhen the discriminator is optimal, when L_GENWhen the optimal condition is reached, the generator is optimal, so that a final cross-modal pedestrian re-identification model of the symmetric convolutional neural network is obtained;

step 13, utilizing the final symmetrical convolutional neural network model to query and match the cross-modal pedestrian re-identification;

inputting the pedestrian image to be inquired into a final symmetrical convolutional neural network model to extract features, then carrying out similarity comparison with the features of the pedestrians in a search library, and finding corresponding pedestrian identity information from the ranking list according to the sequence of the similarity, thereby obtaining an identification result.

Compared with the prior art, the invention has the beneficial effects that:

1. aiming at the difference between the modes, the invention combines the mode confusion idea based on probability distribution with counterstudy to construct a symmetrical convolutional neural network, the symmetrical convolutional neural network is composed of a generator and a discriminator, and the network generates the mode invariant feature by minimizing the output probability distribution difference of a classifier in the discriminator, thereby achieving the purpose of mode confusion and realizing higher detection precision under the conditions of shielding, pedestrian posture change, illumination change and mode change.

2. In order to solve the two problems of the difference between the modes and the difference in the modes, the invention combines the ternary loss with the counterstudy and provides a mixed ternary loss to reduce the difference between the modes and the difference in the modes. When mode confusion is achieved through counterlearning, the positive sample and the negative sample are selected under the condition that modes are not distinguished to conduct feature alignment and reduce mode difference, so that high detection precision is achieved under the condition that the mode difference is large, and the adaptability of the method is high.

3. According to the capability of describing structure and spatial information of the hidden layer convolution characteristics, the invention adopts d-1 layer hidden layer convolution characteristics (namely characteristics from ResNet50 network residual submodule) as a rear full-connection layer S₁And the input of the following discriminator, so that the network can learn more space structure information and reduce the color differenceThe influence is reduced, and the difference between the two modes is reduced, so that the detection precision of the invention is improved, and the invention has stronger applicability in the fields of target tracking, video monitoring, public security and the like.

4. The invention aligns the features at different depths of the symmetrical convolutional neural network, so that the network can learn more deep information, the robustness of the network is improved, the problem of inaccurate detection of the existing pedestrian re-identification method in a cross-mode state can be greatly solved, and the accurate detection can be realized under the condition of appearance difference and other problems.

Drawings

FIG. 1 is a schematic view of two modes of a prior art mid-modal pedestrian;

FIG. 2 is a network architecture proposed by the present invention;

FIG. 3 is a schematic view of inter-modal and intra-modal losses to which the present invention relates;

FIG. 4 is a graph of the results of the alpha variable on the RegDB data set in the present invention;

FIG. 5 is a graph of the results of the alpha variable on the SYSU-MM01 dataset in accordance with the present invention;

FIG. 6 is a graph of the results of the beta variable on the RegDB data set in the present invention;

FIG. 7 is a graph of the results of the beta variable on the SYSU-MM01 dataset in accordance with the present invention.

Detailed Description

In the embodiment, a cross-modal pedestrian re-identification method based on a symmetric convolutional neural network mainly reduces inter-modal and intra-modal differences by using the symmetric convolutional neural network and counterstudy; and the network is optimized on different network depths, and the appearance difference is reduced by utilizing the shallow feature with more space structure information. Referring to fig. 1, there is shown a schematic diagram of images in two different modalities, the detailed steps are as follows:

step 1, collecting a visible light image set V of N pedestrians, wherein j visible light images of the ith pedestrian are recorded as V_iAnd V is_i＝{V_i¹,V_i²,...,V_i^j}，V_i^jRepresenting the i-th pedestrianThe jth visible light picture and endows ith identity information y to the ith pedestrian_i；i＝1,2,…,N；

this embodiment utilizes the RegDB dataset and the SYSU-MM01 dataset. The SYSU-MM01 is a large-scale cross-modal pedestrian re-identification dataset collected by four visible light cameras and two infrared light cameras. The data set has two different scenes, namely indoor and outdoor, and the training set comprises 395 pieces of pedestrian identity data information, wherein 11909 infrared light pedestrian images and 22258 visible light pedestrian images are shared.

The RegDB data set contains 412 pedestrian identity information, which are captured by the dual camera system. Each pedestrian ID contains 10 visible light images and 10 infrared light images in total. The invention adopts a recognized data set processing method, randomly divides all data in the data set into two parts, and randomly selects a part of data for training.

the generator consists of two independent columns of ResNet50 networks, where the ResNet50 network consists of d residual sub-modules, a column of fully connected layers S is added after the d-1 th residual sub-module₁Adding a column of fully-connected layers S after the d-th residual sub-module₂；S₁，S₂For extracting modal sharing information; the ResNet50 network adopted by the invention is composed of 4 residual submodules, wherein d is 4, and d-1 is 3; full connection layer S₁，S₂The number of the neurons is set to 1024;

the discriminator consists of a visible light image classifier and an infrared light image classifier, which are shown in fig. 2;

initializing network weights for the ResNet50 network;

step 3, respectively inputting the visible light image set V and the infrared light image set T of N pedestrians into two independent ResNet50 networks for extracting the characteristic information of the pedestrians, and outputting d-1 group of visible light characteristic information V after the d-1 residual sub-module_d-1And d-1 group infrared light characteristic information t_d-1Respectively inputting the d-th residual error sub-module and outputting the d-th group of visible light characteristic information v_dAnd d group infrared light characteristic information t_d；

Step 4, constructing the d-1 th sample feature space X_d-1；

Selecting visible light characteristic information and infrared light characteristic information of P pedestrians from all the characteristic information output by the d-1 th residual error submodule, wherein the visible light characteristic information v of each pedestrian_i,d-1And infrared light characteristic information t_i,d-1Respectively selecting K pieces of feature information to construct a d-1 th sample feature space X_d-1(ii) a In the invention, P is 16, K is 4;

the d-1 th sample feature space X_d-1Are input together to the subsequent fully-connected layer S₁And d-1 group visible light feature vector v 'is output for extracting modal sharing information'_d-1And infrared light feature vector t'_d-1；

Step 5, constructing the d sample feature space X_d；

Selecting visible light characteristic information and infrared light characteristic information of P pedestrians from all the characteristic information output by the d residual error submodule, wherein the visible light characteristic information v of each pedestrian_i,dAnd infrared light characteristic information t_i,dRespectively selecting K pieces of feature information to construct a d-th sample feature space X_d；P＝16，K＝4；

Then the d sample feature space X_dAre input together to the subsequent fully-connected layer S₂For extracting modal sharing information and outputting d-th group visible light feature vector v'_dAnd infrared light feature vector t'_d；

Step 6, setting the d-1 group visible light feature vector v'_d-1Inputting into a visible light image classifier, outputting an initial probability distribution GV of visible light, and converting the d-1 th group of infrared light feature vectors t'_d-1Inputting the infrared light image into an infrared light image classifier, and outputting an initial probability distribution GT of the infrared light;

construction of identity loss function L using equation (1)_ID：

Step 7, from the d-1 th sample feature space X_d-1To select the kth characteristic information of the a-th pedestrian

Is denoted as the z-th positive sample feature vector, and

In the formula (2), the reaction mixture is,

representing anchor sample feature vectors

And the z-th positive sample feature vector

The Euclidean distance of (a) is,

Euclidean distance of (p)₁Is a mixed ternary loss function L_TRI1(X_d-1) A predefined minimum interval; is set to rho₁0.5. The distance between the anchor sample feature vector and the positive sample feature vector can be reduced by optimizing equation (2) and the distance between the anchor sample feature vector and the negative sample feature vector can be increased. As shown in fig. 3;

Is denoted as the b-th positive sample feature vector, and

In the formula (3), the reaction mixture is,

representing anchor sample feature vectors

And the b-th positive sample feature vector

The Euclidean distance of (a) is,

Euclidean distance of (p)₂Is a mixed ternary loss function L_TRI2(X_d) A predefined minimum interval; is set to rho₂＝0.5。

L_TRI＝L_TRI1+L_TRI2 (4)

Establishing a global penalty function L using equation (5)_ALL：

L_ALL＝L_ID+βL_TRI (5)

In the formula (5), β represents a mixed ternary loss function L_TRIThe coefficient of (a). The coefficient β is set to β 1.4.

step 10, setting a d-1 group visible light feature vector v'_d-1Inputting the infrared light characteristic vectors t ' of the d-1 group into a visible light image classifier in a symmetrical convolutional neural network model after preliminary training, outputting the probability distribution GV ' of visible light '_d-1Inputting the infrared light image into an infrared light image classifier in a preliminarily trained symmetric convolutional neural network model, and outputting the probability distribution GT' of infrared light; d-1 group visible light feature vector v'_d-1Inputting the pseudo visible light probability distribution GV' into an infrared light classifier in the preliminarily trained symmetric convolutional neural network model;

construction of divergence loss function L between pseudo visible light eigenvector GV 'and visible light probability distribution GV' using equation (6)_KL：

L_KL＝KL(GV″,GV′) (6)

In the formula (6), the reaction mixture is,

to represent

And

difference values of probability distributions;

establishing a discriminator loss function L using equation (7)_DIS：

L_DIS＝L_ID-αL_KL (7)

In the formula (7), α represents L_KLThe coefficient of (a). The coefficient α is set to α ═ 1.

L_GEN＝αL_KL+βL_TRI (8)

The invention has performed a verification experiment on the setting of α, β, and fig. 4 is the effect of the coefficient α on the RegDB data set in the invention; FIG. 5 is a graph of the effect of the coefficient α on the SYSU-MM01 data set; the performance of the invention is proved to be better when alpha is 1;

FIG. 6 is a graph of the effect of the coefficient β on the RegDB data set in the present invention; FIG. 7 is a graph of the effect of the coefficient β on the SYSU-MM01 data set; when alpha is 1 and beta is 1.4, the performance of the invention is optimal, and experiments prove that good results can be obtained in a wider value range of alpha and beta, which reflects the superiority of the invention.

And 12, sequentially carrying out optimization solving on the formula (5), the formula (7) and the formula (8) by a gradient descent method. The invention optimizes a network model using an adaptive gradient optimizer (Adam).

after training in turn, make L_ALL，L_DIS，L_GENConverge to optimum in antagonistic learning when L_DISWhen the optimum is reached, the discriminator is optimum, when L is_GENWhen the optimal condition is reached, the generator is optimal, so that a final cross-modal pedestrian re-identification model of the symmetric convolutional neural network is obtained;

Example (b):

in order to prove the effectiveness of the invention, some comparative tests are carried out with other methods, and as shown in table 1, compared with other methods in the prior art, the effect of the invention is obviously better, and the effectiveness of the invention is proved. Ablation experiments were also performed on each module of the network of the present invention, and the results of the experiments are shown in table 2, demonstrating the effectiveness of each module of the present invention.

Table 1 is a graph comparing the effectiveness of the present invention with other methods

Table 2 shows the related ablation experimental graphs of the present invention

Experiments prove that the method can greatly relieve the problem that the existing pedestrian re-identification method is inaccurate in detection in the cross-mode, and still has higher detection precision under the condition of larger modal difference.

Claims

Translated fromChinese

1.一种基于对称卷积神经网络的跨模态行人重识别方法，其特征是按如下步骤进行：1. a cross-modal pedestrian re-identification method based on symmetric convolutional neural network is characterized in that carrying out as follows:

步骤1、采集N个行人的可见光图像集V，其中第i个行人的j张可见光图像记为V_i，且V_i＝{V_i¹,V_i²,...,V_i^j}，V_i^j表示第i个行人的第j张可见光图片，并为第i个行人赋予第i身份信息y_i；i＝1,2,…,N；Step 1. Collect the visible light image set V of N pedestrians, wherein j visible light images of the ith pedestrian are denoted as V_i , and V_i ={V_i¹ ,V_i² ,...,V_i^j },_Vi^j represents the jth visible light picture of the ith pedestrian, and gives the ith identity information yi to the_ith pedestrian; i=1,2,...,N;

用红外光相机或者深度相机采集N个行人的红外光图像集T，其中第i个行人的m张红外光图像记为T_i，且T_i＝{T_i¹,T_i²,...,T_i^m}，T_i^m表示第i个行人的第m个红外光图像；Use an infrared light camera or a depth camera to collect the infrared light image set T of N pedestrians, wherein m infrared light images of the ith pedestrian are denoted as T_i , and T_i ={T_i¹ ,T_i² ,... , T_i^m }, T_i^m represents the m-th infrared light image of the i-th pedestrian;

由其他已知身份信息的行人的可见光图片和红外光图像来构建检索库；Build a retrieval library from visible and infrared images of pedestrians with other known identity information;

步骤2、构建由生成器和鉴别器组成的对称卷积神经网络；Step 2. Build a symmetric convolutional neural network consisting of a generator and a discriminator;

所述生成器由两列独立的ResNet50网络构成，其中ResNet50网络由d个残差子模块构成，在第d-1个残差子模块之后添加一列全连接层S₁，在第d个残差子模块之后添加一列全连接层S₂；The generator is composed of two independent ResNet50 networks, where the ResNet50 network is composed of d residual sub-modules, and a column of fully connected layer S₁ is added after the d-1 th residual sub-module. A column of fully connected layer S₂ is added after the sub-module;

所述鉴别器由可见光图像分类器和红外光图像分类器组成；The discriminator is composed of a visible light image classifier and an infrared light image classifier;

初始化所述ResNet50网络的网络权重；Initialize the network weights of the ResNet50 network;

采用随机初始化方式来初始化所述全连接层与鉴别器的参数；A random initialization method is used to initialize the parameters of the fully connected layer and the discriminator;

步骤3、将所述N个行人的可见光图像集V和红外光图像集T分别输入到两列独立的ResNet50网络中，并在第d-1个残差子模块后输出第d-1组可见光特征信息v_d-1，以及第d-1组红外光特征信息t_d-1，再分别输入所述第d个残差子模块后，输出第d组可见光特征信息v_d以及第d组红外光特征信息t_d；Step 3. Input the visible light image set V and infrared light image set T of the N pedestrians into two independent ResNet50 networks respectively, and output the d-1th group of visible light after the d-1th residual sub-module The characteristic information v_d-1 and the d-1 group of infrared light characteristic information t_d-1 are respectively input to the d-th residual sub-module, and the d-th group of visible light characteristic information v_d and the d-th group of infrared light are output. light characteristic information t_d ;

步骤4、构建第d-1个样本特征空间X_d-1；Step 4, constructing the d-1 th sample feature space X_d-1 ;

从第d-1个残差子模块输出的所有特征信息中，选取P个行人的可见光特征信息和红外光特征信息，每个行人的可见光特征信息v_i,d-1和红外光特征信息t_i,d-1各选取K个特征信息，构建第d-1个样本特征空间X_d-1；From all the feature information output by the d-1th residual sub-module, select the visible light feature information and infrared light feature information of P pedestrians, the visible light feature information v_{i, d-1} and the infrared light feature information t of each pedestrian Each of_{i, d-1} selects K pieces of feature information to construct the d-1 th sample feature space X_d-1 ;

将所述第d-1个样本特征空间X_d-1一起输入到后续的全连接层S₁中，并输出第d-1组可见光特征向量v′_d-1和红外光特征向量t′_d-1；The d-1 th sample feature space X_d-1 is input together into the subsequent fully connected layer S₁ , and the d-1 th group of visible light feature vector v'_d-1 and infrared light feature vector t'_d are output_-1 ;

步骤5、构建第d个样本特征空间X_d；Step 5. Construct the d-th sample feature space X_d ;

从第d个残差子模块输出的所有特征信息中，选取P个行人的可见光特征信息和红外光特征信息，每个行人的可见光特征信息v_i,d和红外光特征信息t_i,d各选取K个特征信息，构建第d个样本特征空间X_d；From all the feature information output by the d-th residual sub-module, select the visible light feature information and infrared light feature information of P pedestrians, and the visible light feature information v_i,d and infrared light feature information t_i,d of each pedestrian are respectively Select K pieces of feature information to construct the d-th sample feature space X_d ;

再将所述第d个样本特征空间X_d一起输入到后续的全连接层S₂中，并输出第d组可见光特征向量v′_d和红外光特征向量t′_d；Then input the d-th sample feature space X_d into the subsequent fully connected layer S₂ , and output the d-th group of visible light feature vectors v′_d and infrared light feature vectors t′_d ;

步骤6、将所述第d-1组可见光特征向量v′_d-1输入所述可见光图像分类器中，输出可见光的初始概率分布GV，将第d-1组红外光特征向量t′_d-1输入到所述红外光图像分类器中，并输出红外光的初始概率分布GT；Step 6. Input the d-1th group of visible light feature vectors v′_d-1 into the visible light image classifier, output the initial probability distribution GV of visible light, and put the d-1th group of infrared light feature vectors t′_{d- 1} is input into the infrared light image classifier, and the initial probability distribution GT of infrared light is output;

利用式(1)构建身份损失函数L_ID：Use formula (1) to construct the identity loss function L_ID :

步骤7、从所述第d-1个样本特征空间X_d-1中选择第a个行人的第k个特征信息

记为锚点样本特征向量，则与锚点样本特征向量

具有相同身份信息的第a个行人的第z个特征信息

记为第z个正样本特征向量，与

具有不同身份信息的第f个行人的第c个特征信息

记为第c个负样本特征向量，则利用式(2)建立混合三元损失函数L_TRI1(X_d-1)：Step 7. Select the k-th feature information of the a-th pedestrian from the d-1-th sample feature space X_d-1

Denoted as the anchor point sample feature vector, then the same as the anchor point sample feature vector

The z-th feature information of the a-th pedestrian with the same identity information

Denoted as the zth positive sample feature vector, and

The c-th feature information of the f-th pedestrian with different identity information

Denoted as the c-th negative sample feature vector, then use formula (2) to establish a mixed ternary loss function L_TRI1 (X_d-1 ):

式(2)中,

代表锚点样本特征向量

与第z个正样本特征向量

的欧式距离，

代表锚点样本特征向量与第f个行人的第c个负样本特征向量

的欧式距离，ρ₁为混合三元损失函数L_TRI1(X_d-1)预定义的最小间隔；In formula (2),

Represents the anchor sample feature vector

with the zth positive sample feature vector

the Euclidean distance,

Represents the anchor point sample feature vector and the c-th negative sample feature vector of the f-th pedestrian

The Euclidean distance of , ρ₁ is the minimum interval predefined by the hybrid ternary loss function L_TRI1 (X_d-1 );

步骤8、从所述第d个样本特征空间X_d中选择第r个行人的第s个特征信息

记为锚点样本特征向量，则与锚点样本特征向量

具有相同身份信息的第r个行人的第b个特征信息

记为第b个正样本特征向量，与

具有不同身份信息的第h个行人的第q个特征信息

记为第q个负样本特征向量，则利用式(3)建立混合三元损失函数L_TRI2(X_d)：Step 8. Select the s-th feature information of the r-th pedestrian from the d-th sample feature space X_d

The b-th feature information of the r-th pedestrian with the same identity information

Denoted as the b-th positive sample feature vector, and

The qth feature information of the hth pedestrian with different identity information

Denote it as the qth negative sample feature vector, then use formula (3) to establish a mixed ternary loss function L_TRI2 (X_d ):

式(3)中,

代表锚点样本特征向量

与第b个正样本特征向量

的欧式距离，

代表锚点样本特征向量与第h个行人的第q个负样本特征向量

的欧式距离，ρ₂为混合三元损失函数L_TRI2(X_d)预定义的最小间隔；In formula (3),

Represents the anchor sample feature vector

with the b-th positive sample feature vector

the Euclidean distance,

Represents the anchor point sample feature vector and the qth negative sample feature vector of the hth pedestrian

The Euclidean distance of , ρ₂ is the minimum interval predefined by the hybrid ternary loss function L_TRI2 (X_d );

步骤9、利用式(4)建立混合三元损失函数L_TRI：Step 9. Use formula (4) to establish a mixed ternary loss function L_TRI :

L_TRI＝L_TRI1+L_TRI2 (4)L_TRI =L_TRI1 +L_TRI2 (4)

利用式(5)建立全局损失函数L_ALL：Use formula (5) to establish the global loss function L_ALL :

L_ALL＝L_ID+βL_TRI (5)L_ALL = L_ID + βL_TRI (5)

式(5)中，β表示混三元损失函数L_TRI的系数；In formula (5), β represents the coefficient of the mixed ternary loss function L_TRI ;

通过随机梯度下降法对式(5)进行优化求解，并进行梯度反向传播，训练所述对称卷积神经网的各个参数，得到初步训练后的对称卷积神经网络模型；Formula (5) is optimized and solved by the stochastic gradient descent method, and gradient back-propagation is performed, and each parameter of the symmetric convolutional neural network is trained to obtain the symmetric convolutional neural network model after preliminary training;

步骤10、将所述第d-1组可见光特征向量v′_d-1输入所述初步训练后的对称卷积神经网络模型中可见光图像分类器中，输出可见光的概率分布GV′，将第d-1组红外光特征向量t′_d-1输入到所述初步训练后的对称卷积神经网络模型中红外光图像分类器中，并输出红外光的概率分布GT′；将所述第d-1组可见光特征向量v′_d-1输入到所述初步训练后的对称卷积神经网络模型中红外光分类器中得到伪可见光概率分布GV″；Step 10: Input the visible light feature vector v'_d-1 of the d-1th group into the visible light image classifier in the symmetric convolutional neural network model after preliminary training, output the probability distribution GV' of visible light, and put the d-th -1 group of infrared light feature vectors t'_d-1 are input into the infrared light image classifier in the symmetric convolutional neural network model after preliminary training, and the probability distribution GT' of infrared light is output; the d- A group of visible light feature vectors v′_d-1 are input into the infrared light classifier in the symmetric convolutional neural network model after preliminary training to obtain a pseudo visible light probability distribution GV″;

利用式(6)构建所述伪可见光特征向量GV″与可见光概率分布GV′之间的散度损失函数L_KL：Use formula (6) to construct the divergence loss function L_KL between the pseudo visible light feature vector GV″ and the visible light probability distribution GV′:

L_KL＝KL(GV″,GV′) (6)L_KL =KL(GV″,GV′) (6)

式(6)中，KL(·,·)表示两者概率分布的差异值；In formula (6), KL(·,·) represents the difference between the two probability distributions;

利用式(7)建立鉴别器损失函数L_DIS：Use equation (7) to establish the discriminator loss function L_DIS :

L_DIS＝L_ID-αL_KL (7)L_DIS =L_ID -αL_KL (7)

式(7)中，α代表L_KL的系数；In formula (7), α represents the coefficient of L_KL ;

步骤11、利用式(8)建立生成器损失函数L_GEN：Step 11. Use formula (8) to establish the generator loss function L_GEN :

L_GEN＝αL_KL+βL_TRI (8)L_GEN = αL_KL + βL_TRI (8)

步骤12、通过梯度下降法依次对式(5)、式(7)、式(8)进行优化求解：Step 12: Optimizing and solving equations (5), (7), and (8) in turn by gradient descent method:

首先对式(5)进行优化求解，训练网络所有参数；First, optimize and solve equation (5), and train all parameters of the network;

其次对式(7)进行优化求解，在梯度反向传播过程中，仅对鉴别器的梯度进行反向传播，将生成器的梯度置零，从而冻结生成器参数，训练鉴别器参数；Secondly, formula (7) is optimized and solved. In the process of gradient backpropagation, only the gradient of the discriminator is backpropagated, and the gradient of the generator is set to zero, thereby freezing the generator parameters and training the discriminator parameters;

最后对式(8)进行优化求解，在梯度反向传播过程中，仅对生成器的梯度进行反向传播，将鉴别器的梯度置零，从而冻结鉴别器参数，训练生成器参数；Finally, formula (8) is optimized and solved. In the process of gradient backpropagation, only the gradient of the generator is backpropagated, and the gradient of the discriminator is set to zero, thereby freezing the discriminator parameters and training the generator parameters;

依次训练后使得L_ALL，L_DIS，L_GEN在对抗学习中收敛到最优，当L_DIS达到最优时，鉴别器达到最优，当L_GEN达到最优时，生成器达到最优，从而获得最终的对称卷积神经网络跨模态行人重识别模型；After training in sequence, L_ALL , L_DIS , and L_GEN converge to the optimum in adversarial learning. When L_DIS reaches the optimum, the discriminator reaches the optimum, and when L_GEN reaches the optimum, the generator reaches the optimum, thus Obtain the final symmetric convolutional neural network cross-modal person re-identification model;

步骤13、利用最终的对称卷积神经网络模型对跨模态行人重识别进行查询匹配；Step 13, using the final symmetric convolutional neural network model to query and match cross-modal pedestrian re-identification;

将待查询的行人图像输入最终的对称卷积神经网络模型中提取特征，然后与检索库中行人的特征进行相似度比对，并按照相似度的高低排序，从排序列表中找到对应的行人身份信息，从而得到识别结果。Input the pedestrian image to be queried into the final symmetric convolutional neural network model to extract features, then compare the similarity with the features of the pedestrians in the retrieval database, and sort them according to the similarity, and find the corresponding pedestrian identity from the sorted list. information to obtain identification results.