CN106203395A - Face character recognition methods based on the study of the multitask degree of depth - Google Patents

Abstract

Translated fromChinese

基于多任务深度学习的人脸属性识别方法，涉及计算机视觉中的人脸属性识别。准备图像数据集；对图像数据集中的每幅图像逐一进行人脸检测；对所有检测到的人脸进行人脸关键点检测；对检测到的人脸关键点将每幅人脸根据人脸对齐方法，对齐到标准的人脸图像上，构成人脸图像训练集；计算出训练集中的平均人脸图像；构建多任务深度卷积神经网络，把人脸图像训练集中的每幅人脸图像减去平均人脸图像后进行网络参数的训练，得卷积神经网络模型；将待识别的测试图像进行人脸检测和人脸关键点检测，并根据人脸关键点将图像中的人脸对齐到标准的人脸图像上；将标准的人脸图像减去平均人脸图像，并放到构建好的卷积神经网络模型中进行前馈运算操作，即得。A face attribute recognition method based on multi-task deep learning relates to face attribute recognition in computer vision. Prepare the image data set; perform face detection on each image in the image data set one by one; perform face key point detection on all detected faces; align each face according to the face for the detected face key points method, aligned to the standard face image to form a face image training set; calculate the average face image in the training set; construct a multi-task deep convolutional neural network, subtract each face image in the face image training set After removing the average face image, train the network parameters to obtain a convolutional neural network model; perform face detection and face key point detection on the test image to be recognized, and align the face in the image to the On the standard face image; subtract the average face image from the standard face image, and put it into the constructed convolutional neural network model for feed-forward operation, that is.

Description

Translated fromChinese

基于多任务深度学习的人脸属性识别方法Face attribute recognition method based on multi-task deep learning

技术领域technical field

本发明涉及计算机视觉中的人脸属性识别，尤其是涉及一种基于多任务深度学习的人脸属性识别方法。The invention relates to face attribute recognition in computer vision, in particular to a face attribute recognition method based on multi-task deep learning.

背景技术Background technique

基于图像的人脸属性识别方法是根据给定的输入图像，利用模式识别技术判断图像中人脸属性的过程。人脸图像中包含的人脸属性主要有：年龄、性别、表情、种族、是否戴眼镜、是否化妆等。利用计算机自动地进行人脸属性识别，可以有效地提高人机交互性能，具有非常重要的实际应用价值。人脸属性识别的流程包括：人脸检测技术、人脸图像预处理技术、人脸特征提取、人脸属性分类器的训练等步骤。其中，人脸特征提取和人脸属性分类器性能的好坏直接影响到了最终人脸属性识别的性能。The image-based face attribute recognition method is a process of judging the face attributes in the image by using pattern recognition technology according to a given input image. The face attributes contained in the face image mainly include: age, gender, expression, race, whether to wear glasses, whether to make up, etc. Using computer to automatically recognize face attributes can effectively improve the performance of human-computer interaction and has very important practical application value. The process of face attribute recognition includes: face detection technology, face image preprocessing technology, face feature extraction, face attribute classifier training and other steps. Among them, the performance of face feature extraction and face attribute classifier directly affects the performance of final face attribute recognition.

目前，人脸属性识别技术主要分为两个步骤完成：人脸特征提取和人脸属性分类器训练。人脸特征提取技术根据特征的获取方式的不同分为两大类：手工设计特征和自动学习特征。人脸特征的好坏，直接影响到分类器的性能。手工设计特征主要有：SIFT特征(D.G.Lowe.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91–110)，LBP特征(T.Ahonen,A.Hadid,M.Pietikainen.Face description with local binary patterns:Application to face recognition[J].IEEE Transactions on Pattern Analysis andMachine Intelligence,2006,28(12):2037–2041)，HOG特征(N.Dalal,B.Triggs.Histograms of oriented gradients for human detection[C].IEEEComputer Society Conference onComputerVision and Pattern Recognition,2005,886–893)。但是这些手工设计特征主要依赖于专家经验设计，并且通常提取的特征也很难对各种任务均有效。其次，手工设计的特征提取与分类器设计相分离，导致选择的特征不会最合适于特定的分类器。At present, face attribute recognition technology is mainly divided into two steps: face feature extraction and face attribute classifier training. Face feature extraction technology is divided into two categories according to the different ways of feature acquisition: manual design features and automatic learning features. The quality of facial features directly affects the performance of the classifier. The manual design features mainly include: SIFT features (D.G.Lowe.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision,2004,60(2):91–110), LBP features (T.Ahonen,A .Hadid,M.Pietikainen.Face description with local binary patterns:Application to face recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006,28(12):2037–2041), HOG features (N.Dalal,B . Triggs. Histograms of oriented gradients for human detection [C]. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005, 886–893). However, these manually designed features mainly rely on expert experience design, and usually the extracted features are difficult to be effective for various tasks. Second, hand-designed feature extraction is separated from classifier design, resulting in the selected features not being most suitable for a specific classifier.

最近深度学习成为了计算机视觉领域的研究热点之一。与传统的将识别任务分解为特征提取和分类器训练不同，深度学习将二者有机地结合起来，直接将原始的数据作为输入，同时进行特征提取学习和分类器训练学习。深度学习将特征学习和分类器训练放在同一个框架中进行统计学习的方法，有效地避免了特征提取与目标任务分类器之间的意义鸿沟，使得二者之间相互促进，并且克服了人工设计特征的麻烦。由多层神经网络构成的深度模型，具有自动获取由低级到高级、由简单到复杂、通用到专用的特征的特点。例如：在典型的图像分类网络中，前面几层网络常常提取到的是边缘信息，中间层提取到的是角信息，后面层提取到的是轮廓信息和目标信息等。层次越低提取到的特征越简单，越通用，并逐步提取目标任务相关的特征。Recently, deep learning has become one of the research hotspots in the field of computer vision. Different from the traditional decomposition of recognition tasks into feature extraction and classifier training, deep learning organically combines the two, directly takes the original data as input, and performs feature extraction learning and classifier training learning at the same time. Deep learning puts feature learning and classifier training in the same framework for statistical learning, effectively avoiding the meaning gap between feature extraction and target task classifiers, making the two promote each other, and overcoming artificial Trouble with designing features. The deep model composed of multi-layer neural network has the characteristics of automatically acquiring features from low-level to high-level, from simple to complex, and from general to special. For example: in a typical image classification network, the first few layers of the network often extract edge information, the middle layer extracts corner information, and the latter layer extracts contour information and target information. The lower the level, the simpler and more general the features extracted, and gradually extract the features related to the target task.

训练一个深度学习模型，常常需要大量的标签数据，才能避免学习模型过拟合少量的训练数据。然而，获取大量的标签数据往往是非常费时费力，探索利用深度网络模型逐层获取不同特征的特性来解决数据不足的问题，是一个值得解决的关键问题。Training a deep learning model often requires a large amount of labeled data to avoid overfitting the learning model to a small amount of training data. However, obtaining a large amount of labeled data is often very time-consuming and laborious. Exploring the use of deep network models to obtain different features layer by layer to solve the problem of insufficient data is a key problem worth solving.

发明内容Contents of the invention

本发明的目的是提供一种基于多任务深度学习的人脸属性识别方法。The object of the present invention is to provide a method for recognizing face attributes based on multi-task deep learning.

本发明包括如下步骤：The present invention comprises the steps:

A.准备图像数据集，其包含大量的人脸以及对应的人脸属性标签。A. Prepare an image dataset, which contains a large number of faces and corresponding face attribute labels.

B.对图像数据集中的每幅图像逐一进行人脸检测，获取人脸在每幅图像中的位置。B. Perform face detection on each image in the image data set one by one, and obtain the position of the face in each image.

C.对所有检测到的人脸进行人脸关键点检测，获取人脸关键点在每幅图像中的位置。C. Perform face key point detection on all detected faces, and obtain the position of the face key points in each image.

D.对检测到的人脸关键点将每幅人脸根据人脸对齐方法，对齐到标准的人脸图像上，构成人脸图像训练集。D. For the detected face key points, each face is aligned to a standard face image according to the face alignment method to form a face image training set.

E.对人脸图像训练集，计算出训练集中的平均人脸图像。E. For the face image training set, calculate the average face image in the training set.

F.构建多任务深度卷积神经网络，把人脸图像训练集中的每幅人脸图像减去平均人脸图像后进行网络参数的训练，得到卷积神经网络模型。F. Construct a multi-task deep convolutional neural network, subtract the average face image from each face image in the face image training set, and perform network parameter training to obtain a convolutional neural network model.

G.将待识别的测试图像分别进行人脸检测和人脸关键点检测，并根据人脸关键点将图像中的人脸对齐到标准的人脸图像上。G. Perform face detection and face key point detection on the test image to be recognized, and align the faces in the image to the standard face image according to the face key points.

H.将标准的人脸图像减去平均人脸图像，并放到构建好的卷积神经网络模型中进行前馈运算操作，即得到人脸的多个属性识别结果。H. Subtract the average face image from the standard face image, and put it into the constructed convolutional neural network model for feed-forward operation, that is, to obtain the recognition results of multiple attributes of the face.

在步骤A中，所述准备图像数据集可采用复杂场景下采集的多样性较好的包含人脸的图像数据，同时提供对应的K个人脸属性标签，其中K为学习的任务数且为自然数；本发明采用的基于多任务学习的卷积神经网络结构，可以不需要每张人脸图像同时具备所有的人脸属性标签数据，因而可以充分利用现有的人脸数据库进行组合形成大规模图像数据集。In step A, the prepared image data set can adopt the image data containing human faces with good diversity collected in complex scenes, and provide corresponding K human face attribute labels, where K is the number of tasks to be learned and is a natural number The convolutional neural network structure based on multi-task learning that the present invention adopts can not require each face image to possess all face attribute label data simultaneously, thus can make full use of existing face databases to combine to form large-scale images data set.

在步骤B中，所述对图像数据集中的每幅图像逐一进行人脸检测可采用常用的人脸检测方法，以得到人脸在每幅图像中的位置，所述常用的人脸检测方法可采用OpenCV自带的人脸检测方法。In step B, the face detection method for each image in the image data set can be used to obtain the position of the face in each image, and the commonly used face detection method can be Use the face detection method that comes with OpenCV.

在步骤C中，所述人脸关键点检测可采用常用的人脸关键点检测方法，以得到人脸关键点在每幅图像中的位置，所述常用的人脸关键点检测方法可采用Dlib自带的人脸关键点检测方法。In step C, the human face key point detection method can adopt the commonly used human face key point detection method to obtain the position of the human face key point in each image, and the commonly used human face key point detection method can use Dlib Built-in face key point detection method.

在步骤D中，所述人脸对齐方法为基于二维图像的仿射变换，具体包括以下步骤：In step D, the face alignment method is an affine transformation based on two-dimensional images, which specifically includes the following steps:

D1.根据人脸关键点和标准的人脸关键点的匹配关系，利用最小二乘法进行拟合得到一个最佳的变换矩阵。假设标准人脸关键点的d个坐标为其中为第i个标准人脸关键点坐标；d为人脸关键点数目且d为自然数；而检测到的人脸关键点坐标为{(x₁；y₁),(x₂；y₂),...,(x_d；y_d)}，其中(x_i,y_i)为检测到的人脸的第i个人脸关键点坐标，并将其扩展为src＝{(x₁；y₁；1),(x₂；y₂；1),...,(x_d；y_d；1)}，通过最小二乘法进行优化。具体计算公式如下：D1. According to the matching relationship between face key points and standard face key points, use the least squares method to fit to obtain an optimal transformation matrix. Suppose the d coordinates of standard face key points are in is the i-th standard face key point coordinates; d is the number of face key points and d is a natural number; and the detected face key point coordinates are {(x₁ ; y₁ ),(x₂ ; y₂ ),. ..,(x_d ; y_d )}, where (_xi , y_i ) is the i-th face key point coordinates of the detected face, and it is extended to src={(x₁ ; y₁ ; 1), (x₂ ; y₂ ; 1), . . . , (x_d ; y_d ; 1)}, optimized by the method of least squares. The specific calculation formula is as follows:

${\mathrm{<span><span>A</span>A</span>}}_{\mathrm{<span><span>0</span>0</span>}}<span><span>=</span>=</span>\mathrm{<span><span>arg</span>arg</span>}\underset{\mathrm{<span><span>A</span>A</span>}}{\mathrm{<span><span>m</span>m</span>}\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>n</span>no</span>}}<span><span>(</span>(</span><span><span>|</span>|</span><span><span>|</span>|</span>\mathrm{<span><span>A</span>A</span>}<span><span>\&CenterDot;</span>\&Center\; Dot;</span>\mathrm{<span><span>s</span>the\; s</span>}\mathrm{<span><span>r</span>r</span>}\mathrm{<span><span>c</span>c</span>}<span><span>-</span>-</span>\mathrm{<span><span>d</span>d</span>}\mathrm{<span><span>s</span>the\; s</span>}\mathrm{<span><span>t</span>t</span>}<span><span>|</span>|</span>{<span><span>|</span>|</span>}_{\mathrm{<span><span>2</span>2</span>}}^{\mathrm{<span><span>2</span>2</span>}}<span><span>)</span>)</span>$

其中表示仿射变换矩阵；A₀为最佳的变换矩阵；in Indicates the affine transformation matrix; A₀ is the best transformation matrix;

D2.利用优化得到的变换矩阵A₀将所有人脸图像进行对齐到标准的人脸图像上，并剪切成大小统一的图像。D2. Use the optimized transformation matrix A₀ to align all the face images to the standard face images, and cut them into images of uniform size.

在步骤E中，所述对人脸图像训练集，计算出训练集中的平均人脸图像的具体方法可为：In step E, described to face image training set, the concrete method that calculates the average face image in training set can be:

E1.均值图像为人脸图像训练集每个通道(包含了RGB共3个通道)计算得到的算术均值图像M，其中每个通道和M的计算方式如下：E1. The mean image is the arithmetic mean image M calculated for each channel of the face image training set (including a total of 3 channels of RGB), wherein the calculation method of each channel and M is as follows:

${\mathrm{<span><span>M</span>m</span>}}_{\mathrm{<span><span>R</span>R</span>}}<span><span>=</span>=</span>\frac{\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>X</span>x</span>}}_{\mathrm{<span><span>n</span>no</span>}}^{\mathrm{<span><span>R</span>R</span>}}}{\mathrm{<span><span>N</span>N</span>}}<span><span>,</span>,</span>{\mathrm{<span><span>M</span>m</span>}}_{\mathrm{<span><span>G</span>G</span>}}<span><span>=</span>=</span>\frac{\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>X</span>x</span>}}_{\mathrm{<span><span>n</span>no</span>}}^{\mathrm{<span><span>G</span>G</span>}}}{\mathrm{<span><span>N</span>N</span>}}<span><span>,</span>,</span>{\mathrm{<span><span>M</span>m</span>}}_{\mathrm{<span><span>B</span>B</span>}}<span><span>=</span>=</span>\frac{\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>X</span>x</span>}}_{\mathrm{<span><span>n</span>no</span>}}^{\mathrm{<span><span>B</span>B</span>}}}{\mathrm{<span><span>N</span>N</span>}}<span><span>,</span>,</span>$

其中，和分别为第n幅人脸图像的R通道，G通道和B通道；N为人脸图像训练集的总图像数，N为自然数。M_R，M_G和M_B分别为R通道，G通道和B通道的算术均值图像；in, and are the R channel, G channel and B channel of the nth face image respectively; N is the total number of images in the face image training set, and N is a natural number. M_R , M_G and M_B are the arithmetic mean images of R channel, G channel and B channel respectively;

E2.把RGB三个通道得到的算术均值图像组成平均人脸图像，计算公式为：E2. The arithmetic mean image obtained by the three channels of RGB is used to form an average face image, and the calculation formula is:

M＝[M_R,M_G,M_B]；M = [M_R , M_G , M_B ];

E3.将人脸图像训练集按照9︰1的比例进行划分，分为训练数据和验证数据。E3. Divide the face image training set according to the ratio of 9:1, and divide it into training data and verification data.

在步骤F中，所述构建多任务深度卷积神经网络的具体方法可为：In step F, the concrete method of described construction multi-task depth convolutional neural network can be:

F1.将训练数据随机打乱，设置每批大小为m个样本，进行数据划分，其中m为自然数。在训练深度卷积神经网络模型的过程中，利用批量梯度更新方法进行学习神经网络的权重参数；F1. Randomly shuffle the training data, set the size of each batch to m samples, and divide the data, where m is a natural number. In the process of training the deep convolutional neural network model, the weight parameters of the neural network are learned by using the batch gradient update method;

F2.设计卷积神经网络的结构，其包含卷积层、下采样层和全连接层。每个卷积层和全连接层采用非线性矫正激活函数；在多任务深度学习的人脸属性识别中，整体网络结构分为共享层和独有层两个部分，共享层为所有的任务所共享，多个任务共同参与共享层参数的训练。独有层则是每个任务单独占用，并用每个任务单独的数据进行参数学习，假设共享层的个数为S，其中S为自然数；独有层的个数为U，其中U为自然数；根据不同的多任务组合，设置不同的S和U的个数；F2. Design the structure of the convolutional neural network, which includes a convolutional layer, a downsampling layer, and a fully connected layer. Each convolutional layer and fully connected layer uses a non-linear correction activation function; in the face attribute recognition of multi-task deep learning, the overall network structure is divided into two parts: the shared layer and the unique layer, and the shared layer is used by all tasks. Sharing, multiple tasks participate in the training of shared layer parameters. The unique layer is occupied by each task separately, and uses the separate data of each task for parameter learning, assuming that the number of shared layers is S, where S is a natural number; the number of unique layers is U, where U is a natural number; According to different multi-task combinations, set different numbers of S and U;

F3.设置卷积神经网络结构中所需要的卷积层滤波器数目和特征图数目、滤波器大小、下采样层中核的大小、每一层的学习率、权重初始值等超参数；F3. Set the hyperparameters such as the number of convolutional layer filters and the number of feature maps required in the convolutional neural network structure, filter size, kernel size in the downsampling layer, learning rate of each layer, and weight initial value;

F4.在卷积神经网络训练的过程中，采用冲量和丢弃等训练技巧，用于加速卷积神经网络的训练；F4. In the process of convolutional neural network training, training techniques such as impulse and discarding are used to accelerate the training of convolutional neural network;

F5.对于训练好的网络模型参数，根据其在验证数据上的性能，判断是否停止训练；F5. For the trained network model parameters, judge whether to stop training according to its performance on the verification data;

F6.提取出训练好的网络模型参数W。F6. Extract the trained network model parameter W.

本发明利用深度卷积神经网络将人脸图像的特征提取和属性识别同时进行学习，可以使得学习到的特征更有利于分类器的识别，而无需分别进行特征提取和分类器训练。对于K个任务的多任务人脸属性识别，无需所有的训练数据同时具备K个标签属性，只需要具备一个或者多个属性的人脸数据都可以用来训练网络参数，并且各个不同的属性任务都可以因此而获益。The present invention uses a deep convolutional neural network to simultaneously learn the feature extraction and attribute recognition of the face image, which can make the learned features more conducive to the recognition of the classifier without separately performing feature extraction and classifier training. For multi-task face attribute recognition of K tasks, it is not necessary for all training data to have K label attributes at the same time, only face data with one or more attributes can be used to train network parameters, and each different attribute task All can benefit from this.

不同于传统的深度卷积神经网络只有单一的标签和单一的网络输出，基于多任务深度学习的卷积神经网络具有多个输出。网络训练的目标函数为多个Softmax损失函数和L2损失函数的组合。假设有K个任务需要共同学习。那么对于第i个分类任务的人脸属性识别，其损失函数定义如下：Different from the traditional deep convolutional neural network with only a single label and a single network output, the convolutional neural network based on multi-task deep learning has multiple outputs. The objective function of network training is a combination of multiple Softmax loss functions and L2 loss functions. Suppose there are K tasks that need to be learned together. Then for the face attribute recognition of the i-th classification task, the loss function is defined as follows:

${\mathrm{<span><span>L</span>L</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>=</span>=</span><span><span>-</span>-</span>\frac{\mathrm{<span><span>1</span>1</span>}}{\mathrm{<span><span>N</span>N</span>}}\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}\mathrm{<span><span>l</span>l</span>}\mathrm{<span><span>o</span>o</span>}\mathrm{<span><span>g</span>g</span>}<span><span>(</span>(</span>{\mathrm{<span><span>P</span>P</span>}}_{\mathrm{<span><span>n</span>no</span>}<span><span>,</span>,</span>{\mathrm{<span><span>y</span>the\; y</span>}}_{\mathrm{<span><span>i</span>i</span>}}}<span><span>)</span>)</span>$

其中表示Softmax损失函数对每一个属性类计算出的概率值；表示全连接分类输出在该目标类别中的值；C_i表示第i个任务的类别数目，i为自然数。in Indicates the probability value calculated by the Softmax loss function for each attribute class; Indicates the value of the fully connected classification output in the target category; C_i indicates the category number of the i-th task, and i is a natural number.

对于第j个回归任务的人脸属性识别，其损失函数定义如下：For the face attribute recognition of the jth regression task, its loss function is defined as follows:

${\mathrm{<span><span>L</span>L</span>}}_{\mathrm{<span><span>j</span>j</span>}}<span><span>=</span>=</span><span><span>-</span>-</span>\frac{\mathrm{<span><span>1</span>1</span>}}{\mathrm{<span><span>N</span>N</span>}}\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}<span><span>|</span>|</span><span><span>|</span>|</span>{\mathrm{<span><span>y</span>the\; y</span>}}_{\mathrm{<span><span>n</span>no</span>}}<span><span>-</span>-</span>{\stackrel{<span><span>\&OverBar;</span>\&OverBar;</span>}{\mathrm{<span><span>y</span>the\; y</span>}}}_{\mathrm{<span><span>n</span>no</span>}}<span><span>|</span>|</span>{<span><span>|</span>|</span>}_{\mathrm{<span><span>2</span>2</span>}}^{\mathrm{<span><span>2</span>2</span>}}$

其中y_n为真实的标签值，为回归器的预测值。where y_n is the real label value, is the predicted value of the regressor.

在网络训练中，所有任务的代价损失函数进行组合，并形成总优化目标函数为：In network training, the cost loss functions of all tasks are combined to form the total optimization objective function as:

$\mathrm{<span><span>L</span>L</span>}<span><span>=</span>=</span>\underset{\mathrm{<span><span>k</span>k</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>K</span>K</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>\&alpha;</span>\&alpha;</span>}}_{\mathrm{<span><span>k</span>k</span>}}{\mathrm{<span><span>L</span>L</span>}}_{\mathrm{<span><span>k</span>k</span>}}$

其中α_k表示第k个任务的损失函数占总的损失函数的权重。默认情况下α_k的值都为1，表示各个任务同等重要。Where α_k represents the weight of the loss function of the kth task in the total loss function. By default, the value of α_k is 1, indicating that each task is equally important.

与现有的技术相比，本发明具有减少训练网络模型参数所需数据量要求，降低了过拟合训练数据的风险，减少了平摊在单个任务中的识别时间，同时有效提高人脸属性识别准确率的优点。Compared with the existing technology, the present invention has the requirement of reducing the amount of data required for training network model parameters, reducing the risk of over-fitting training data, reducing the recognition time shared in a single task, and effectively improving face attributes The advantage of recognition accuracy.

附图说明Description of drawings

图1为女性人脸微笑、性别、吸引力三个属性示意图。Figure 1 is a schematic diagram of the three attributes of female faces: smile, gender, and attractiveness.

图2为男性人脸微笑、性别、吸引力三个属性示意图。Figure 2 is a schematic diagram of three attributes of a male face: smile, gender, and attractiveness.

具体实施方式detailed description

下面结合附图和实施例对本发明的方法作详细说明。The method of the present invention will be described in detail below in conjunction with the accompanying drawings and embodiments.

本发明包括以下步骤：The present invention comprises the following steps:

S1.准备图像数据集，其包含大量的人脸以及对应的人脸属性标签。本实例采用的人脸属性数据库为CelebrayA数据库中的图像数据集，其包含了超过20多万张人脸图像以及40个人脸属性。采用其中的三个代表性的人脸属性任务(K＝3)：人脸性别属性、人脸微笑属性以及人脸吸引力属性进行说明。三个属性的示意图如图1和2所示，标签分别设置为y₁,y₂,y₃。S1. Prepare an image data set, which contains a large number of faces and corresponding face attribute labels. The face attribute database used in this example is the image dataset in the CelebrayA database, which contains more than 200,000 face images and 40 face attributes. Three representative face attribute tasks (K=3): face gender attribute, face smile attribute and face attractiveness attribute are used for illustration. The schematic diagrams of the three attributes are shown in Figures 1 and 2, and the labels are set to y₁ , y₂ , and y₃ respectively.

S2.对图像数据集中的每幅图像逐一进行人脸检测，获取人脸在每幅图像中的位置。本步骤可采用现有的任意一种人脸检测方法进行人脸检测。本实例采用OpenCV自带的人脸检测方法，其具有能够快速检测人脸的优点。S2. Perform face detection on each image in the image data set one by one, and obtain the position of the face in each image. In this step, any existing face detection method may be used for face detection. This example uses the face detection method that comes with OpenCV, which has the advantage of being able to detect faces quickly.

S3.对所有检测到的人脸进行人脸关键点检测，获取人脸关键点在每幅图像中的位置。本步骤可采用现有的任意一种人脸关键点检测方法进行检测。本实施例中采用Dlib自带的人脸关键点检测方法，可以得到68个人脸关键点。S3. Perform face key point detection on all detected faces, and obtain the position of the face key points in each image. In this step, any existing face key point detection method can be used for detection. In this embodiment, the face key point detection method provided by Dlib can be used to obtain 68 face key points.

S4.对检测到的人脸关键点将每幅人脸根据人脸对齐方法，对齐到标准的人脸图像上，构成人脸图像训练集，具体包括：S4. For the detected face key points, each face is aligned to a standard face image according to the face alignment method to form a face image training set, which specifically includes:

(1)根据人脸关键点和标准的人脸关键点的匹配关系，利用最小二乘法进行拟合得到一个最佳的变换矩阵。假设标准人脸关键点的68个坐标为其中为第i个标准人脸关键点坐标；d为人脸关键点数目且d为自然数；而检测到的人脸关键点坐标为{(x₁；y₁),(x₂；y₂),...,(x₆₈；y₆₈)}，其中(x_i,y_i)为检测到的人脸的第i个人脸关键点坐标，并将其扩展为src＝{(x₁；y₁；1),(x₂；y₂；1),...,(x₆₈；y₆₈；1)}，通过最小二乘法进行优化。具体计算公式如下所示，(1) According to the matching relationship between the key points of the face and the standard key points of the face, the least square method is used for fitting to obtain an optimal transformation matrix. Suppose the 68 coordinates of standard face key points are in is the i-th standard face key point coordinates; d is the number of face key points and d is a natural number; and the detected face key point coordinates are {(x₁ ; y₁ ),(x₂ ; y₂ ),. .., (x₆₈ ; y₆₈ )}, where (_xi , y_i ) is the i-th face key point coordinates of the detected face, and it is expanded to src={(x₁ ; y₁ ; 1), (x₂ ; y₂ ; 1), . . . , (x₆₈ ; y₆₈ ; 1)}, optimized by the method of least squares. The specific calculation formula is as follows,

${\mathrm{<span><span>A</span>A</span>}}_{\mathrm{<span><span>0</span>0</span>}}<span><span>=</span>=</span>\mathrm{<span><span>arg</span>arg</span>}\underset{\mathrm{<span><span>A</span>A</span>}}{\mathrm{<span><span>m</span>m</span>}\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>n</span>no</span>}}<span><span>(</span>(</span><span><span>|</span>|</span><span><span>|</span>|</span>\mathrm{<span><span>A</span>A</span>}<span><span>\&CenterDot;</span>\&Center\; Dot;</span>\mathrm{<span><span>s</span>the\; s</span>}\mathrm{<span><span>r</span>r</span>}\mathrm{<span><span>c</span>c</span>}<span><span>-</span>-</span>\mathrm{<span><span>d</span>d</span>}\mathrm{<span><span>s</span>the\; s</span>}\mathrm{<span><span>t</span>t</span>}<span><span>|</span>|</span>{<span><span>|</span>|</span>}_{\mathrm{<span><span>2</span>2</span>}}^{\mathrm{<span><span>2</span>2</span>}}<span><span>)</span>)</span>$

其中表示仿射变换矩阵；A₀为最佳的变换矩阵。in Indicates the affine transformation matrix; A₀ is the best transformation matrix.

(2)利用优化得到的变换矩阵A₀将所有人脸图像进行对齐到标准的人脸图像上，并剪切成大小统一为128×128的图像。(2) Use the optimized transformation matrix A₀ to align all face images to the standard face image, and cut them into images with a uniform size of 128×128.

S5.对人脸图像训练集，计算出训练集中的平均人脸图像，具体包括：S5. To the face image training set, calculate the average face image in the training set, specifically including:

(1)均值图像为人脸图像训练集每个通道(包含了RGB共3个通道)计算得到的算术均值图像M，其中每个通道和M的计算方式如下所示，(1) The mean image is the arithmetic mean image M calculated for each channel of the face image training set (including 3 channels of RGB), and the calculation method of each channel and M is as follows,

${\mathrm{<span><span>M</span>m</span>}}_{\mathrm{<span><span>R</span>R</span>}}<span><span>=</span>=</span>\frac{\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>X</span>x</span>}}_{\mathrm{<span><span>n</span>no</span>}}^{\mathrm{<span><span>R</span>R</span>}}}{\mathrm{<span><span>N</span>N</span>}}<span><span>,</span>,</span>{\mathrm{<span><span>M</span>m</span>}}_{\mathrm{<span><span>G</span>G</span>}}<span><span>=</span>=</span>\frac{\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>X</span>x</span>}}_{\mathrm{<span><span>n</span>no</span>}}^{\mathrm{<span><span>G</span>G</span>}}}{\mathrm{<span><span>N</span>N</span>}}<span><span>,</span>,</span>{\mathrm{<span><span>M</span>m</span>}}_{\mathrm{<span><span>B</span>B</span>}}<span><span>=</span>=</span>\frac{\underset{\mathrm{<span><span>n</span>no</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>X</span>x</span>}}_{\mathrm{<span><span>n</span>no</span>}}^{\mathrm{<span><span>B</span>B</span>}}}{\mathrm{<span><span>N</span>N</span>}}<span><span>,</span>,</span>$

其中和分别为第n幅人脸图像的R通道，G通道和B通道；N为人脸图像训练集的总图像数，N为自然数。M_R，M_G和M_B分别为R通道，G通道和B通道的算术均值图像。in and are the R channel, G channel and B channel of the nth face image respectively; N is the total number of images in the face image training set, and N is a natural number. M_R , M_G and M_B are the arithmetic mean images of R channel, G channel and B channel respectively.

(2)把RGB三个通道得到的算术均值图像组成平均人脸图像，计算公式为M＝[M_R,M_G,M_B]。(2) Compose the arithmetic mean image obtained by the three channels of RGB into an average face image, and the calculation formula is M=[M_R , M_G , M_B ].

(3)将人脸图像训练集按照9:1的比例进行划分，分为训练数据和验证数据。(3) The face image training set is divided into training data and verification data according to the ratio of 9:1.

S6.构建多任务深度卷积神经网络，把人脸图像训练集中每幅人脸图像减去平均人脸图像后进行网络参数的训练，得到卷积神经网络模型，具体包括：S6. Build a multi-task deep convolutional neural network, subtract the average face image from each face image in the face image training set, and perform network parameter training to obtain a convolutional neural network model, specifically including:

(1)将训练数据随机打乱，设置每批大小为m＝128个样本，进行数据划分。在训练深度卷积神经网络模型的过程中，利用批量梯度更新方法进行学习神经网络的权重参数。(1) Shuffle the training data randomly, set the size of each batch as m=128 samples, and divide the data. In the process of training the deep convolutional neural network model, the weight parameters of the neural network are learned by using the batch gradient update method.

(2)设计卷积神经网络的结构，其包含卷积层、下采样层和全连接层。每个卷积层和全连接层采用非线性矫正激活函数。在多任务深度学习的人脸属性识别中，整体网络结构分为共享层和独有层两个部分。共享层为所有的任务所共享，多个任务共同参与共享层参数的训练。独有层则是每个任务单独占用，并用每个任务单独的数据进行参数学习。假设共享层的个数为S＝10；独有层的个数为U＝2。所采用的损失函数均为Softmax函数，α_k均设置为1。(2) Design the structure of convolutional neural network, which includes convolutional layer, downsampling layer and fully connected layer. Each convolutional layer and fully connected layer uses a nonlinear rectified activation function. In the face attribute recognition of multi-task deep learning, the overall network structure is divided into two parts: shared layer and unique layer. The shared layer is shared by all tasks, and multiple tasks participate in the training of shared layer parameters. The exclusive layer is occupied by each task separately, and uses the separate data of each task for parameter learning. Assume that the number of shared layers is S=10; the number of unique layers is U=2. The loss functions used are all Softmax functions, and α_k is set to 1.

(3)设置卷积神经网络结构中所需要的卷积层滤波器数目和特征图数目、滤波器大小、下采样层中核的大小、每一层的学习率、权重初始值等超参数，深度卷积神经网络的网络结构如表1所示。(3) Set the number of convolutional layer filters and the number of feature maps required in the convolutional neural network structure, the size of the filter, the size of the kernel in the downsampling layer, the learning rate of each layer, the initial value of the weight and other hyperparameters, depth The network structure of the convolutional neural network is shown in Table 1.

表1Table 1

网络层名network layer name类型Types of输入大小input size输出大小output size滤波器大小/步长Filter size/step sizeConv1_1Conv1_1卷积层convolutional layer128*128*3128*128*3128*128*32128*128*323*3/13*3/1Conv1_2Conv1_2卷积层convolutional layer128*128*32128*128*32128*128*64128*128*643*3/13*3/1Pool1Pool1下采样层downsampling layer128*128*64128*128*6464*64*6464*64*642*2/22*2/2Conv2_1Conv2_1卷积层convolutional layer64*64*6464*64*6464*64*9664*64*963*3/13*3/1Conv2_2Conv2_2卷积层convolutional layer64*64*9664*64*9664*64*12864*64*1283*3/13*3/1Pool2Pool2下采样层downsampling layer64*64*12864*64*12832*32*12832*32*1282*2/22*2/2Conv3_1Conv3_1卷积层convolutional layer32*32*12832*32*12832*32*12832*32*1283*3/13*3/1Conv3_2Conv3_2卷积层convolutional layer32*32*12832*32*12832*32*19232*32*1923*3/13*3/1Pool3Pool3下采样层downsampling layer32*32*19232*32*19216*16*19216*16*1922*2/22*2/2Conv4_1Conv4_1卷积层convolutional layer16*16*19216*16*19216*16*25616*16*2563*3/13*3/1Conv4_2Conv4_2卷积层convolutional layer16*16*25616*16*25616*16*25616*16*2563*3/13*3/1Pool4Pool4下采样层downsampling layer16*16*25616*16*2568*8*2568*8*2562*2/22*2/2Conv5_1Conv5_1卷积层convolutional layer8*8*2568*8*2568*8*3208*8*3203*3/13*3/1Conv5_2Conv5_2卷积层convolutional layer8*8*3208*8*3208*8*3208*8*3203*3/13*3/1Pool5Pool5下采样层downsampling layer8*8*3208*8*3204*4*3204*4*3202*2/22*2/2Dropout1Dropout1Dropout层Dropout layer4*4*3204*4*3204*4*3204*4*320Fc1Fc1全连接层fully connected layer4*4*3204*4*3201*1*1281*1*128Fc2Fc2全连接层fully connected layer1*1*1281*1*1281*1*C_i1*1*C_i

(4)在卷积神经网络训练的过程中，采用冲量和丢弃等训练技巧，用于加速卷积神经网络的训练。(4) In the process of convolutional neural network training, training techniques such as impulse and discarding are used to accelerate the training of convolutional neural network.

(5)对于训练好的网络模型参数，根据其在验证数据上的性能，判断是否停止训练。(5) For the trained network model parameters, judge whether to stop the training according to its performance on the verification data.

(6)提取出训练好的网络模型参数W。(6) Extract the trained network model parameter W.

S7.对于任意给定的一张待人脸属性识别的图像，使用步骤S1～S4中一样的数据预处理方法，将待测试图像变为一个128×128大小的彩色图像，并减去训练数据中的均值图像，输入到训练好的深度卷积神经网络中，最终获得3个不同的人脸属性的识别结果。S7. For any given image to be recognized by face attributes, use the same data preprocessing method in steps S1 to S4 to change the image to be tested into a 128×128 color image, and subtract the training data The mean image is input into the trained deep convolutional neural network, and finally the recognition results of 3 different face attributes are obtained.

S8.将CelebrayA测试数据的每个人脸图像，进行步骤S7操作，进行属性识别精度和预测时间比较。单任务网络和多任务网络在人脸CelebrayA测试数据上的精度及其时间对比结果如表2所示，从表中可以看出在相同网络结构下，多任务深度学习属性识别方法可以提高属性识别的准确度，同时，利用多任务属性识别可以大大减少每个任务的平均预测时间。S8. Perform the operation of step S7 on each face image of the CelebrayA test data to compare attribute recognition accuracy and prediction time. The accuracy and time comparison results of the single-task network and multi-task network on the face CelebrayA test data are shown in Table 2. From the table, it can be seen that under the same network structure, the multi-task deep learning attribute recognition method can improve attribute recognition. accuracy, meanwhile, utilizing multi-task attribute recognition can greatly reduce the average prediction time per task.

表2Table 2

方法methodT1T1T2T2T3T3T1+T2T1+T2T1+T3T1+T3T2+T3T2+T3T1+T2+T3T1+T2+T3人脸性别(T1)Face gender (T1)0.96700.9670N/AN/AN/AN/A0.95800.95800.96900.9690N/AN/A0.96800.9680人脸微笑(T2)Smile (T2)N/AN/A0.92300.9230N/AN/A0.92100.9210N/AN/A0.93500.93500.93600.9360人脸吸引力(T3)Facial Attraction (T3)N/AN/AN/AN/A0.80600.8060N/AN/A0.81300.81300.81600.81600.82200.8220总预测时间(秒)Total prediction time (seconds)0.02300.02300.02300.02300.02300.02300.02700.02700.02700.02700.02700.02700.02900.0290平均预测时间(秒)Average prediction time (seconds)0.02300.02300.02300.02300.02300.02300.01350.01350.01350.01350.01350.01350.00970.0097

本发明针对于不同的人脸属性识别任务，可以共同的学习共享的网络权重，且独立训练独有的网络权重，大大缩小了整体的训练数据的要求。本发明有效地提高了人脸属性识别的性能。The present invention can jointly learn shared network weights for different human face attribute recognition tasks, and independently train unique network weights, greatly reducing the overall training data requirements. The invention effectively improves the performance of human face attribute recognition.

Claims (9)

1. The face attribute recognition method based on the multitask deep learning is characterized by comprising the following steps of:

A. preparing an image data set comprising a plurality of faces and corresponding face attribute labels;

B. carrying out face detection on each image in the image data set one by one to obtain the position of a face in each image;

C. performing face key point detection on all detected faces to acquire the position of the face key point in each image;

D. aligning each face to a standard face image according to a face alignment method for the detected face key points to form a face image training set;

E. calculating an average face image in a training set by using a face image training set;

F. constructing a multitask deep convolutional neural network, and carrying out network parameter training after subtracting an average face image from each face image in a face image training set to obtain a convolutional neural network model;

G. respectively carrying out face detection and face key point detection on a test image to be recognized, and aligning a face in the image to a standard face image according to the face key point;

H. and subtracting the average face image from the standard face image, and putting the standard face image into the constructed convolutional neural network model for feedforward operation to obtain a plurality of attribute recognition results of the face.

2. The method for identifying human face attributes based on multitask deep learning as claimed in claim 1, characterized in that in step A, said prepared image data set adopts image data containing human faces collected under the scene, and provides corresponding K personal face attribute labels.

3. The method according to claim 1, wherein in step B, the face detection is performed on each image in the image data set one by using a common face detection method to obtain the position of the face in each image.

4. The method for identifying the human face attribute based on the multitask deep learning as claimed in claim 3, wherein the commonly used human face detection method adopts an OpenCV self-contained human face detection method.

5. The method for identifying human face attributes based on multitask deep learning as claimed in claim 1, wherein in step C, said human face key point detection adopts a commonly used human face key point detection method to obtain the position of the human face key point in each image.

6. The method as claimed in claim 5, wherein the commonly used face key point detection method is a Dlib-carried face key point detection method.

7. The method for identifying the attributes of the human face based on the multitask deep learning as claimed in claim 1, wherein in the step D, the human face alignment method is affine transformation based on a two-dimensional image, and specifically comprises the following steps:

D1. fitting by using a least square method according to the matching relation between the face key points and the standard face key points to obtain an optimal transformation matrix; assuming d coordinates of the key points of the standard face asWhereinThe coordinates of the ith standard face key point are obtained; d is the number of key points of the face and d is a natural number; and the coordinates of the key points of the detected face are { (x)₁；y₁),(x₂；y₂),...,(x_d；y_d) In which (x)_i,y_i) Is the ith individual face key point coordinate of the detected face and extends the coordinate to src { (x)₁；y₁；1),(x₂；y₂；1),...,(x_d；y_d(ii) a 1) Optimizing by a least square method; the specific calculation formula is as follows:

$A_0=\arg \underset{A}{min}\left(\right||A\&CenterDot;src-dst|{|}_2^2)$

whereinRepresenting an affine transformation matrix; a. the₀Is an optimal transformation matrix;

D2. transformation matrix A obtained by optimization₀And aligning all the face images to the standard face images, and cutting the face images into images with uniform sizes.

8. The method for identifying human face attributes based on multitask deep learning as claimed in claim 1, wherein in step E, the specific method for calculating the average human face image in the training set for the human face image training set is as follows:

E1. the mean image is each channel of a face image training set, comprises 3 channels of RGB, and is an arithmetic mean image M obtained through calculation, wherein the calculation mode of each channel and M is as follows:

$M_R=\frac{\underset{n=1}{\overset{N}{\&Sigma;}}{X}_n^R}{N},M_G=\frac{\underset{n=1}{\overset{N}{\&Sigma;}}{X}_n^G}{N},M_B=\frac{\underset{n=1}{\overset{N}{\&Sigma;}}{X}_n^B}{N},$

wherein,andthe R channel, the G channel and the B channel of the nth human face image are respectively arranged; n is the total image number of the face image training set, and N is a natural number; m_R，M_GAnd M_BArithmetic mean images of an R channel, a G channel and a B channel respectively;

E2. the arithmetic mean value image obtained by the RGB three channels is combined into an average face image, and the calculation formula is as follows:

M＝[M_R,M_G,M_B]；

E3. dividing the face image training set into training data and verification data according to the ratio of 9: 1.

9. The method for identifying the human face attribute based on the multitask deep learning as claimed in claim 1, wherein in the step F, the specific method for constructing the multitask deep convolutional neural network is as follows:

F1. randomly disordering training data, setting the size of each batch of samples to be m, and dividing the data, wherein m is a natural number; in the process of training the deep convolutional neural network model, learning the weight parameters of the neural network by using a batch gradient updating method;

F2. designing a structure of a convolutional neural network, wherein the structure comprises a convolutional layer, a downsampling layer and a full-connection layer; each convolution layer and the full-connection layer adopt nonlinear rectification activation functions; in the face attribute recognition of multi-task deep learning, the whole network structure is divided into a sharing layer and a unique layer, the sharing layer is shared by all tasks, and a plurality of tasks participate in the training of parameters of the sharing layer together; the unique layer is occupied by each task independently, parameter learning is carried out by using independent data of each task, and the number of the shared layers is assumed to be S, wherein S is a natural number; the number of the unique layers is U, wherein U is a natural number; setting different numbers of S and U according to different multi-task combinations;

F3. setting the number of convolutional layer filters and the number of characteristic graphs required in a convolutional neural network structure, the size of the filters, the size of kernels in downsampling layers, the learning rate of each layer and a weight initial value;

F4. in the training process of the convolutional neural network, training skills such as impulse and abandon are adopted for accelerating the training of the convolutional neural network;

F5. judging whether to stop training according to the performance of the trained network model parameters on the verification data;

F6. and extracting the trained network model parameters W.