技术领域Technical Field
本发明属于计算机视觉领域，特别涉及一种应用深度学习实现人脸聚类与识别的方法。The present invention belongs to the field of computer vision, and in particular relates to a method for realizing face clustering and recognition by applying deep learning.
背景技术Background Art
人脸聚类是挖掘未标注人脸数据的重要工具，在人脸标注和检索等领域有着广泛的应用。随着面部数据量的不断增多，使得继续扩大人脸数据集的规模变得越来越困难，即使是人工手工标注，也难免会引入噪声。面部聚类能够大大减少了数据标注的工作量，实现自动化，因此在生活中具有广泛的应用。面部聚类的一般过程如下：首先通过机器学习或者深度学习的方法对面部数据提取特征，然后通过对提取后的特征以及关系对面部数据进行分组。聚类的目标是对同一类的图像给定同一种标签，不同类的图像给定不同的标签。虽然现在许多聚类的技术和产品在面部聚类上的效果已经很不错了，但是如何在大规模数据上对面部数据进行更快更准确的聚类这个问题仍未解决。Face clustering is an important tool for mining unlabeled face data, and has a wide range of applications in face annotation and retrieval. With the increasing amount of facial data, it is becoming increasingly difficult to continue to expand the size of face datasets. Even manual annotation will inevitably introduce noise. Face clustering can greatly reduce the workload of data annotation and achieve automation, so it has a wide range of applications in life. The general process of face clustering is as follows: first, extract features from facial data through machine learning or deep learning methods, and then group facial data based on the extracted features and relationships. The goal of clustering is to give the same label to images of the same class, and different labels to images of different classes. Although many clustering technologies and products have achieved good results in face clustering, the problem of how to cluster facial data faster and more accurately on large-scale data remains unsolved.
现有的人脸聚类方法大致可以分为两类，即无监督方法和有监督方法。无监督方法，比如K-Means(“StuartLloyd.Leastsquaresquantizationinpcm.IEEEtransactions oninformation theory,28(2):129–137,1982.1,6”)和DBSCAN(“Martin Ester,Hans-PeterKriegel,J¨org Sander,Xiaowei Xu,et al.A density-based algorithm fordiscovering clusters in large spatial databases with noise.In KDD,1996.1,2,6,7”)等方法依赖于特定的假设，导致其缺乏处理真实数据集的复杂聚类结构的能力。为了提高对不同数据的适应性，在“Zhongdao Wang,Liang Zheng,Yali Li,and ShengjinWang.Linkage based face clustering via graph convolution network.InProceedings of the IEEE Conference on Computer Vision and PatternRecognition,pages 1117–1125,2019.1,2,4,5,6”、“Lei Yang,Xiaohang Zhan,DapengChen,Junjie Yan,Chen Change Loy,and Dahua Lin.Learning to cluster faces on anaf finitygraph.In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition,pages 2298–2306,2019.1,2,5,6,7”提出了监督聚类方法来学习聚类模式。然而，这几种方法无论是准确性还是效率都不能令人满意。同时，为了能够完成大规模人脸数据聚类这项任务，现有方法普遍是生成许多子图来数据，这样就会导致两个主要问题。首先，处理子图包括基于简单假设的启发式步骤。子图生成和预测聚合都依赖于启发式过程，因此限制了它们的性能上限。此外，这些方法所需的子图通常高度重叠，导致过多的冗余计算成本。Existing face clustering methods can be roughly divided into two categories, namely unsupervised methods and supervised methods. Unsupervised methods, such as K-Means ("Stuart Lloyd. Least squares quantization in PCM. IEEE transactions on information theory, 28 (2): 129–137, 1982. 1, 6") and DBSCAN ("Martin Ester, Hans-Peter Kriegel, J¨org Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, 1996. 1, 2, 6, 7"), rely on specific assumptions, resulting in their lack of ability to handle the complex clustering structure of real data sets. In order to improve the adaptability to different data, supervised clustering methods are proposed to learn clustering patterns in “Zhongdao Wang, Liang Zheng, Yali Li, and Shengjin Wang. Linkage based face clustering via graph convolution network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1117–1125, 2019. 1, 2, 4, 5, 6” and “Lei Yang, Xiaohang Zhan, Dapeng Chen, Junjie Yan, Chen Change Loy, and Dahua Lin. Learning to cluster faces on anaffinitygraph. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2298–2306, 2019. 1, 2, 5, 6, 7”. However, these methods are not satisfactory in terms of accuracy or efficiency. At the same time, in order to complete the task of clustering large-scale face data, existing methods generally generate many subgraphs to learn data, which will lead to two major problems. First, processing subgraphs involves heuristic steps based on simple assumptions. Both subgraph generation and predicted aggregation rely on heuristic procedures, thus limiting their upper performance limits. In addition, the subgraphs required by these methods are usually highly overlapping, resulting in excessive redundant computational costs.
因此，在对人脸图像的处理过程中，我们寻求一种能够更准确有效地学习聚类的算法。为了更高的准确性，我们希望结构中的所有的参数都可以学习，通过改变图中的链接值对参数进行更好的学习。另一方面，为了减少冗余计算，我们打算减少所需子图的数量。Therefore, in the process of processing face images, we seek an algorithm that can learn clustering more accurately and effectively. For higher accuracy, we hope that all parameters in the structure can be learned, and the parameters can be better learned by changing the link values in the graph. On the other hand, in order to reduce redundant calculations, we intend to reduce the number of required subgraphs.
发明内容Summary of the invention
本发明的目的是提供一种基于链接预测的人脸聚类方法，以提高聚类的速度和准确率。The purpose of the present invention is to provide a face clustering method based on link prediction to improve the speed and accuracy of clustering.
为实现上述目的，本发明采用如下技术方案：To achieve the above object, the present invention adopts the following technical solution:
一种基于链接预测的人脸聚类方法，包括如下步骤：A face clustering method based on link prediction comprises the following steps:
步骤一，选取人脸图像数据库，对数据库中的人脸图像进行预处理，然后提取单个人脸图像的特征，并得到单个人脸图像的近邻关系列表；Step 1: Select a face image database, pre-process the face images in the database, then extract the features of a single face image, and obtain a neighbor relationship list of the single face image;
步骤二，构建人脸关系图，根据步骤一得到的单个人脸图像的近邻关系列表以及特征对于每一个人脸构建局部子图，得到人脸关系局部子图集合；Step 2: construct a face relationship graph, and construct a local subgraph for each face according to the neighbor relationship list and features of the single face image obtained in step 1, to obtain a face relationship local subgraph set;
步骤三，链接预测，通过步骤二得到的人脸关系局部子图集合，输入到图卷积网络中进行预测，并通过线性层对节点之间的关系进行预测，得到更改后的近邻列表，通过不断地更新迭代，最后判断该节点是否与中心存在连接，来达到将节点划分的目的，并得到节点的聚类结果；Step 3: Link prediction: The face relationship local subgraph set obtained in step 2 is input into the graph convolutional network for prediction, and the relationship between nodes is predicted through the linear layer to obtain the modified neighbor list. Through continuous updating and iteration, it is finally determined whether the node is connected to the center to achieve the purpose of node division and obtain the node clustering result.
步骤四，根据已经得到的聚类结果，选择其中存在孤立点的簇进行合并调整，得到最终的聚类结果。Step 4: Based on the obtained clustering results, select the clusters with isolated points for merging and adjustment to obtain the final clustering results.
所述步骤一中，对数据库中的人脸图像进行预处理是使用SeetaFace人脸识别引擎从数据库中的图片提取人脸图像并矫正。In the step 1, the face images in the database are preprocessed by using the SeetaFace face recognition engine to extract and correct the face images from the pictures in the database.
所述步骤一中，通过深度卷积神经网络提取人脸图像的面部特征Fn＝{fi|fi＝g(xi),xi∈X}，其中，n表示提取面部特征的维度，g(·)表示将特征提取出来的CNN网络，X表示人脸图像集，xi表示第i个人脸图像；使用已经提取好的人脸图像的面部特征Fn，计算相似度，最终得到每一张人脸图像的k-近邻关系列表，将C(·)定义为人脸图像集X中一张人脸图像的近邻列表C，则人脸图像xi的近邻列表表示为其中，k表示当前人脸图像xi的最大近邻的数量。In the step 1, facial featuresFn = {fi |fi = g(xi ),xi ∈X} of the face image are extracted by a deep convolutional neural network, wherein n represents the dimension of the extracted facial features, g(·) represents the CNN network that extracts the features, X represents a face image set, andxi represents the i-th face image; the facial featuresFn of the extracted face image are used to calculate the similarity, and finally a k-nearest neighbor relationship list of each face image is obtained, and C(·) is defined as a neighbor list C of a face image in the face image set X, then the neighbor list of the face imagexi is expressed as Where k represents the number of the maximum neighbors of the current face imagexi .
所述步骤二中，通过每张人脸图像的邻近关系列表C(xi)，将人脸数据连接成一个图G，从图G中构建人脸关系图，构建关系图的步骤包括：In the step 2, the face data are connected into a graph G through the neighbor relationship list C(xi ) of each face image, and a face relationship graph is constructed from the graph G. The step of constructing the relationship graph includes:
(1)构建中心节点：首先定义积极节点数μ，它表示近邻关系列表中的前μ个节点，然后对于一个节点xi，如果它已经被访问过了，则继续寻找下一个节点的中心子图，如果还没被访问，那么将xi的近邻列表C(xi)中的μ个积极节点选择出来，并判断这个几个节点之间能否存在一个完全图的关系，如果存在，那么这个图就组成了一个中心子图如果不存在，那么这一个节点就自己构成一个中心子图重复以上过程在图G中搜索每个节点xi组成的完全图最后得到中心子图的集合(1) Constructing the central node: First, define the number of active nodes μ, which represents the first μ nodes in the neighbor relationship list. Then, for a node xi , if it has been visited, continue to look for the central subgraph of the next node. If it has not been visited, then select μ active nodes in the neighbor list C(xi ) of xi and determine whether there is a complete graph relationship between these nodes. If so, then this graph constitutes a central subgraph. If it does not exist, then this node itself constitutes a central subgraph Repeat the above process to search for the complete graph composed of each nodexi in graph G Finally, we get the set of central subgraphs
(2)加入其他近邻的节点：通过已经得到的中心子图集合Gc，将每一个中心子图作为子图的中心，然后从中节点的近邻关系列表中挑选一部分节点来构建人脸关系图；根据节点在近邻列表中的顺序以及特征之间差异对几个近邻列表中的节点进行筛选，选择一部分节点用于生成以为中心的子图(2) Add other neighboring nodes: Use the obtained central subgraph set Gc to add each central subgraph as the center of the subgraph, and then from Select some nodes from the neighbor relationship list of the nodes in the face relationship graph to build the face relationship graph; filter the nodes in several neighbor lists according to the order of the nodes in the neighbor list and the differences between the features, and select some nodes to generate the following The subgraph centered
首先将中心节点子图中的每一个节点列表进行合并，我们使用p表示当前子图中的节点数量，对于子图中的每一个节点这样就得到一个新的近邻列表其中节点的数量就是k·p；对于其中每一个节点计算与当前子图之间的距离定义如下：First, the central node subgraph We use p to represent the current subgraph. The number of nodes in the subgraph Each node in this way gets a new neighbor list The number of nodes is k·p; for each node, calculate the The distance between The definition is as follows:
其中，p表示有中心子图中节点的数量，j表示当前子图是中心子图集合Gc中的第几个子图，k表示在子图中每一个人脸图像xi的原近邻列表C(xi)中最大近邻数量；v表示中心子图中的一个节点，表示在新的近邻列表中C(xi)p第m个节点，表示v和这两个点在原近邻列表中是否存在链接，如果存在，等于1，如果不存在，等于0，fv和表示节点特征，表示fv的转置，最后计算两个向量之间的内积；Among them, p represents the number of nodes in the central subgraph, and j represents the current subgraph is the number of subgraphs in the central subgraph set Gc , k represents the number of subgraphs in the subgraph The maximum number of neighbors in the original neighbor list C(xi ) of each face image xiin ; v represents the central subgraph A node in represents the mth node in the new neighbor list C(xi )p , represents v and Are there links between these two points in the original neighbor list? If so, Equal to 1, if not present, equals 0, fv and Represents node characteristics, represents the transpose of fv , and finally calculates the inner product between the two vectors;
通过对得到的距离进行排序，得到新的近邻关系列表，通过将C(xi)的链接关系加入到中心子图以及筛选后的节点中，得到最后的子图最后重复上述步骤，得到人脸关系局部子图集合Gs。By sorting the obtained distances, a new neighbor relationship list is obtained, and by adding the link relationship of C(xi ) to the central subgraph and the filtered nodes, the final subgraph is obtained. Finally, repeat the above steps to obtain the face relationship local subgraph setGs .
所述步骤三具体为：首先根据生成的人脸关系局部子图集合Gs，将其中每一个子图的结构表示成邻接矩阵的形式，记作Aj，同时也要将子图中的节点与中心子图是否存在链接表示成一个0-1的向量，记作qj，然后将该子图中存在的节点特征与邻接矩阵Aj输入图卷积网络中，形式如下：The step three is specifically as follows: first, according to the generated face relationship local sub-graph set Gs , each of the sub-graphs The structure of is expressed as an adjacency matrix, denoted as Aj , and the subgraph The nodes and central subgraphs in Whether there is a link is represented as a 0-1 vector, denoted as qj , and then the subgraph Node features present in The adjacency matrixAj is input into the graph convolutional network in the following form:
其中，σ(·)表示卷积操作，W表示要学习的权重，表示第t次和第t+1次迭代得到的特征；Among them, σ(·) represents the convolution operation, W represents the weight to be learned, Represents the features obtained at the tth and t+1th iterations;
然后根据迭代得到的特征使用映射函数g(·)对这个特征与中心子图之间是否存在链接进行预测，形式如下：Then according to the features obtained by iteration Use the mapping function g(·) to map this feature to the central subgraph Whether there is a link between them is predicted in the following form:
其中，映射函数g(·)是由几个线性层叠加而成，为了学习特征中的隐藏属性，最后将特征进行分类预测，一部分是与中心子图之间存在链接，一部分则是不存在链接，最后得到就是表示是否存在链接的0-1向量；通过对来完成对邻接矩阵Aj在迭代过程中的更新，更新方式如下：The mapping function g(·) is composed of several linear layers. In order to learn the hidden attributes in the features, the features are finally classified and predicted. Part of it is related to the central subgraph. There are links between them, and some do not have links. It is a 0-1 vector indicating whether there is a link; To complete the update of the adjacency matrix Aj during the iteration process, the update method is as follows:
其中，是由得到的预测邻接矩阵，α是计算因子，用来计算之间的差距，计算方式如下：in, Is The predicted adjacency matrix obtained, α is the calculation factor, used to calculate The difference between them is calculated as follows:
其中，qj是一个表示子图中的节点与中心子图是否存在链接的0-1向量，这样通过不断的更新得到一个相对准确的邻接矩阵之后将更新得到的邻接矩阵和新的输入进行下一次迭代。Among them,qj is a subgraph The nodes and central subgraphs in Is there a linked 0-1 vector, so that a relatively accurate adjacency matrix can be obtained through continuous updating? The adjacency matrix is then updated and new Enter for the next iteration.
所述步骤四中，通过寻找孤立点组成簇，计算它与每个簇之间的距离，选择距离最近的簇进行合并，得到最终的聚类结果。In the step 4, isolated points are found to form clusters, the distance between them and each cluster is calculated, and the clusters with the closest distance are selected for merging to obtain the final clustering result.
有益效果：与一些传统的人脸聚类方式不同的是，本发明在两方面提出了创新。一方面，通过将三角约束进行扩展得到多个定点之间的完全图关系，减少了预测过程中要输入较多的子图。另一方面，本发明将通过预测对图卷积中邻接矩阵进行迭代修改，以此在训练过程中得到更准确的预测结果。Beneficial effects: Different from some traditional face clustering methods, the present invention proposes innovations in two aspects. On the one hand, by expanding the triangular constraints to obtain the complete graph relationship between multiple fixed points, the number of subgraphs to be input in the prediction process is reduced. On the other hand, the present invention iteratively modifies the adjacency matrix in the graph convolution through prediction, so as to obtain more accurate prediction results during the training process.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1为一个图中的三个节点Pi、Pj、Pk存在的四种情况的示意图。FIG1 is a schematic diagram of four situations in which three nodes Pi , Pj , and Pk exist in a graph.
具体实施方式DETAILED DESCRIPTION
下面结合附图对本发明做更进一步的解释。The present invention will be further explained below in conjunction with the accompanying drawings.
本发明的一种基于链接预测的人脸聚类方法，包括以下步骤：A face clustering method based on link prediction of the present invention comprises the following steps:
步骤一，选取人脸图像数据库，对数据库中的人脸图像进行预处理是使用SeetaFace人脸识别引擎从数据库中的图片提取人脸图像并矫正，然后提取单个人脸图像的特征，并得到单个人脸图像的近邻关系列表；具体为：Step 1: Select a face image database and pre-process the face images in the database by using the SeetaFace face recognition engine to extract face images from the pictures in the database and correct them, then extract the features of a single face image and obtain a neighbor relationship list of a single face image; specifically:
通过深度卷积神经网络提取人脸图像的面部特征Fn＝{fi|fi＝g(xi),xi∈X}，其中，n表示提取面部特征的维度，g(·)表示将特征提取出来的CNN网络，X表示人脸图像集，xi表示第i个人脸图像；使用已经提取好的人脸图像的面部特征Fn，计算相似度，最终得到每一张人脸图像的k-近邻关系列表，将C(·)定义为人脸图像集X中一张人脸图像的近邻列表C，则人脸图像xi的近邻列表表示为其中，k表示当前人脸图像xi的最大近邻的数量。The facial features of the face image are extracted by a deep convolutional neural networkFn = {fi |fi = g(xi ),xi∈X }, where n represents the dimension of the extracted facial features, g(·) represents the CNN network that extracts the features, X represents the face image set, andxi represents the i-th face image; the facial featuresFn of the extracted face image are used to calculate the similarity, and finally the k-nearest neighbor relationship list of each face image is obtained. C(·) is defined as the neighbor list C of a face image in the face image set X, then the neighbor list of face imagexi is expressed as Where k represents the number of the maximum neighbors of the current face imagexi .
步骤二，构建人脸关系图，根据步骤一得到的单个人脸图像的近邻关系列表以及特征对于每一个人脸构建局部子图，得到人脸关系局部子图集合；Step 2: construct a face relationship graph, and construct a local subgraph for each face according to the neighbor relationship list and features of the single face image obtained in step 1, to obtain a face relationship local subgraph set;
通过每张人脸图像的邻近关系列表C(xi)，就已经可以将人脸数据连接成一个图G，但是由于图G包含的节点过多，不能够直接进行处理，因此，要从图G中构建人脸关系图，构建关系图的步骤主要分为以下两个步骤：Through the neighbor relationship list C(xi ) of each face image, the face data can be connected into a graph G. However, since the graph G contains too many nodes, it cannot be processed directly. Therefore, to construct a face relationship graph from the graph G, the steps of constructing the relationship graph are mainly divided into the following two steps:
(1)构建中心节点：(1) Building a central node:
先对一个图中的三个节点Pi、Pj、Pk进行分析，一共存在四种情况，如图1所示；First, we analyze three nodesPi ,Pj , andPk in a graph. There are four situations in total, as shown in Figure 1;
根据图1分析得到，分析图1中(a)可以看出，如果Pi、Pj、Pk三个点之间都存在相似关系，那么它们能组成一个三角形，分析图1中(b)可以看出，如果Pi、Pj、Pk只有两个点之间存在链接，也就是Pi与Pj、Pj与Pk相似，但是Pi与Pk不相似，这样的关系就不是很合理；而图1中(c)和图1中(d)中表明了另外的两种关系。那么，把问题扩展到m个点之间的关系时，如果这m个点之间均存在相似关系，就能够得到一个有m个点组成的完全图。According to the analysis of Figure 1, it can be seen from Figure 1 (a) that if there is a similarity relationship between the three pointsPi ,Pj , andPk , then they can form a triangle. It can be seen from Figure 1 (b) that if there is only a link between two pointsPi ,Pj , andPk , that is,Pi andPj ,Pj andPk are similar, butPi andPk are not similar, such a relationship is not very reasonable; and Figure 1 (c) and Figure 1 (d) show two other relationships. Then, when the problem is extended to the relationship between m points, if there is a similarity relationship between these m points, a complete graph composed of m points can be obtained.
基于上述分析，首先定义积极节点数μ，它表示近邻关系列表中的前μ个节点，然后对于一个节点xi，如果它已经被访问过了，则继续寻找下一个节点的中心子图，如果还没被访问，那么将xi的近邻列表C(xi)中的μ个积极节点选择出来，并判断这几个节点之间能否存在一个完全图的关系，如果存在，那么这个图就组成了一个中心子图如果不存在，那么这一个节点就自己构成一个中心子图重复以上过程在图G中搜索每个节点xi组成的完全图最后得到中心子图的集合这样就将节点重新整理，可以减少了一部分需要计算的节点数量。Based on the above analysis, we first define the number of active nodes μ, which represents the first μ nodes in the neighbor relationship list. Then, for a node xi , if it has been visited, we continue to look for the central subgraph of the next node. If it has not been visited, we select μ active nodes in the neighbor list C(xi ) of xi and determine whether there is a complete graph relationship between these nodes. If so, then this graph constitutes a central subgraph. If it does not exist, then this node itself constitutes a central subgraph Repeat the above process to search for the complete graph composed of each nodexi in graph G Finally, we get the set of central subgraphs This will reorganize the nodes and reduce the number of nodes that need to be calculated.
(2)加入其他近邻的节点：(2) Add other neighboring nodes:
通过已经得到的中心子图集合Gc，将每一个中心子图作为子图的中心，然后从中节点的近邻关系列表中挑选一部分节点来构建人脸关系图；根据节点在近邻列表中的顺序以及特征之间差异对几个近邻列表中的节点进行筛选，选择一部分节点用于生成以为中心的子图Through the obtained central subgraph set Gc , each central subgraph as the center of the subgraph, and then from Select some nodes from the neighbor relationship list of the nodes in the face relationship graph to build the face relationship graph; filter the nodes in several neighbor lists according to the order of the nodes in the neighbor list and the differences between the features, and select some nodes to generate the following The subgraph centered
首先将中心节点子图中的每一个节点列表进行合并，我们使用p表示当前子图中的节点数量，对于子图中的每一个节点这样就得到一个新的近邻列表其中节点的数量就是k·p；对于其中每一个节点计算与当前子图之间的距离定义如下：First, the central node subgraph We use p to represent the current subgraph. The number of nodes in the subgraph Each node in this way gets a new neighbor list The number of nodes is k·p; for each node, calculate the The distance between The definition is as follows:
其中，p表示有中心子图中节点的数量，j表示当前子图是中心子图集合Gc中的第几个子图，k表示在子图中每一个人脸图像xi的原近邻列表C(xi)中最大近邻数量；v表示中心子图中的一个节点，表示在新的近邻列表中C(xi)p第m个节点，表示v和这两个点在原近邻列表中是否存在链接，如果存在，等于1，如果不存在，等于0，fv和表示节点特征，表示fv的转置，最后计算两个向量之间的内积；Among them, p represents the number of nodes in the central subgraph, and j represents the current subgraph is the number of subgraphs in the central subgraph set Gc , k represents the number of subgraphs in the subgraph The maximum number of neighbors in the original neighbor list C(xi ) of each face image xiin ; v represents the central subgraph A node in represents the mth node in the new neighbor list C(xi )p , represents v and Are there links between these two points in the original neighbor list? If so, Equal to 1, if not present, equals 0, fv and Represents node characteristics, represents the transpose of fv , and finally calculates the inner product between the two vectors;
通过对得到的距离进行排序，得到新的近邻关系列表，通过将C(xi)的链接关系加入到中心子图以及筛选后的节点中，得到最后的子图最后重复上述步骤，得到人脸关系局部子图集合Gs。By sorting the obtained distances, a new neighbor relationship list is obtained, and by adding the link relationship of C(xi ) to the central subgraph and the filtered nodes, the final subgraph is obtained. Finally, repeat the above steps to obtain the face relationship local subgraph setGs .
步骤三，链接预测，通过步骤二得到的人脸关系局部子图集合Gs，输入到图卷积网络中进行预测，并通过线性层对节点之间的关系进行预测，得到更改后的近邻列表，通过不断地更新迭代，最后判断该节点是否与中心存在连接，来达到将节点划分的目的，并得到节点的聚类结果；Step 3: Link prediction: The face relationship local subgraph setGs obtained in step 2 is input into the graph convolutional network for prediction, and the relationship between nodes is predicted through the linear layer to obtain the modified neighbor list. Through continuous updating and iteration, it is finally determined whether the node is connected to the center to achieve the purpose of node division and obtain the node clustering result.
由于过多层数的卷积可能会导致过拟合的问题，本发明选择使用四层图卷积网络作为模型的基础。具体过程如下：Since too many layers of convolution may lead to overfitting problems, this paper chooses to use a four-layer graph convolutional network as the basis of the model. The specific process is as follows:
首先根据生成的人脸关系局部子图集合Gs，将其中每一个子图的结构表示成邻接矩阵的形式，记作Aj，同时也要将子图中的节点与中心子图是否存在链接表示成一个0-1的向量，记作qj，然后将该子图中存在的节点特征与邻接矩阵Aj输入图卷积网络中，形式如下：First, according to the generated face relationship local subgraph set Gs , each of the subgraphs The structure of is expressed as an adjacency matrix, denoted as Aj , and the subgraph The nodes and central subgraphs in Whether there is a link is represented as a 0-1 vector, denoted as qj , and then the subgraph Node features present in The adjacency matrixAj is input into the graph convolutional network in the following form:
其中，σ(·)表示卷积操作，W表示要学习的权重，表示第t次和第t+1次迭代得到的特征；Among them, σ(·) represents the convolution operation, W represents the weight to be learned, Represents the features obtained at the tth and t+1th iterations;
然后根据迭代得到的特征使用映射函数g(·)对这个特征与中心子图之间是否存在链接进行预测，形式如下：Then according to the features obtained by iteration Use the mapping function g(·) to map this feature to the central subgraph Whether there is a link between them is predicted in the following form:
其中，映射函数g(·)是由几个线性层叠加而成，为了学习特征中的隐藏属性，最后将特征进行分类预测，一部分是与中心子图之间存在链接，一部分则是不存在链接，最后得到就是表示是否存在链接的0-1向量；由于只是一个预测结果，不能保证准确性，同样的邻接矩阵Ai也只是一个预测结果，同样不能保证准确性，通过对来完成对邻接矩阵Aj在迭代过程中的更新，更新方式如下：The mapping function g(·) is composed of several linear layers. In order to learn the hidden attributes in the features, the features are finally classified and predicted. Part of it is related to the central subgraph. There are links between them, and some do not have links. It is a 0-1 vector indicating whether there is a link; It is just a prediction result and cannot guarantee accuracy. Similarly, the adjacency matrixAi is also just a prediction result and cannot guarantee accuracy. To complete the update of the adjacency matrix Aj during the iteration process, the update method is as follows:
其中，是由得到的预测邻接矩阵，α是计算因子，用来计算之间的差距，计算方式如下：in, Is The predicted adjacency matrix obtained, α is the calculation factor, used to calculate The difference between them is calculated as follows:
其中，qj是一个表示子图中的节点与中心子图是否存在链接的0-1向量，这样通过不断的更新得到一个相对准确的邻接矩阵之后将更新得到的邻接矩阵和新的输入进行下一次迭代。Among them,qj is a subgraph The nodes and central subgraphs in Is there a linked 0-1 vector, so that a relatively accurate adjacency matrix can be obtained through continuous updating? The adjacency matrix is then updated and new Enter for the next iteration.
步骤四：合并链接，由于通过链接预测之后，可以判断中心是否存在连接，得到聚类结果，但是结果中可能还有一部分簇是由一个个孤立的点组成的；所以，通过寻找孤立点组成簇，计算它与每个簇之间的距离，选择距离最近的簇进行合并，这样就得到最终的聚类结果。Step 4: Merge links. After link prediction, we can determine the center Whether there is a connection, the clustering result is obtained, but some clusters in the result may be composed of isolated points; therefore, by finding isolated points to form clusters, calculating the distance between it and each cluster, and selecting the cluster with the closest distance to merge, the final clustering result is obtained.
本实施例中，选择了两个人脸数据库，分别是MS-Celeb-1M和IJB-B Dataset人脸检测数据集。对于人脸特征的提取，使用已经训练好的ArcFace模型对人脸特征进行提取。然后将MS-Celeb-1M数据集提取的特征输入定义的模型中进行训练，目的是固定化参数。然后将IJB-B Dataset数据集中的图像提取的特征输入到训练好的模型中，得到聚类后的结果，最后通过对聚类后的孤立点进行处理得到最终的结果。In this embodiment, two face databases are selected, namely MS-Celeb-1M and IJB-B Dataset face detection data sets. For the extraction of facial features, the trained ArcFace model is used to extract facial features. The features extracted from the MS-Celeb-1M data set are then input into the defined model for training, with the purpose of fixing the parameters. The features extracted from the images in the IJB-B Dataset data set are then input into the trained model to obtain the clustered results, and finally the final result is obtained by processing the isolated points after clustering.
人脸聚类可以挖掘未标注人脸数据中存在的信息，人脸聚类是的重要工具相对于手工标注节约了大量的时间和精力。在人脸聚类中，无监督方法过于依赖于特定的假设，导致处理真实数据集的复杂聚类结构时，可能出现较大的偏差，然而，在监督聚类方法中，子图过于简单，同时数量较多导致在学习聚类模式的过程中需要耗费大量的时间。在本发明中，对于一张图片中的多张人脸图像，提取出人脸图像中的固有属性和特征，同时根据人脸图像之间的特征和近邻列表来将三角约束扩展到多个顶点。使用图卷积网络建立模型，结合各种约束条件来对邻接矩阵进行修改，通过迭代学习固定参数，最后，在不同的数据集上进行测试，使用聚类评价方法评估聚类效果。通过实验发现，本发明方法不仅准确率很高，而且聚类的过程很快。Face clustering can mine the information existing in unlabeled face data. Face clustering is an important tool that saves a lot of time and energy compared to manual labeling. In face clustering, unsupervised methods rely too much on specific assumptions, which may cause large deviations when processing complex clustering structures of real data sets. However, in supervised clustering methods, subgraphs are too simple, and the large number of them leads to a large amount of time spent in the process of learning clustering patterns. In the present invention, for multiple face images in a picture, the inherent attributes and features in the face images are extracted, and the triangular constraints are extended to multiple vertices according to the features and neighbor lists between the face images. A graph convolutional network is used to establish a model, and the adjacency matrix is modified in combination with various constraints. Fixed parameters are learned through iteration. Finally, tests are performed on different data sets, and clustering evaluation methods are used to evaluate the clustering effect. It is found through experiments that the method of the present invention not only has a high accuracy rate, but also a fast clustering process.
以上所述为本发明的优选实施方案，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010721776.8ACN112101086B (en) | 2020-07-24 | 2020-07-24 | A face clustering method based on link prediction |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010721776.8ACN112101086B (en) | 2020-07-24 | 2020-07-24 | A face clustering method based on link prediction |
| Publication Number | Publication Date |
|---|---|
| CN112101086A CN112101086A (en) | 2020-12-18 |
| CN112101086Btrue CN112101086B (en) | 2024-10-01 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010721776.8AActiveCN112101086B (en) | 2020-07-24 | 2020-07-24 | A face clustering method based on link prediction |
| Country | Link |
|---|---|
| CN (1) | CN112101086B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112766421B (en)* | 2021-03-12 | 2024-09-24 | 清华大学 | Human face clustering method and device based on structure perception |
| CN114170664B (en)* | 2021-12-11 | 2024-08-27 | 南京行者易智能交通科技有限公司 | Face image clustering method and device for link prediction based on self-attention mechanism |
| CN114511905B (en)* | 2022-01-20 | 2024-11-05 | 哈尔滨工程大学 | A face clustering method based on graph convolutional neural network |
| CN115083003B (en)* | 2022-08-23 | 2022-11-22 | 浙江大华技术股份有限公司 | Clustering network training and target clustering method, device, terminal and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109635647B (en)* | 2018-11-05 | 2022-06-10 | 南京航空航天大学 | Multi-picture multi-face clustering method based on constraint condition |
| Title |
|---|
| Linkage Based Face Clustering via Graph Convolution Network;Zhongdao Wang等;《arXiv:1903.11306v3》;20190408;第1-9页* |
| 基于下文约束的人脸聚类算法;罗恒利等;《计算机科学》;20191130;第第46卷卷(第第11A期期);第260-263页* |
| Publication number | Publication date |
|---|---|
| CN112101086A (en) | 2020-12-18 |
| Publication | Publication Date | Title |
|---|---|---|
| CN112101086B (en) | A face clustering method based on link prediction | |
| CN112836672B (en) | Unsupervised data dimension reduction method based on self-adaptive neighbor graph embedding | |
| CN109858390B (en) | Human skeleton behavior recognition method based on end-to-end spatiotemporal graph learning neural network | |
| CN111626128B (en) | A Pedestrian Detection Method Based on Improved YOLOv3 in Orchard Environment | |
| CN112069920A (en) | Cross-domain pedestrian re-identification method based on attribute feature-driven clustering | |
| CN113378913A (en) | Semi-supervised node classification method based on self-supervised learning | |
| CN114511905B (en) | A face clustering method based on graph convolutional neural network | |
| CN108595558A (en) | A kind of image labeling method of data balancing strategy and multiple features fusion | |
| CN119152193B (en) | A YOLO target detection method and system based on differentiable architecture search | |
| CN116524301A (en) | 3D point cloud scene instance shape searching and positioning method based on contrast learning | |
| CN104008177B (en) | Rule base structure optimization and generation method and system towards linguistic indexing of pictures | |
| Mandelli et al. | CAD 3D Model classification by Graph Neural Networks: A new approach based on STEP format | |
| CN116883746A (en) | A graph node classification method based on partition pooling hypergraph neural network | |
| Kalifullah et al. | Retracted: Graph‐based content matching for web of things through heuristic boost algorithm | |
| CN112183464A (en) | Video pedestrian identification method based on deep neural network and graph convolution network | |
| Jiang et al. | Learning to count arbitrary industrial manufacturing workpieces | |
| Odetola et al. | A scalable multilabel classification to deploy deep learning architectures for edge devices | |
| Jiang et al. | On spectral graph embedding: A non-backtracking perspective and graph approximation | |
| Liu et al. | The network representation learning algorithm based on semi-supervised random walk | |
| CN110288606B (en) | Three-dimensional grid model segmentation method of extreme learning machine based on ant lion optimization | |
| CN112132184A (en) | Distribution center site selection method based on N-order neighbor analysis clustering | |
| Wu et al. | Active 3-d shape cosegmentation with graph convolutional networks | |
| CN111160077A (en) | Large-scale dynamic face clustering method | |
| Zhao et al. | Spatial-temporal consistency based crowd emotion recognition | |
| CN115100694A (en) | Fingerprint quick retrieval method based on self-supervision neural network |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |