技术领域technical field
本发明涉及图像处理与计算机视觉领域，特别涉及一种融合深度和色彩信息的图像分割方法。The invention relates to the fields of image processing and computer vision, in particular to an image segmentation method that combines depth and color information.
背景技术Background technique
图像分割是计算机视觉领域的热门课题，在物体识别、目标定位与跟踪、图像检索、三维重建、机器人导航与定位等诸多应用中发挥着重要作用。传统的RGB图像分割方法利用颜色空间、纹理、颜色分布直方图等低层次特征将图像划分成各个互不重叠的连通区域，使得同一区域具有高度相似性，而不同的区域具有较大的差异。当图像中相邻的不同物体颜色相似时，或者边缘特征对比度较低时，这些方法难以将其进行区分。Image segmentation is a hot topic in the field of computer vision, and it plays an important role in many applications such as object recognition, target positioning and tracking, image retrieval, 3D reconstruction, robot navigation and positioning, etc. Traditional RGB image segmentation methods use low-level features such as color space, texture, and color distribution histogram to divide the image into non-overlapping connected regions, so that the same region has a high degree of similarity, while different regions have greater differences. When different adjacent objects in the image have similar colors, or when the contrast of edge features is low, these methods have difficulty in distinguishing them.
近年来，随着传感器技术的快速发展，大量消费级的深度获取设备逐步走向市场，得到了广泛应用，如微软的Kinect和Intel的 RealSense系列。这些传感器通常搭载彩色摄像机，可以同步获取配准的深度和彩色影像，这为传统的基于二维彩色空间的图像分割技术提供了更多的信息和可能。In recent years, with the rapid development of sensor technology, a large number of consumer-level depth acquisition devices have gradually entered the market and have been widely used, such as Microsoft's Kinect and Intel's RealSense series. These sensors are usually equipped with color cameras, which can simultaneously acquire the registered depth and color images, which provides more information and possibilities for the traditional image segmentation technology based on two-dimensional color space.
发明内容Contents of the invention
本发明的目的在于提供一种利用深度传感器获取的RGB-D数据、融合深度和色彩信息的图像分割方法，主要应用于室内场景的物体识别。The purpose of the present invention is to provide an image segmentation method using RGB-D data acquired by a depth sensor and fusing depth and color information, which is mainly applied to object recognition in indoor scenes.
同步获取配准的深度和彩色影像，不仅可以得到场景的二维彩色信息，还可以得到场景的三维空间信息，因此，在二维彩色空间中难以区分的物体，便有可能通过三维空间中的位置信息进行区分，将目标从背景中识别出来。基于该原理，本发明提供的一种融合深度和色彩信息的图像分割方法，主要包括以下步骤：Obtaining the registered depth and color images simultaneously can not only obtain the two-dimensional color information of the scene, but also obtain the three-dimensional space information of the scene. Position information is used to distinguish objects from backgrounds. Based on this principle, an image segmentation method that combines depth and color information provided by the present invention mainly includes the following steps:
步骤1，RGB-D数据预处理：以图像边缘信息为引导进行中值滤波，增强深度图像质量；Step 1, RGB-D data preprocessing: conduct median filtering guided by image edge information to enhance depth image quality;
步骤2，RGB-D图像超像素分割：融合彩色和深度信息，对图像进行过分割；Step 2, RGB-D image superpixel segmentation: fuse color and depth information to over-segment the image;
步骤3，超像素合并：采用基于图论的谱聚类方法将相似的超像素进行合并，将聚类转化为图的划分问题，完成图像的分割。Step 3, superpixel merging: use the graph theory-based spectral clustering method to merge similar superpixels, transform the clustering into a graph division problem, and complete the image segmentation.
进一步地，所述步骤1的方法包括：Further, the method of step 1 includes:
对彩色图像和深度图像分别进行边缘提取，生成各自的边缘图像；Edge extraction is performed on the color image and the depth image separately to generate respective edge images;
将彩色边缘图像和深度边缘图像进行融合处理：在深度边缘图像上，计算每一个像素的梯度方向，如果对应的彩色边缘图像边缘方向与该方向近似，则使用彩色图像边缘，否则采用深度图像边缘，如此得到融合后的边缘图像信息；Fusion processing of color edge image and depth edge image: on the depth edge image, calculate the gradient direction of each pixel, if the edge direction of the corresponding color edge image is similar to this direction, use the color image edge, otherwise use the depth image edge , so that the fused edge image information is obtained;
在深度图像上，采用3×3大小的模板对每个有效像素进行中值滤波；On the depth image, a 3×3 template is used to perform median filtering on each effective pixel;
颜色空间变换：将相机获取的彩色图像由RGB颜色空间转换为 CIELab颜色空间。Color space conversion: convert the color image acquired by the camera from the RGB color space to the CIELab color space.
进一步地，所述步骤2包括以下步骤：Further, said step 2 includes the following steps:
2.1基于深度信息的三维点云重建：基于深度图中的深度信息以及相机的内参数，将深度图中的像素坐标(x,y)转化为三维空间坐标(X，Y，Z)；2.1 3D point cloud reconstruction based on depth information: Based on the depth information in the depth map and the internal parameters of the camera, the pixel coordinates (x, y) in the depth map are converted into three-dimensional space coordinates (X, Y, Z);
2.2超像素预分割并初始化聚类中心：假设图像有N个像素，设定将图像分割为K个超像素块，则每个超像素块有N/K个像素，每个超像素块的间距为将初始聚类中心设定于超像素块的中心；2.2 Superpixel pre-segmentation and initialization of the clustering center: Assuming that the image has N pixels, and the image is divided into K superpixel blocks, each superpixel block has N/K pixels, and the distance between each superpixel block for Set the initial clustering center to the center of the superpixel block;
2.3区域聚类标记：在各个超像素聚类中心的2S×2S邻域范围内，计算每个有效像素在8维特征空间[L,a,b,x,y,X,Y,Z]与聚类中心的距离：2.3 Regional clustering labeling: within the 2S×2S neighborhood of each superpixel cluster center, calculate each effective pixel in the 8-dimensional feature space [L, a, b, x, y, X, Y, Z] and Distance from cluster centers:
D＝DLab+αDxy+βDXYZD＝DLab +αDxy +βDXYZ
其中下标c表示每个超像素的聚类中心，下标i表示搜索范围内每个有效像素，DLab,Dxy,DXYZ分别表示有效像素i在颜色特征Lab、图像二维坐标xy和基于深度的三维空间坐标XYZ上与聚类中心点的欧式距离，α,β为权重，用以平衡各项的重要程度，根据具体场景数据设定，D为最终的特征距离；The subscript c indicates the clustering center of each superpixel, the subscript i indicates each effective pixel within the search range, DLab , Dxy , DXYZ respectively indicate the effective pixel i in the color feature Lab, image two-dimensional coordinates xy and Based on the Euclidean distance between the depth-based three-dimensional space coordinates XYZ and the cluster center point, α and β are weights to balance the importance of each item, set according to the specific scene data, D is the final feature distance;
将每个像素划分到与其特征距离D最小的聚类中心所属的超像素中，即与该聚类中心标记为相同的类；Divide each pixel into the superpixel belonging to the cluster center with the smallest feature distance D, that is, mark the same class as the cluster center;
2.4更新聚类中心：当所有有效像素处理标记完成后，根据像素分类的结果，更新聚类中心，即统计各个超像素块中包含的像素个数，并计算其平均值，得到新的聚类中心；2.4 Update the clustering center: when all effective pixel processing marks are completed, the clustering center is updated according to the result of pixel classification, that is, the number of pixels contained in each superpixel block is counted, and the average value is calculated to obtain a new clustering center;
2.5迭代聚类：重复2.3、2.4两步，直到聚类稳定，即聚类中心、像素标记不再发生变化，至此完成超像素分割。2.5 Iterative clustering: Repeat steps 2.3 and 2.4 until the clustering is stable, that is, the clustering center and pixel labels do not change, and the superpixel segmentation is completed.
进一步地，所述步骤3包括以下步骤：Further, said step 3 includes the following steps:
3.1相似矩阵构建：将每个超像素作为图的节点，超像素之间的相似度作为边的权值，构建所有超像素的无向图。假设超像素的个数为K，则图的相似度矩阵W∈RK×K是一个对称阵，其每个元素wpq为超像素p和超像素q(p＝1，…，K；q＝1,…，K)之间的相似程度，采用颜色相似度、纹理相似度和三维空间连通性进行衡量：3.1 Construction of similarity matrix: each superpixel is used as a node of the graph, and the similarity between superpixels is used as the weight of the edge to construct an undirected graph of all superpixels. Assuming that the number of superpixels is K, the similarity matrix W∈RK×K of the graph is a symmetric matrix, and each element wpq is a superpixel p and a superpixel q (p=1,...,K; q =1,...,K), the degree of similarity between them is measured by color similarity, texture similarity and three-dimensional spatial connectivity:
wpq＝Dcolor(p,q)+Dtexture(p,q)+γDspacewpq ＝Dcolor (p,q)+Dtexture (p,q)+γDspace
其中，Dcolor(p,q)表示颜色相似度，Dtexture(p,q)表示纹理相似度， Dspace表示三维空间连通性，γ为权重因子，根据深度数据质量调节三维空间连通性的权重，三维空间指所述基于深度信息重建的三维空间；Among them, Dcolor (p, q) represents color similarity, Dtexture (p, q) represents texture similarity, Dspace represents three-dimensional spatial connectivity, and γ is a weight factor, which adjusts the weight of three-dimensional spatial connectivity according to the quality of depth data , the three-dimensional space refers to the three-dimensional space reconstructed based on the depth information;
3.2根据构造的相似度矩阵，对超像素进行谱聚类，将高度相似的超像素进行合并，完成图像分割。3.2 According to the constructed similarity matrix, perform spectral clustering on superpixels, merge highly similar superpixels, and complete image segmentation.
进一步地，所述中值滤波的方法为：Further, the method of median filtering is:
记D(x,y)为当前有效像素深度值，ND(x,y)为其8邻域，则有：Note that D(x, y) is the current effective pixel depth value, and ND (x, y) is its 8 neighbors, then:
D(x,y)＝median(ND(x,y))D(x,y)=median(ND (x,y))
如果当前像素8邻域内不存在边缘像素时，则按照上式进行计算；如果存在边缘像素，则根据当前像素所在边缘一侧的邻域像素进行中值滤波。If there is no edge pixel in the neighborhood of the current pixel 8, the calculation is performed according to the above formula; if there is an edge pixel, median filtering is performed according to the neighboring pixels on the side of the edge where the current pixel is located.
进一步地，所述将深度图的单维度信息转化为三维空间坐标的具体方法为Further, the specific method for converting the single-dimensional information of the depth map into three-dimensional space coordinates is as follows
其中，(x,y)为深度图中某点的图像像素坐标，d为深度图像中该点的深度值，fx、fy分别为深度相机在x、y方向上的焦距，(cx,cy)为像主点坐标，fx,fy,cx,cy由相机生产商给出，(X,Y,Z)为该点的三维空间坐标。Among them, (x, y) is the image pixel coordinates of a point in the depth map, d is the depth value of the point in the depth image, fx and fy are the focal lengths of the depth camera in the x and y directions respectively, (cx ,cy ) are the principal point coordinates of the image, fx ,fy ,cx ,cy are given by the camera manufacturer, and (X,Y,Z) are the three-dimensional space coordinates of the point.
进一步地，所述颜色相似度的计算方法为：Further, the calculation method of the color similarity is:
在RGB颜色空间上，对于超像素p，分别统计RGB三个通道上的归一化颜色直方图Hp,R,Hp,G,Hp,B，同理对超像素q；计算超像素p 和q三个通道颜色直方图之间的Bhattacharyya系数：In the RGB color space, for the superpixel p, count the normalized color histograms Hp, R , Hp, G , Hp, B on the three channels of RGB, respectively, and for the superpixel q in the same way; calculate the superpixel Bhattacharyya coefficient between p and q three channel color histograms:
其中i为归一化的像素值；where i is the normalized pixel value;
则RGB三个通道直方图之间的Bhattacharyya系数之和则为颜色相似度：Then the sum of the Bhattacharyya coefficients between the three RGB channel histograms is the color similarity:
Dcolor(p,q)＝BhR(p,q)+BhG(p,q)+BhB(p,q)。Dcolor (p, q) = BhR (p, q) + BhG (p, q) + BhB (p, q).
进一步地，所述纹理相似度的计算方法为：Further, the calculation method of the texture similarity is:
在Lab颜色空间上，选用L通道值对超像素采用局部二值模式 LBP直方图描述，采用Bhattacharyya系数衡量超像素p、q之间的纹理相似度Dtexture(p,q)。In the Lab color space, the L channel value is selected to describe the superpixel using the local binary mode LBP histogram, and the Bhattacharyya coefficient is used to measure the texture similarity Dtexture (p,q) between the superpixel p and q.
进一步地，所述三维空间连通性的计算方法为：Further, the calculation method of the three-dimensional space connectivity is:
采用高斯核函数，其中σ为尺度因子，根据实际场景设定，XYZ 为超像素聚类中心的三维空间坐标，则点p和q之间的三维空间连通性Using the Gaussian kernel function, where σ is the scale factor, set according to the actual scene, XYZ is the three-dimensional space coordinates of the superpixel clustering center, then the three-dimensional space connectivity between points p and q
本发明根据成像几何原理，将深度图转化为三维点云，进而综合深度和色彩两方面的信息，对图像进行超像素分割和聚类，从而完成图像分割。该方法充分利用了与彩色影像配准的深度图像中蕴含的空间位置信息，从而将二维彩色空间中难以区分的物体，通过三维空间中的位置信息进行区分，将目标从背景中识别出来，提高了图像分割的质量和精度。According to the principle of imaging geometry, the present invention converts the depth map into a three-dimensional point cloud, and then integrates the information of both depth and color, and performs superpixel segmentation and clustering on the image, thereby completing the image segmentation. This method makes full use of the spatial position information contained in the depth image registered with the color image, so that the objects that are difficult to distinguish in the two-dimensional color space are distinguished through the position information in the three-dimensional space, and the target is identified from the background. Improved the quality and accuracy of image segmentation.
应了解的是，上述一般描述及以下具体实施方式仅为示例性及阐释性的，其并不能限制本发明所欲主张的范围。It should be understood that the above general description and the following specific embodiments are only exemplary and explanatory, and should not limit the scope of the present invention.
附图说明Description of drawings
下面的附图是本发明的说明书的一部分，其绘示了本发明的示例实施例，所附附图与说明书的描述一起用来说明本发明的原理。The accompanying drawings, which are a part of the specification of this invention, illustrate example embodiments of the invention and together with the description serve to explain the principles of the invention.
图1为根据示例性实施例的融合深度和色彩信息的图像分割方法流程图；1 is a flowchart of an image segmentation method for fusing depth and color information according to an exemplary embodiment;
图2为应用根据示例性实施例的融合深度和色彩信息的图像分割方法的分割效果举例。FIG. 2 is an example of a segmentation effect of applying the image segmentation method of fusing depth and color information according to an exemplary embodiment.
具体实施方式Detailed ways
现详细说明本发明的多种示例性实施方式，该详细说明不应认为是对本发明的限制，而应理解为是对本发明的某些方面、特性和实施方案的更详细的描述。Various exemplary embodiments of the present invention will now be described in detail. The detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features and embodiments of the present invention.
在不背离本发明的范围或精神的情况下，可对本发明说明书的具体实施方式做多种改进和变化，这对本领域技术人员而言是显而易见的。由本发明的说明书得到的其他实施方式对技术人员而言是显而易见得的。本申请说明书和实施例仅是示例性的。It will be apparent to those skilled in the art that various modifications and changes can be made in the specific embodiments of the present invention described herein without departing from the scope or spirit of the present invention. Other embodiments will be apparent to the skilled person from the description of the present invention. The specification and examples in this application are exemplary only.
附图1中给出了本发明示例性实施例的流程图。如图所示，本实施例的具体步骤包括：A flowchart of an exemplary embodiment of the present invention is given in FIG. 1 . As shown in the figure, the specific steps of this embodiment include:
步骤1：RGB-D数据预处理。由于工作距离、遮挡、噪声等影响，需要针对深度图像进行预处理，以图像边缘信息为引导进行中值滤波，增强深度图像质量。具体包括以下步骤：Step 1: RGB-D data preprocessing. Due to the influence of working distance, occlusion, noise, etc., it is necessary to preprocess the depth image, and use the image edge information as a guide to perform median filtering to enhance the quality of the depth image. Specifically include the following steps:
1.1对彩色图像和深度图像进行边缘提取，生成各自的边缘图像。作为优选方案，本实施例采用Canny边缘检测算子。1.1 Perform edge extraction on color images and depth images to generate respective edge images. As a preferred solution, this embodiment uses a Canny edge detection operator.
1.2在深度边缘图像中，对每一个边缘像素的8邻域进行检测，如果没有无效像素存在，则该像素为真正的边缘像素。之后将彩色边缘图像和深度边缘图像进行融合处理：在深度边缘图像上，计算每一个像素的梯度方向，如果对应的彩色边缘图像边缘方向该方向近似，则使用彩色图像边缘，否则采用深度图像边缘。如此得到融合后的边缘图像信息。1.2 In the depth edge image, detect the 8 neighbors of each edge pixel, if there is no invalid pixel, then the pixel is a real edge pixel. Then the color edge image and the depth edge image are fused: on the depth edge image, calculate the gradient direction of each pixel, if the direction of the corresponding color edge image edge direction is similar, use the color image edge, otherwise use the depth image edge . In this way, the fused edge image information is obtained.
1.3在深度图像上，对每个有效像素进行中值滤波。优选采用3× 3大小的模板：记D(x,y)为当前有效像素深度值，ND(x,y)为其8邻域，则有：1.3 On the depth image, median filtering is performed on each effective pixel. It is preferable to use a template with a size of 3 × 3: record D(x, y) as the current effective pixel depth value, and ND (x, y) as its 8 neighbors, then:
D(x,y)＝median(ND(x,y))D(x,y)=median(ND (x,y))
如果当前像素8邻域内不存在边缘像素时，则按照上式进行计算；如果存在边缘像素，则根据当前像素所在边缘一侧的邻域像素进行中值滤波。If there is no edge pixel in the neighborhood of the current pixel 8, the calculation is performed according to the above formula; if there is an edge pixel, median filtering is performed according to the neighboring pixels on the side of the edge where the current pixel is located.
1.4颜色空间变换：将相机获取的彩色图像由RGB颜色空间转换为CIELab颜色空间。因为其具有更宽广的色域，同时可以弥补RGB 色彩模型分布不均匀的不足。1.4 Color space transformation: Convert the color image acquired by the camera from RGB color space to CIELab color space. Because it has a wider color gamut and can make up for the uneven distribution of the RGB color model.
步骤2：RGB-D图像超像素分割。场景的复杂性使得难以直接对图像进行准确的分割，因而先将整幅图像分割成较大数量的小块，使得相邻的相似像素划分到同一个块中，称之为超像素。融合彩色和深度信息，对图像进行过分割，具体包括以下步骤：Step 2: RGB-D image superpixel segmentation. The complexity of the scene makes it difficult to directly segment the image accurately, so the entire image is first divided into a large number of small blocks, so that adjacent similar pixels are divided into the same block, which is called superpixels. Integrating color and depth information to over-segment the image includes the following steps:
2.1三维点云重建。深度图包含三维空间信息，但其x、y坐标为像素坐标，因此只能反映出物体在三维空间中一个方向上的信息，直接使用会丢失其余两个维度的信息，因而需要将深度图的单维度信息转化为三维空间坐标。2.1 3D point cloud reconstruction. The depth map contains three-dimensional space information, but its x and y coordinates are pixel coordinates, so it can only reflect the information of the object in one direction in the three-dimensional space. Direct use will lose the information of the other two dimensions, so the depth map needs to be Single-dimensional information is transformed into three-dimensional space coordinates.
记(X,Y,Z)为某个点的三维空间坐标，(x,y)为深度图像中对应的图像像素坐标，d为深度图像中对应的深度值。fx,fy分别为深度相机在 x,y方向上的焦距，(cx,cy)为像主点坐标，fx,fy,cx,cy由相机生产商给出，通常称之为内参数。根据成像几何原理，由深度值可计算出三维空间坐标：Record (X, Y, Z) as the three-dimensional space coordinates of a certain point, (x, y) as the corresponding image pixel coordinates in the depth image, and d as the corresponding depth value in the depth image. fx , fy are the focal lengths of the depth camera in the x and y directions respectively, (cx , cy ) are the coordinates of the principal point of the image, fx , fy , cx , cy are given by the camera manufacturer, usually called internal parameters. According to the principle of imaging geometry, the three-dimensional space coordinates can be calculated from the depth value:
此时，RGB-D图像的每个有效像素可用颜色特征L ab、图像二维坐标x y和三维空间坐标XYZ描述，用以衡量颜色相似性以及二维空间和三维空间的连续性。At this time, each effective pixel of the RGB-D image can be described by the color feature Lab, the image two-dimensional coordinates x y and the three-dimensional space coordinates XYZ, which are used to measure the color similarity and the continuity of the two-dimensional space and the three-dimensional space.
2.2初始化聚类中心。假设图像有N个像素，设定将图像分割为 K个超像素块，则每个超像素块有N/K个像素，每个超像素块的间距为将初始聚类中心设定于超像素块的中心。2.2 Initialize the cluster center. Assuming that the image has N pixels, and the image is divided into K superpixel blocks, each superpixel block has N/K pixels, and the distance between each superpixel block is Set the initial clustering center to the center of the superpixel block.
2.3区域聚类标记。在各个超像素聚类中心的2S×2S邻域范围内，计算每个有效像素在8维特征空间[L,a,b,x,y,X,Y,Z]与聚类中心的距离：2.3 Region clustering labeling. Within the 2S×2S neighborhood of each superpixel cluster center, calculate the distance between each effective pixel in the 8-dimensional feature space [L, a, b, x, y, X, Y, Z] and the cluster center:
D＝DLab+αDxy+βDXYZD＝DLab +αDxy +βDXYZ
其中下标c表示每个超像素的聚类中心，下标i表示搜索范围内每个有效像素，DLab,Dxy,DXYZ分别表示有效像素i在颜色特征L ab、图像二维坐标xy和三维空间坐标XYZ上与聚类中心点的欧式距离，α,β为权重，用以平衡各项的重要程度，根据具体场景数据设定。D为最终的特征距离。将每个像素划分到与其特征距离最小的聚类中心所属的超像素中，即与该聚类中心标记为相同的类。The subscript c indicates the clustering center of each superpixel, the subscript i indicates each effective pixel within the search range, DLab , Dxy , DXYZ respectively indicate the effective pixel i in the color feature Lab and the image two-dimensional coordinate xy and the Euclidean distance from the cluster center point on the three-dimensional space coordinates XYZ, α, β are weights to balance the importance of each item, set according to the specific scene data. D is the final feature distance. Divide each pixel into the superpixel to which the cluster center with the smallest distance to its feature belongs, that is, it is labeled with the same class as the cluster center.
2.4更新聚类中心。当所有有效像素处理标记完成后，根据像素分类的结果，更新聚类中心。即统计各个超像素块中包含的像素个数，并计算其平均值，得到新的聚类中心。2.4 Update the cluster center. When all valid pixels are processed and marked, the clustering center is updated according to the result of pixel classification. That is, count the number of pixels contained in each superpixel block, and calculate the average value to obtain a new cluster center.
2.5迭代聚类，重复2.3、2.4两步，直到聚类稳定，即聚类中心、像素标记不再发生变化。一般迭代5-6次即可获得稳定的聚类，至此完成超像素分割。2.5 Iterative clustering, repeating steps 2.3 and 2.4 until the clustering is stable, that is, the clustering center and pixel markers no longer change. Generally, stable clustering can be obtained after 5-6 iterations, and the superpixel segmentation is completed so far.
步骤3：超像素合并。采用基于图论的谱聚类方法将相似的超像素进行合并，将聚类转化为图的划分问题，完成图像的分割，具体包括以下步骤：Step 3: Superpixel binning. The spectral clustering method based on graph theory is used to merge similar superpixels, transform the clustering into a graph partition problem, and complete the image segmentation, which specifically includes the following steps:
3.1相似矩阵构建。将每个超像素作为图的节点，超像素之间的相似度作为边的权值，构建所有超像素的无向图。假设超像素的个数为K，则图的相似度矩阵W∈RK×K是一个对称阵，其每个元素wpq为超像素p和超像素q之间的相似程度，采用颜色相似度、纹理相似度和三维空间连通性进行衡量：3.1 Similarity matrix construction. Taking each superpixel as a node of the graph, and the similarity between superpixels as the weight of the edge, an undirected graph of all superpixels is constructed. Assuming that the number of superpixels is K, the similarity matrix W∈RK×K of the graph is a symmetric matrix, and each element wpq is the similarity between superpixel p and superpixel q, using color similarity , texture similarity and 3D spatial connectivity are measured:
wpq＝Dcolor(p,q)+Dtexture(p,q)+γDspacewpq ＝Dcolor (p,q)+Dtexture (p,q)+γDspace
其中γ为权重因子，根据深度数据质量调节三维空间连通性的权重。where γ is a weighting factor that adjusts the weight of 3D spatial connectivity according to the quality of depth data.
颜色相似度：在RGB颜色空间上，对于超像素p，分别统计RGB 三个通道上的归一化颜色直方图Hp,R,Hp,G,Hp,B，同理对超像素q。计算超像素p和q三个通道颜色直方图之间的Bhattacharyya系数：Color similarity: In the RGB color space, for the superpixel p, count the normalized color histograms Hp, R , Hp, G , Hp, B on the three channels of RGB respectively, and similarly for the superpixel q . Calculate the Bhattacharyya coefficient between the three channel color histograms of superpixels p and q:
其中i为归一化的像素值。RGB三个通道直方图之间的 Bhattacharyya系数之和则为颜色相似度：where i is the normalized pixel value. The sum of the Bhattacharyya coefficients between the three RGB channel histograms is the color similarity:
Dcolor(p,q)＝BhR(p,q)+BhG(p,q)+BhB(p,q)Dcolor (p,q)＝BhR (p,q)+BhG (p,q)+BhB (p,q)
纹理相似度：在Lab颜色空间上，选用L通道值对超像素采用局部二值模式LBP直方图描述，与颜色相似度类似，采用Bhattacharyya 系数衡量超像素之间的纹理相似度Dtexture(p,q)。Texture similarity: In the Lab color space, the L channel value is selected to describe the superpixel using the local binary mode LBP histogram, which is similar to the color similarity, and the Bhattacharyya coefficient is used to measure the texture similarity between superpixels Dtexture (p, q).
三维空间连通性：采用高斯核函数，其中σ为尺度因子，根据实际场景设定，XYZ为超像素聚类中心的三维空间坐标：Three-dimensional spatial connectivity: Gaussian kernel function is used, where σ is the scale factor, set according to the actual scene, XYZ is the three-dimensional spatial coordinates of the superpixel clustering center:
3.2根据构造的相似度矩阵，对超像素进行谱聚类，将高度相似的超像素进行合并，完成图像分割。3.2 According to the constructed similarity matrix, perform spectral clustering on superpixels, merge highly similar superpixels, and complete image segmentation.
可见，本实施例所述方法，根据成像几何原理，将深度图转化为三维点云，进而综合深度和色彩两方面的信息，对图像进行超像素分割和聚类，从而完成图像分割。该方法充分利用了与彩色影像配准的深度图像中蕴含的空间位置信息，从而将二维彩色空间中难以区分的物体，通过三维空间中的位置信息进行区分，将目标从背景中识别出来，提高了图像分割的质量和精度。It can be seen that the method described in this embodiment converts the depth map into a 3D point cloud according to the principle of imaging geometry, and then integrates the information of depth and color to perform superpixel segmentation and clustering on the image, thereby completing the image segmentation. This method makes full use of the spatial position information contained in the depth image registered with the color image, so that the objects that are difficult to distinguish in the two-dimensional color space are distinguished through the position information in the three-dimensional space, and the target is identified from the background. Improved the quality and accuracy of image segmentation.
应用效果示例：Examples of applied effects:
如附图2所示，图中景物和背景颜色相近(如图2(a))，边缘特征对比度也较低，用传统的RGB图像分割方法对其进行分割，存在困难，采用本发明所述方法，综合利用色彩和深度(如图2(b))信息进行分割，得到结果如图2(c)，图中已对每一部分进行了清楚的分割。As shown in accompanying drawing 2, the scene in the figure and background color are similar (as Fig. 2 (a)), and edge feature contrast is also low, it is difficult to segment it with traditional RGB image segmentation method, adopts the present invention method, the information of color and depth (as shown in Figure 2(b)) is used for segmentation, and the result is shown in Figure 2(c), in which each part has been clearly segmented.
以上所述仅为本发明示意性的具体实施方式，在不脱离本发明的构思和原则的前提下，任何本领域的技术人员所做出的等同变化与修改，均应属于本发明保护的范围。The above are only illustrative specific implementations of the present invention. Under the premise of not departing from the concept and principle of the present invention, any equivalent changes and modifications made by those skilled in the art shall fall within the protection scope of the present invention. .
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910909933.5ACN110610505A (en) | 2019-09-25 | 2019-09-25 | An Image Segmentation Method Combining Depth and Color Information |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910909933.5ACN110610505A (en) | 2019-09-25 | 2019-09-25 | An Image Segmentation Method Combining Depth and Color Information |
| Publication Number | Publication Date |
|---|---|
| CN110610505Atrue CN110610505A (en) | 2019-12-24 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910909933.5APendingCN110610505A (en) | 2019-09-25 | 2019-09-25 | An Image Segmentation Method Combining Depth and Color Information |
| Country | Link |
|---|---|
| CN (1) | CN110610505A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111079713A (en)* | 2019-12-31 | 2020-04-28 | 帷幄匠心科技（杭州）有限公司 | Method for extracting pedestrian color features and terminal equipment |
| CN111190981A (en)* | 2019-12-25 | 2020-05-22 | 中国科学院上海微系统与信息技术研究所 | Method and device for constructing three-dimensional semantic map, electronic equipment and storage medium |
| CN111709483A (en)* | 2020-06-18 | 2020-09-25 | 山东财经大学 | A multi-feature-based superpixel clustering method and device |
| CN112183378A (en)* | 2020-09-29 | 2021-01-05 | 北京深睿博联科技有限责任公司 | Road slope estimation method and device based on color and depth image |
| CN112364730A (en)* | 2020-10-29 | 2021-02-12 | 济南大学 | Hyperspectral ground object automatic classification method and system based on sparse subspace clustering |
| CN112785608A (en)* | 2021-02-09 | 2021-05-11 | 哈尔滨理工大学 | Medical image segmentation method for improving SNIC (single noise integrated circuit) based on adaptive parameters |
| CN112880563A (en)* | 2021-01-22 | 2021-06-01 | 北京航空航天大学 | Single-dimensional pixel combination mode equivalent narrow-area-array camera spatial position measuring method |
| CN112991238A (en)* | 2021-02-22 | 2021-06-18 | 上海市第四人民医院 | Texture and color mixing type food image segmentation method, system, medium and terminal |
| CN113199479A (en)* | 2021-05-11 | 2021-08-03 | 梅卡曼德（北京）机器人科技有限公司 | Trajectory generation method and apparatus, electronic device, storage medium, and 3D camera |
| CN113542142A (en)* | 2020-04-14 | 2021-10-22 | 中国移动通信集团浙江有限公司 | Portrait anti-counterfeiting detection method and device and computing equipment |
| WO2021228194A1 (en)* | 2020-05-15 | 2021-11-18 | 上海非夕机器人科技有限公司 | Cable detection method, robot and storage device |
| CN113689549A (en)* | 2021-08-03 | 2021-11-23 | 长沙宏达威爱信息科技有限公司 | Modeling method and digital design system |
| CN113793349A (en)* | 2021-01-05 | 2021-12-14 | 北京京东乾石科技有限公司 | Target detection method and apparatus, computer-readable storage medium, and electronic device |
| CN114255326A (en)* | 2020-09-10 | 2022-03-29 | 广东博智林机器人有限公司 | Point cloud data processing method and device, electronic equipment and storage medium |
| CN115187626A (en)* | 2021-04-06 | 2022-10-14 | 奥比中光科技集团股份有限公司 | A method and system for automatic labeling of target objects |
| CN115511718A (en)* | 2021-06-07 | 2022-12-23 | 广州视源电子科技股份有限公司 | PCB image correction method and device, terminal equipment and storage medium |
| CN116778095A (en)* | 2023-08-22 | 2023-09-19 | 苏州海赛人工智能有限公司 | Three-dimensional reconstruction method based on artificial intelligence |
| CN120107291A (en)* | 2025-05-08 | 2025-06-06 | 中知厚德(北京)信息科技有限公司 | A text image superpixel segmentation method and system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455984A (en)* | 2013-09-02 | 2013-12-18 | 清华大学深圳研究生院 | Method and device for acquiring Kinect depth image |
| CN104574375A (en)* | 2014-12-23 | 2015-04-29 | 浙江大学 | Image Saliency Detection Method Combining Color and Depth Information |
| CN105469369A (en)* | 2015-11-27 | 2016-04-06 | 中国科学院自动化研究所 | Digital image filtering method and system based on segmented image |
| CN106997591A (en)* | 2017-03-21 | 2017-08-01 | 南京理工大学 | A kind of super voxel dividing method of RGB D image mutative scales |
| CN108154104A (en)* | 2017-12-21 | 2018-06-12 | 北京工业大学 | A kind of estimation method of human posture based on depth image super-pixel union feature |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103455984A (en)* | 2013-09-02 | 2013-12-18 | 清华大学深圳研究生院 | Method and device for acquiring Kinect depth image |
| CN104574375A (en)* | 2014-12-23 | 2015-04-29 | 浙江大学 | Image Saliency Detection Method Combining Color and Depth Information |
| CN105469369A (en)* | 2015-11-27 | 2016-04-06 | 中国科学院自动化研究所 | Digital image filtering method and system based on segmented image |
| CN106997591A (en)* | 2017-03-21 | 2017-08-01 | 南京理工大学 | A kind of super voxel dividing method of RGB D image mutative scales |
| CN108154104A (en)* | 2017-12-21 | 2018-06-12 | 北京工业大学 | A kind of estimation method of human posture based on depth image super-pixel union feature |
| Title |
|---|
| 杨龙等: "基于超像素分割的红外图像细节增强算法", 《红外》* |
| 涂淑琴等: "RGB-D图像分类方法研究综述", 《激光与光电子学进展》* |
| 赵轩等: "RGB-D图像中的分步超像素聚合和多模态融合目标检测", 《中国图象图形学报》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111190981B (en)* | 2019-12-25 | 2020-11-06 | 中国科学院上海微系统与信息技术研究所 | A method, device, electronic device and storage medium for constructing a three-dimensional semantic map |
| CN111190981A (en)* | 2019-12-25 | 2020-05-22 | 中国科学院上海微系统与信息技术研究所 | Method and device for constructing three-dimensional semantic map, electronic equipment and storage medium |
| CN111079713A (en)* | 2019-12-31 | 2020-04-28 | 帷幄匠心科技（杭州）有限公司 | Method for extracting pedestrian color features and terminal equipment |
| CN113542142B (en)* | 2020-04-14 | 2024-03-22 | 中国移动通信集团浙江有限公司 | Portrait anti-fake detection method and device and computing equipment |
| CN113542142A (en)* | 2020-04-14 | 2021-10-22 | 中国移动通信集团浙江有限公司 | Portrait anti-counterfeiting detection method and device and computing equipment |
| WO2021228194A1 (en)* | 2020-05-15 | 2021-11-18 | 上海非夕机器人科技有限公司 | Cable detection method, robot and storage device |
| CN111709483A (en)* | 2020-06-18 | 2020-09-25 | 山东财经大学 | A multi-feature-based superpixel clustering method and device |
| CN114255326A (en)* | 2020-09-10 | 2022-03-29 | 广东博智林机器人有限公司 | Point cloud data processing method and device, electronic equipment and storage medium |
| CN112183378A (en)* | 2020-09-29 | 2021-01-05 | 北京深睿博联科技有限责任公司 | Road slope estimation method and device based on color and depth image |
| CN112364730A (en)* | 2020-10-29 | 2021-02-12 | 济南大学 | Hyperspectral ground object automatic classification method and system based on sparse subspace clustering |
| CN112364730B (en)* | 2020-10-29 | 2023-01-17 | 济南大学 | Method and system for automatic classification of hyperspectral objects based on sparse subspace clustering |
| CN113793349A (en)* | 2021-01-05 | 2021-12-14 | 北京京东乾石科技有限公司 | Target detection method and apparatus, computer-readable storage medium, and electronic device |
| CN112880563B (en)* | 2021-01-22 | 2021-12-28 | 北京航空航天大学 | A method for measuring the spatial position of an equivalent narrow-area array camera in single-dimensional pixel binning mode |
| CN112880563A (en)* | 2021-01-22 | 2021-06-01 | 北京航空航天大学 | Single-dimensional pixel combination mode equivalent narrow-area-array camera spatial position measuring method |
| CN112785608A (en)* | 2021-02-09 | 2021-05-11 | 哈尔滨理工大学 | Medical image segmentation method for improving SNIC (single noise integrated circuit) based on adaptive parameters |
| CN112991238B (en)* | 2021-02-22 | 2023-08-22 | 上海市第四人民医院 | Food image segmentation method, system and medium based on texture and color mixing |
| CN112991238A (en)* | 2021-02-22 | 2021-06-18 | 上海市第四人民医院 | Texture and color mixing type food image segmentation method, system, medium and terminal |
| CN115187626A (en)* | 2021-04-06 | 2022-10-14 | 奥比中光科技集团股份有限公司 | A method and system for automatic labeling of target objects |
| CN113199479A (en)* | 2021-05-11 | 2021-08-03 | 梅卡曼德（北京）机器人科技有限公司 | Trajectory generation method and apparatus, electronic device, storage medium, and 3D camera |
| CN115511718A (en)* | 2021-06-07 | 2022-12-23 | 广州视源电子科技股份有限公司 | PCB image correction method and device, terminal equipment and storage medium |
| CN113689549A (en)* | 2021-08-03 | 2021-11-23 | 长沙宏达威爱信息科技有限公司 | Modeling method and digital design system |
| CN113689549B (en)* | 2021-08-03 | 2024-04-09 | 长沙宏达威爱信息科技有限公司 | Modeling method and digital design system |
| CN116778095A (en)* | 2023-08-22 | 2023-09-19 | 苏州海赛人工智能有限公司 | Three-dimensional reconstruction method based on artificial intelligence |
| CN116778095B (en)* | 2023-08-22 | 2023-10-27 | 苏州海赛人工智能有限公司 | Three-dimensional reconstruction method based on artificial intelligence |
| CN120107291A (en)* | 2025-05-08 | 2025-06-06 | 中知厚德(北京)信息科技有限公司 | A text image superpixel segmentation method and system |
| Publication | Publication Date | Title |
|---|---|---|
| CN110610505A (en) | An Image Segmentation Method Combining Depth and Color Information | |
| CN109166077B (en) | Image alignment method and device, readable storage medium and computer equipment | |
| US11727661B2 (en) | Method and system for determining at least one property related to at least part of a real environment | |
| Hirschmuller | Stereo vision in structured environments by consistent semi-global matching | |
| CN107424142B (en) | Weld joint identification method based on image significance detection | |
| CN106228507B (en) | A kind of depth image processing method based on light field | |
| CN106127791B (en) | A kind of contour of building line drawing method of aviation remote sensing image | |
| CN105989604A (en) | Target object three-dimensional color point cloud generation method based on KINECT | |
| CN109544606B (en) | Fast automatic registration method and system based on multiple Kinects | |
| CN113177977A (en) | Non-contact three-dimensional human body size measuring method | |
| Wedel et al. | Detection and segmentation of independently moving objects from dense scene flow | |
| Urban et al. | Finding a good feature detector-descriptor combination for the 2D keypoint-based registration of TLS point clouds | |
| CN107369158B (en) | Indoor scene layout estimation and target area extraction method based on RGB-D images | |
| CN110533774B (en) | Three-dimensional model reconstruction method based on smart phone | |
| CN104240264A (en) | Height detection method and device for moving object | |
| CN108182705A (en) | A kind of three-dimensional coordinate localization method based on machine vision | |
| CN103093470A (en) | Rapid multi-modal image synergy segmentation method with unrelated scale feature | |
| CN106295657A (en) | A kind of method extracting human height's feature during video data structure | |
| CN111915517A (en) | Global positioning method for RGB-D camera in indoor illumination adverse environment | |
| CN101488224A (en) | Characteristic point matching method based on relativity measurement | |
| CN111274944A (en) | A 3D Face Reconstruction Method Based on Single Image | |
| CN116630423A (en) | ORB (object oriented analysis) feature-based multi-target binocular positioning method and system for micro robot | |
| CN110910497B (en) | Method and system for realizing augmented reality map | |
| CN115035089B (en) | Brain anatomical structure localization method for two-dimensional brain image data | |
| CN109215122B (en) | A street view three-dimensional reconstruction system and method, intelligent car |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20191224 |