技术领域technical field
本发明属于场景深度估计技术领域，更为具体地讲，涉及一种基于地面线索传播的道路目标深度估计方法。The invention belongs to the technical field of scene depth estimation, and more specifically relates to a road target depth estimation method based on ground clue propagation.
背景技术Background technique
视觉系统中的深度是度量观测物体到成像平面之间的垂直距离，是将从普通传感器设备捕获的二维平面转换为三维立体结构描述的重要的2.5维信息表征。在计算机视觉中，深度估计有助于指导人类进行场景的三维现实模拟，在道路监控设备、机器人导航等领域也具有广泛的应用前景。由于真实道路场景的复杂性，从单一图像中恢复道路目标的深度信息存在多方面的挑战。Depth in the vision system is a measure of the vertical distance between the observed object and the imaging plane, and is an important 2.5-dimensional information representation that converts the two-dimensional plane captured by ordinary sensor equipment into a three-dimensional structure description. In computer vision, depth estimation is helpful to guide humans in 3D realistic simulation of scenes, and it also has broad application prospects in road monitoring equipment, robot navigation and other fields. Due to the complexity of real road scenes, there are multiple challenges in recovering the depth information of road objects from a single image.
单目图像深度信息估计主要依赖于一些特定场景中的单目深度线索，如运动、消失点、遮挡、T连接、阴影、亮度、凸形、纹理梯度等低层线索。Palou等人2015年发表的《Monocular Depth-Ordering Reasoning with Occlusion Edge Detection and CoupleLayers Inference》依据T连接3个区域形成的角度中接近平角对应的区域更趋向于是遮挡区域以及凸形特征更可能出现在前景区域中，来获取物体之间的遮挡关系，并利用获取的遮挡关系构建深度估计图模型进行深度信息全局推理。申请号为201610528752.4的中国专利《基于场景深度映射的道路目标深度估计》利用中心视野消失线构建场景深度映射关系，通过查找深度映射关系表，获得道路目标的深度估计结果。此外，显著性区域检测技术同样也被采用。申请号为201410637107.7的中国专利《一种单幅静态图像深度估计方法及装置》通过对给定图像进行显著性检测，对获得的显著性灰度图确定深度参考点，进而得到深度信息。在实际的道路系统中，利用低层线索进行场景深度估计获得的往往是局部目标间的深度关系，且无法消除遮挡歧义等深度线索歧义性。Monocular image depth information estimation mainly relies on some monocular depth cues in specific scenes, such as low-level cues such as motion, vanishing point, occlusion, T-connection, shadow, brightness, convexity, and texture gradient. "Monocular Depth-Ordering Reasoning with Occlusion Edge Detection and CoupleLayers Inference" published by Palou et al. in 2015, according to the angle formed by T connecting the three regions, the region corresponding to the flat angle is more likely to be an occluded region and convex features are more likely to appear in the foreground In the area, to obtain the occlusion relationship between objects, and use the obtained occlusion relationship to build a depth estimation graph model for global reasoning of depth information. The Chinese patent "Deep Estimation of Road Objects Based on Scene Depth Mapping" with the application number of 201610528752.4 uses the disappearing line of the central field of view to construct the scene depth mapping relationship, and obtains the depth estimation result of road objects by searching the depth mapping relationship table. In addition, salient region detection techniques are also adopted. The Chinese patent "A Single Static Image Depth Estimation Method and Device" with the application number 201410637107.7 determines the depth reference point for the obtained saliency grayscale image by performing saliency detection on a given image, and then obtains depth information. In the actual road system, the scene depth estimation using low-level cues often obtains the depth relationship between local objects, and the ambiguity of depth cues such as occlusion ambiguity cannot be eliminated.
场景布局结构和语义信息是典型的高层信息，同样可以为深度感知提供先验知识。Ji等人2016年发表的《Depth Estimation with Occlusion Modeling Using Light-Field Cameras》基于超像素分割基础上，构建Elastic Conditional Random Field(E-CRF)，对深度和语义之间的相关联系进行建模，E-CRF在环路置信度传播推理中能够自如的进行边界显示或隐藏。Wang等人2016年发表的《Joint Inference of Depth and Labelingfrom a Single Image in Real Time》基于语义信息标记和深度相互促进的框架下，并采用卷积神经网络来训练深度估计模型，联合考虑了全局和局部深度线索进行推理。然而，利用布局结构和语义信息获取的全局深度关系不够精细，往往和实际深度信息不符。Scene layout structure and semantic information are typical high-level information, which can also provide prior knowledge for depth perception. "Depth Estimation with Occlusion Modeling Using Light-Field Cameras" published by Ji et al. in 2016 builds an Elastic Conditional Random Field (E-CRF) based on superpixel segmentation to model the correlation between depth and semantics. E-CRF can freely display or hide boundaries in loop confidence propagation reasoning. "Joint Inference of Depth and Labeling from a Single Image in Real Time" published by Wang et al. in 2016 is based on the framework of semantic information labeling and depth mutual promotion, and uses convolutional neural network to train the depth estimation model, jointly considering the global and Local depth cues for inference. However, the global depth relation obtained by using the layout structure and semantic information is not fine enough and often does not match the actual depth information.
本文针对上述深度估计结果不理想的问题，提出来一种基于地面线索传播的道路目标深度估计。首先，以概率分布方式获取垂直目标的场景深度概率分布，并使用学习的方法检测目标之间的遮挡关系，在此基础上，利用消息传播框架，根据区域间的遮挡关系实现地面线索传播，获取全局的深度信息。Aiming at the problem of unsatisfactory depth estimation results above, this paper proposes a road target depth estimation based on ground clue propagation. Firstly, the scene depth probability distribution of the vertical target is obtained by means of probability distribution, and the occlusion relationship between the targets is detected by using the learning method. Global depth information.
发明内容Contents of the invention
本发明克服了局部遮挡、全局场景线索之间的深度估计不一致等问题，提出了一种基于地面线索传播的道路目标深度估计方法，联合考虑局部遮挡、地面接触与水平消失线的垂直距离获取全局目标间的深度关系。The present invention overcomes the problems of local occlusion and inconsistency in depth estimation among global scene clues, and proposes a road target depth estimation method based on ground clue propagation, which jointly considers local occlusion, ground contact and the vertical distance of the horizontal vanishing line to obtain the global Deep relationships between objects.
本发明采用的技术方案是：The technical scheme adopted in the present invention is:
基于地面线索传播的道路目标深度估计方法，主要包括以下步骤：目标区域合并，分割区域深度提取，边缘遮挡朝向检测器构建，边缘遮挡关系提取，目标深度估计，深度稳定性判断。The road target depth estimation method based on ground clue propagation mainly includes the following steps: target area merging, segmented area depth extraction, edge occlusion orientation detector construction, edge occlusion relationship extraction, target depth estimation, and depth stability judgment.
所述的目标区域合并，对输入道路图像，提取过分割超像素和超像素平均RGB颜色特征，依次遍历，根据RGB平均颜色进行区域合并。The target region merging is to extract the over-segmented superpixels and the average RGB color features of the superpixels from the input road image, traverse them in turn, and perform region merging according to the RGB average color.
所述的分割区域深度提取，对道路图像进行地面、天空、垂直物体区域检测，再对边缘图像进行hough直线检测，计算两两直线的交叉点，将投票数最高的交叉点作为地面消失点；然后，对垂直物体区域，分别求解垂直物体区域的场景深度均值和方差，构建高斯形式的概率密度函数，作为该分割区域的场景深度估计。In the extraction of the depth of the segmented area, the ground, sky, and vertical object areas are detected on the road image, and the hough line detection is performed on the edge image, and the intersection point of two straight lines is calculated, and the intersection point with the highest number of votes is used as the ground vanishing point; Then, for the vertical object area, the mean and variance of the scene depth of the vertical object area are respectively calculated, and a Gaussian probability density function is constructed as the scene depth estimation of the segmented area.
所述的边缘遮挡朝向检测器构建，针对道路图像的分割区域，提取邻接关系矩阵，提取两个邻接区域的共享边界，对共享边界区域正则化，并进行共享边界朝向标记；再构建边缘遮挡朝向检测器，根据支持向量机，训练边缘遮挡朝向检测器模型。The construction of the edge occlusion orientation detector is to extract the adjacency matrix for the segmented area of the road image, extract the shared boundary of two adjacent areas, regularize the shared boundary area, and mark the shared boundary orientation; then construct the edge occlusion orientation The detector, according to the support vector machine, trains the edge occlusion orientation detector model.
所述的边缘遮挡关系提取，根据道路图像中所有的共享边界像素个数，完成对共享边界的多次等间隔采样，并完成多次前景区域投票，投票数较大的分割区域遮挡投票数较小的分割区域，记录遮挡关系元素，获得所有共享边界两侧区域对之间的边缘线索的遮挡关系矩阵。The extraction of the edge occlusion relationship, according to the number of shared boundary pixels in the road image, completes multiple equal-spaced sampling of the shared boundary, and completes multiple foreground area votes. Small segmented regions, record the occlusion relation elements, and obtain the occlusion relation matrix of edge cues between all pairs of regions on both sides of the shared boundary.
所述的目标深度估计，对所有垂直目标间共享边界，如果地面线索和边缘线索得到的遮挡关系相关矛盾，则对区域的地面线索的高斯密度函数进行更新，获得未归一化的分布函数，重新归一化非高斯密度函数，并更新均值和方差，更新区域的地面线索的高斯密度函数，完成一次地面线索传播的更新过程。In the target depth estimation, for all vertical targets sharing boundaries, if the occlusion relationship obtained by ground cues and edge cues is inconsistent, the Gaussian density function of the ground cues in the area is updated to obtain an unnormalized distribution function, Renormalize the non-Gaussian density function, update the mean and variance, update the Gaussian density function of the ground clues in the area, and complete an update process of ground clue propagation.
所述的深度稳定性判断，对地面线索传播进行迭代推理，判断深度是否稳定，如果不是所有垂直目标区域稳定，则重复执行地面线索传播的更新过程，直到所有垂直目标区域稳定，将收敛状态的高斯密度函数均值作为垂直目标区域的目标深度估计值。In the depth stability judgment, iterative reasoning is performed on ground clue propagation to determine whether the depth is stable. If not all vertical target areas are stable, the update process of ground clue propagation is repeated until all vertical target areas are stable, and the convergence state The mean of the Gaussian density function is used as the target depth estimate for the vertical target area.
下面详细描述基于地面线索传播的道路目标深度估计的各个步骤。The steps of road target depth estimation based on ground clue propagation are described in detail below.
所述的目标区域合并，包括以下步骤：The merging of the target areas comprises the following steps:
步骤S1：对输入道路图像，提取过分割超像素，合并生成目标区域，具体操作步骤包括：Step S1: For the input road image, extract over-segmented superpixels and combine them to generate the target area. The specific operation steps include:
步骤S1-1：对输入图像，使用分水岭算法，获得过分割超像素集合SP，记作SP＝{spi},i＝1,...,nsp，其中，spi是编号为i的超像素，nsp是道路图像中超像素的个数；Step S1-1: For the input image, use the watershed algorithm to obtain the over-segmented superpixel set SP, denoted as SP={spi }, i=1,...,nsp , where spi is the number i Superpixels, nsp is the number of superpixels in the road image;
步骤S1-2：计算所有超像素的平均RGB颜色，提取所有超像素的平均RGB颜色C＝{ci},i＝1,...,nsp；Step S1-2: Calculate the average RGB color of all superpixels, and extract the average RGB color C={ci },i=1,...,nsp of all superpixels;
步骤S1-3：将所有超像素，标记为未遍历状态，记录未标记超像素集合Vset＝{spi},i＝1,...,nsp；Step S1-3: Mark all superpixels as untraversed state, and record the unmarked superpixel set Vset={spi },i=1,...,nsp ;
步骤S1-4：对所有超像素，初始化超像素合并编号为labeli＝i，获得超像素合并编号集合Label＝{labeli},i＝1,...,nsp；Step S1-4: For all superpixels, initialize the superpixel merging number as labeli =i, and obtain the superpixel merging number set Label={labeli }, i=1,...,nsp ;
步骤S1-5：如果未标记超像素集合Vset不为空，选择遍历超像素spj，对遍历超像素和周围超像素，进行区域合并；Step S1-5: If the unmarked superpixel set Vset is not empty, select the traversal superpixel spj , and perform region merging on the traversal superpixel and surrounding superpixels;
步骤S1-5-1：从未遍历的超像素中，选择遍历超像素spj，并将该超像素标记为已遍历状态vj＝0；Step S1-5-1: Select a traversed superpixel spj from the untraversed superpixels, and mark this superpixel as traversed state vj =0;
步骤S1-5-2：从未标记超像素集合Vset去除遍历超像素spj编号，更新未标记超像素集合Vset；Step S1-5-2: remove the traversed superpixel spj number from the unmarked superpixel set Vset, and update the unmarked superpixel set Vset;
步骤S1-5-3：对遍历像素spj，统计其周围的超像素编号，获得邻接超像素编号，找出其中未遍历的邻接超像素编号neighj；Step S1-5-3: For the traversable pixel spj , count the superpixel numbers around it, obtain the adjacent superpixel numbers, and find out the adjacent superpixel numbers neighj that have not been traversed;
步骤S1-5-4：如果未遍历的邻接超像素编号neighj为空集合，则停止对遍历超像素spj的遍历；Step S1-5-4: If the untraversed adjacent superpixel number neighj is an empty set, stop traversing the traversed superpixel spj ;
步骤S1-5-5：如果未遍历的邻接超像素编号neighj不为空集合，则选择一个其中未遍历的邻接超像素spk；Step S1-5-5: If the untraversed adjacent superpixel number neighj is not an empty set, select an untraversed adjacent superpixel spk among them;
步骤S1-5-6：根据RGB平均颜色集合C＝{ci},i＝1,...,nsp，使用2范数，计算遍历像素spj和邻接超像素spk的颜色距离Step S1-5-6: Calculate the color distance between the traversed pixel spj and the adjacent superpixel spk according to the RGB average color set C={ci },i=1,...,nsp , using the 2-norm
djk＝||ci-ck||2djk ＝||ci -ck ||2
步骤S1-5-7：如果遍历像素spj和邻接超像素spk的颜色距离djk＝||ci-ck||2≥T，其中T为合并阈值，则不进行区域合并；Step S1-5-7: If the color distance djk = ||ci -ck ||2≥ T between the traversed pixel spj and the adjacent superpixel spk , where T is the merging threshold, region merging is not performed;
步骤S1-5-8：如果遍历像素spj和邻接超像素spk的颜色距离，djk＝||ci-ck||2<T，其中T为合并阈值，则进行区域合并；合并方式为：将遍历超像素spj编号，赋值给邻接超像素spk编号，具体操作方式为：Step S1-5-8: If the color distance between the traversal pixel spj and the adjacent superpixel spk , djk = ||ci -c k||2 <T, where T is the merge threshold, perform region merge; merge The method is: assign the number of the traversed superpixel spj to the number of the adjacent superpixel spk . The specific operation method is:
labelk＝labeljlabelk = labelj
步骤S1-5-9：将超像素spk，从未遍历的超像素编号neighj去除，获得更新后的未遍历的邻接超像素编号neighj；Step S1-5-9: remove the superpixel spk from the untraversed superpixel number neighj to obtain the updated untraversed adjacent superpixel number neighj ;
步骤S1-5-10：如果未遍历的邻接超像素编号neighj不为空集合，重复步骤S1-5-4到步骤S1-5-10，直到完成对邻接超像素编号neighj的遍历；Step S1-5-10: If the untraversed adjacent superpixel number neighj is not an empty set, repeat steps S1-5-4 to step S1-5-10 until the traversal of the adjacent superpixel number neighj is completed;
步骤S1-6：如果未标记超像素集合Vset不为空，则重复执行步骤S1-5，直到完成所有超像素的遍历，最终获得合并后的超像素合并编号Label＝{labeli},i＝1,...,nsp；Step S1-6: If the unlabeled superpixel set Vset is not empty, repeat step S1-5 until the traversal of all superpixels is completed, and finally obtain the merged superpixel number Label={labeli }, i= 1,...,nsp ;
步骤S1-7：根据合并后的超像素合并编号Label＝{labeli},i＝1,...,nsp，将相同编号的超像素进行合并，获得目标区域集合R，记作R＝{ri},i＝1,...,nr，其中，ri是编号为i的目标区域，nr是道路图像中目标的个数。Step S1-7: According to the merged superpixel number Label={labeli },i=1,...,nsp , merge the superpixels with the same number to obtain the target region set R, denoted as R= {ri },i=1,...,nr , where ri is the target area numbered i, and nr is the number of targets in the road image.
所述的分割区域深度提取，包括以下步骤：The extraction of the depth of the segmented region comprises the following steps:
步骤S2-1：采集图像块数据，构建图像表面检测器，获取图像表面标记；Step S2-1: collecting image block data, constructing an image surface detector, and obtaining image surface markers;
步骤S2-1-1：提取分割区域的平均颜色特征xi,i＝1,...,nr；Step S2-1-1: Extracting the average color feature xi of the segmented area, i=1,...,nr ;
步骤S2-1-2：使用地面检测器模型进行地面区域判断如果ygi＝1，则该区域标记为地面yli＝groud；如果ygi＝0，则不更新该区域标记；Step S2-1-2: Use the ground detector model Make ground area judgments If ygi =1, the area is marked as ground yli =groud; if ygi =0, the area mark is not updated;
步骤S2-1-3：使用天空检测器模型进行天空区域判断如果ysi＝1，则该区域标记为天空yli＝sky；如果ysi＝0，则不更新该区域标记；Step S2-1-3: Use the sky detector model Make Sky Region Judgment If ysi =1, the area is marked as sky yli =sky; if ysi =0, the area mark is not updated;
步骤S2-1-4：重复步骤步骤S2-1-2和步骤步骤S2-1-3，对所有分割区域进行标记更新，将非地面、天空区域记做垂直物体区域，获得地面、天空、垂直物体区域检测的标记集合，作为图像表面标记集合，记作YL＝{yli},i＝1,...,nr；Step S2-1-4: Repeat step S2-1-2 and step S2-1-3 to update all the segmented regions, record the non-ground and sky regions as vertical object regions, and obtain the ground, sky, vertical The mark set of object region detection, as the image surface mark set, is recorded as YL={yli }, i=1,...,nr ;
步骤S2-1-5：根据图像表面标记集合YL＝{yli},i＝1,...,nr，对道路图像各像素进行图像表面标记赋值，具体方式为，查找图像分割区域ri，对该区域中的所有像素，赋值对应的表面标记集合yli，最终获得表面标记图像Isurf；Step S2-1-5: According to the image surface label set YL={yli },i=1,...,nr , assign the image surface label value to each pixel of the road image. The specific method is to search for the image segmentation region ri , assign the corresponding surface marker set yli to all the pixels in the area, and finally obtain the surface marker image Isurf ;
步骤S2-2：根据道路图像，提取地面消失点，具体操作步骤包括：Step S2-2: According to the road image, extract the ground vanishing point, the specific operation steps include:
步骤S2-2-1：对道路图像进行hough直线检测，获得直线集合H＝{hi},i＝1,...,nh，其中hi是编号为i的直线，nh是道路图像中直线的个数；Step S2-2-1: Perform hough straight line detection on the road image to obtain a set of straight lines H={hi },i=1,...,nh , where hi is the straight line numbered i, and nh is the road The number of straight lines in the image;
步骤S2-2-2：根据直线hj和直线hk的方程，计算直线hj和直线hk的交叉点，依次遍历直线集合H＝{hi},i＝1,...,nh，计算两两直线的交叉点，获得交叉点集合PT＝{ptjk},j＝1,...,nh,k＝1,...,nh,j≠k；Step S2-2-2: According to the equations of straight line hj and straight line hk , calculate the intersection point of straight line hj and straight line hk , and traverse the set of straight lines H={hi }, i=1,...,n in sequenceh , calculate the intersection point of two straight lines, and obtain the intersection point set PT={ptjk },j=1,...,nh ,k=1,...,nh ,j≠k;
步骤S2-2-3：在交叉点图像中，选择以交叉点ptjk为圆心，半径为rad的圆形区域Circlejk，将圆形区域Circlejk内出现的交叉点个数，作为交叉点作为地面消失点的投票数votejk；Step S2-2-3: In the intersection point image, select the circle area Circlejk with the intersection point ptjk as the center and radius rad, and use the number of intersection points appearing in the circle area Circlejk as the intersection point as The number of votes at the ground vanishing point votejk ;
步骤S2-2-4：根据步骤S2-2-3，计算所有交叉点的投票数，将投票数最高的交叉点，作为道路图像的地面消失点vp＝{vpx,vpy}，其中，vpx是消失点的水平方向的坐标，vpy是消失点垂直方向的坐标；Step S2-2-4: According to step S2-2-3, calculate the number of votes for all intersections, and use the intersection with the highest number of votes as the ground vanishing point of the road image vp={vpx,vpy}, where vpx is The horizontal coordinates of the vanishing point, vpy is the vertical coordinates of the vanishing point;
步骤S2-3：对道路图像的垂直物体区域，利用高斯概率分布估计，提取分割区域深度；Step S2-3: For the vertical object area of the road image, the Gaussian probability distribution is used to estimate the depth of the segmented area;
步骤S2-3-1：根据图像表面标记集合YL＝{yli},i＝1,...,nr，查找其中地面标记的区域，获取所有地面区域Rg；Step S2-3-1: According to the image surface marker set YL={yli }, i=1,...,nr , find the area marked by the ground, and obtain all the ground areas Rg;
步骤S2-3-2：对地面区域Rg进行轮廓分析，获得地面区域边界Bg；Step S2-3-2: Perform contour analysis on the ground area Rg to obtain the ground area boundary Bg;
步骤S2-3-3：对地面区域边界Bg，找出第kv列中最低的地面区域边界像素gpkv＝{kv,gpykv}，其中，kv表示地面区域边界像素点的水平方向的坐标，即第kv列，gpykv是地面区域边界像素点垂直方向的坐标；Step S2-3-3: For the ground area boundary Bg, find out the lowest ground area boundary pixel gpkv ={kv, gpykv } in the kv column, where kv represents the horizontal direction coordinates of the ground area boundary pixel point, That is, the kv column, gpykv is the coordinate in the vertical direction of the boundary pixel of the ground area;
步骤S2-3-4：估计第kv列的最小场景深度depthkv，具体方式为：Step S2-3-4: Estimate the minimum scene depth depthkv of the kv-th column, the specific method is:
其中，A是摄像机高度和摄像机焦距的乘积，gpykv是地面区域边界像素点垂直方向的坐标，vpy是消失点垂直方向的坐标；Among them, A is the product of the height of the camera and the focal length of the camera, gpykv is the coordinate of the vertical direction of the boundary pixel of the ground area, and vpy is the coordinate of the vertical direction of the vanishing point;
步骤S2-3-5：表面标记图像Isurf，查找出第kv列中垂直表面标记的像素，并赋值第kv列的最小场景深度depthkv；Step S2-3-5: surface mark image Isurf , find out the pixels of the vertical surface mark in column kv, and assign the minimum scene depth depth kv in columnkv ;
步骤S2-3-6：重复步骤S2-3-4到步骤S2-3-5，依次计算每列的最小场景深度depthkv，并赋值给对应列的垂直表面标记的像素，获得道路图像的列深度图像Dvertical；Step S2-3-6: Repeat step S2-3-4 to step S2-3-5, calculate the minimum scene depth depthkv of each column in turn, and assign it to the pixel of the vertical surface mark of the corresponding column, and obtain the column of the road image Depth image Dvertical ;
步骤S2-3-7：根据图像表面标记集合YL＝{yli},i＝1,...,nr，找出第i个垂直物体区域的所有像素集合；Step S2-3-7: According to the image surface label set YL={yli }, i=1,...,nr , find out all the pixel sets of the i-th vertical object region;
步骤S2-3-8：根据道路图像的列深度图像Dvertical，提取第i个垂直物体区域的所有像素集合对应的场景深度，获得第i个垂直物体区域的场景深度值集合dseti,i＝1,...,nr；Step S2-3-8: According to the column depth image Dvertical of the road image, extract the scene depth corresponding to all pixel sets of the i-th vertical object area, and obtain the scene depth value set dseti of the i-th vertical object area, i= 1,...,nr ;
步骤S2-3-9：对第i个垂直物体区域的场景深度值集合dseti,i＝1,...,nr，求解第i个垂直物体区域的场景深度均值ui和方差σi，其中i＝1,...,nr表示区域编号；Step S2-3-9: For the scene depth value set dseti of the i-th vertical object area, i=1,...,nr , calculate the scene depth mean ui and variance σi of the i-th vertical object area , where i=1,...,nr represents the area number;
步骤S2-3-10：根据第i个垂直物体区域的场景深度均值ui和方差σi，构建高斯形式的概率密度函数，作为该分割区域的场景深度估计Step S2-3-10: According to the scene depth mean ui and variance σi of the i-th vertical object area, construct a Gaussian probability density function as the scene depth estimation of the segmented area
pdfi＝Gaussian(ui,σi)pdfi ＝Gaussian(ui ,σi )
其中i＝1,...,nr，Gaussian(ui,σi)表示以ui为均值，以σi为方差的高斯概率密度函数；Where i=1,...,nr , Gaussian(ui ,σi ) represents the Gaussian probability density function with ui as the mean and σi as the variance;
步骤S2-3-11：重复步骤S2-3-8到步骤S2-3-10，提取所有垂直物体区域的场景深度均值ui和方差σi，以及其对应的场景深度估计pdfi＝Gaussian(ui,σi),i＝1,...,nr。Step S2-3-11: Repeat steps S2-3-8 to S2-3-10 to extract the scene depth mean value ui and variance σi of all vertical object regions, and the corresponding scene depth estimation pdfi =Gaussian( ui ,σi ), i=1,...,nr .
所述的边缘遮挡朝向检测器构建，包括以下步骤：The construction of the edge occlusion towards the detector comprises the following steps:
步骤S3-1：针对道路图像的分割区域，根据邻接关系矩阵，提取分割区域间的共享边界；Step S3-1: For the segmented area of the road image, extract the shared boundary between the segmented areas according to the adjacency matrix;
步骤S3-1-1：道路图像的分割区域R＝{ri},i＝1,...,nr，提取邻接关系矩阵，记录邻接关系矩阵Adjacent＝{adjj,k}，其中将邻接关系矩阵中的元素赋值为adjj,k＝1，表示分割区域ri和分割区域rk存在邻接关系；Step S3-1-1: In the segmented area R={ri },i=1,...,nr of the road image, extract the adjacency matrix and record the adjacency matrix Adjacent={adjj,k }, where The elements in the adjacency matrix are assigned adjj,k = 1, which means that there is an adjacency relationship between the segmented region ri and the segmented region rk ;
步骤S3-1-2：根据邻接关系矩阵Adjacent＝{adjj,k}，找出两个邻接区域，分割区域ri和分割区域rk，将两个区域之间的共享边界，记为道路图像中所有的共享边界Bound＝{boundj,k}，其中，j＝1,...,nh,k＝1,...,nh,j≠k；Step S3-1-2: According to the adjacency matrix Adjacent={adjj,k }, find two adjacent regions, split region ri and split region rk , record the shared boundary between the two regions as a road All shared boundaries Bound={boundj,k } in the image, where j=1,...,nh ,k=1,...,nh ,j≠k;
步骤S3-1-3：根据道路图像中所有的共享边界Bound＝{boundj,k}，将共享边界对应像素标记为边界，获得共享边界图像Ibound；Step S3-1-3: According to all shared boundaries Bound={boundj,k } in the road image, mark the pixels corresponding to the shared boundaries as boundaries, and obtain the shared boundary image Ibound ;
步骤S3-2：对共享边界区域正则化，并进行共享边界朝向标记；Step S3-2: Regularize the shared boundary area and mark the shared boundary orientation;
步骤S3-2-1：根据道路图像中所有的共享边界Bound＝{boundj,k}，选择一条共享边界boundj,k；Step S3-2-1: Select a shared boundary boundj,k according to all shared boundaries Bound={boundj,k } in the road image;
步骤S3-2-2：根据共享边界boundj,k像素下标，选择第m个像素作为参考像素prefm；Step S3-2-2: Select the mth pixel as the reference pixel prefm according to the shared boundary boundj,k pixel index;
步骤S3-2-3：计算参考像素prefm处的切线，将切线和图像的水平线夹角，作为共享边界的切线方向θrefm；Step S3-2-3: Calculate the tangent at the reference pixel prefm , and use the angle between the tangent and the horizontal line of the image as the tangent direction θrefm of the shared boundary;
步骤S3-2-4：以参考像素prefm为中心旋转，将共享边界图像Ibound，旋转θrefm角度，获得旋转后图像，以保证旋转后的图像中，参考像素prefm处的切线为水平方向；Step S3-2-4: Rotate around the reference pixel prefm , rotate the shared boundary image Ibound by the angle θrefm , and obtain the rotated image, so as to ensure that the tangent at the reference pixel prefm in the rotated image is horizontal direction;
步骤S3-2-5：在旋转后图像中，以共享边界的参考像素prefm，以半径为15，提取参考像素的图像块，作为正则参考像素图像块patchm；Step S3-2-5: In the rotated image, take the reference pixel prefm sharing the boundary and take the radius of 15 to extract the image block of the reference pixel as the regular reference pixel image block patchm ;
步骤S3-2-6：对正则参考像素图像块patchm进行边缘遮挡朝向标记，具体操作为，如果正则参考像素图像块patchm中下方目标遮挡上方目标，则遮挡标记，记为yom＝1；如果正则参考像素图像块patchm中上方目标遮挡下方目标，则遮挡标记，记为yom＝0；Step S3-2-6: mark the edge occlusion direction of the regular reference pixel image block patchm , the specific operation is, if the lower object in the regular reference pixel image block patchm blocks the upper object, then the occlusion mark is recorded as yom = 1 ; If the upper object in the regular reference pixel image block patchm blocks the lower object, then the occlusion mark is recorded as yom = 0;
步骤S3-3：构建边缘遮挡朝向检测器，训练边缘遮挡朝向检测器模型；Step S3-3: building an edge occlusion orientation detector, and training an edge occlusion orientation detector model;
步骤S3-3-1：重复步骤S3-2-1到步骤S3-2-5，可以对不同共享边界上进行多次采样，获得多个遮挡朝向样本，作为遮挡朝向的训练样本集合；Step S3-3-1: Repeat steps S3-2-1 to S3-2-5, multiple samplings can be performed on different shared boundaries, and multiple occlusion orientation samples can be obtained as a training sample set of occlusion orientations;
步骤S3-3-2：对遮挡朝向的训练样本集合，重复步骤S3-2-6，获得集合中各样本的遮挡朝向标记yogt；Step S3-3-2: For the training sample set of occlusion orientation, repeat step S3-2-6 to obtain the occlusion orientation mark yogt of each sample in the set;
步骤S3-3-3：对遮挡朝向样本的图像块，依据行遍历方式，转化为边界行向量特征xo；Step S3-3-3: For the image block occluded towards the sample, convert it into the boundary row vector feature xo according to the row traversal method;
步骤S3-3-4：构建边缘遮挡朝向检测器，预测待检测区域的边缘遮挡朝向标记yotest，具体方式为Step S3-3-4: Build an edge occlusion orientation detector to predict the edge occlusion orientation mark yotest of the area to be detected, the specific method is
yotest＝wo·xotestyotest = wo xotest
其中，xotest是待检测区域的边界行向量，wo是边缘遮挡朝向检测器的参数；Among them, xotest is the boundary row vector of the area to be detected, and wo is the parameter of the edge occlusion orientation detector;
步骤S3-3-5：根据支持向量机，设计边缘遮挡朝向检测器目标函数Step S3-3-5: According to the support vector machine, design the objective function of the edge occlusion orientation detector
步骤S3-3-6：根据支持向量机，训练边缘遮挡朝向检测器模型，求解出最佳的边缘遮挡朝向检测器参数具体方式为Step S3-3-6: According to the support vector machine, train the edge occlusion orientation detector model, and solve the best edge occlusion orientation detector parameters The specific method is
所述的边缘遮挡关系提取，包括以下步骤：The extraction of the edge occlusion relation comprises the following steps:
步骤S4-1：初始化图像中所有区域的边缘线索遮挡关系矩阵Occ＝{occj,k}，其中，j＝1,...,nh,k＝1,...,nh,j≠k，其中初始值为occj,k＝0，表示分割区域j和分割区域k无遮挡关系；Step S4-1: Initialize the edge clue occlusion relationship matrix Occ={occj,k } of all regions in the image, where j=1,...,nh ,k=1,...,nh ,j ≠ k, where the initial value is occj, k = 0, indicating that there is no occlusion relationship between the segmented area j and the segmented area k;
步骤S4-2：根据道路图像中所有的共享边界Bound＝{boundj,k}，选择一条共享边界boundj,k；Step S4-2: Select a shared boundary boundj,k according to all shared boundaries Bound={boundj,k } in the road image;
步骤S4-3：对共享边界两侧的分割区域j和分割区域k，初始化分割区域前景投票向量Votej,k＝{votej,votek}，其中初始值为votej＝0，votek＝0；Step S4-3: For the segmented area j and segmented area k on both sides of the shared boundary, initialize the segmented area foreground voting vector Votej,k ={votej ,votek }, where the initial value is votej =0, votek = 0;
步骤S4-4：根据共享边界boundj,k像素个数，等间隔采样，选择候选像素集合Pref＝{prefm}；Step S4-4: According to the number of shared boundary boundj,k pixels, sampling at equal intervals, select the candidate pixel set Pref={prefm };
步骤S4-5：从候选像素集合Pref＝{prefm}中选择一个候选像素prefm，重复步骤S3-2-3到步骤S3-2-5，获得候选像素的正则图像块patchm；Step S4-5: Select a candidate pixel prefm from the candidate pixel set Pref={prefm }, repeat step S3-2-3 to step S3-2-5, and obtain the regular image block patchm of the candidate pixel;
步骤S4-6：根据原始共享边界两侧的分割区域j和分割区域k，记录候选像素的正则图像块patchm的上下区域编号，上方区域分割编号为j'，下方区域分割编号为k'；Step S4-6: According to the segmentation area j and the segmentation area k on both sides of the original shared boundary, record the upper and lower area numbers of the regular image block patchm of the candidate pixel, the upper area segmentation number is j', and the lower area segmentation number is k';
步骤S4-7：对候选像素的正则图像块patchm，依据行遍历方式，转化为边界行向量特征xotest；Step S4-7: For the regular image block patchm of the candidate pixel, convert it into the boundary row vector feature xotest according to the row traversal method;
步骤S4-8：使用训练完成的边缘遮挡朝向检测器模型对正则图像块判断Step S4-8: Use the trained edge occlusion orientation detector model Judgment on regular image blocks
步骤S4-9：如果yoi＝1，表示下方目标k'遮挡上方目标j'，即下方目标k'为靠近摄像机的前景区域，则对下方目标区域的前景进行一次投票Step S4-9: If yoi = 1, it means that the lower object k' blocks the upper object j', that is, the lower object k' is the foreground area close to the camera, then conduct a vote on the foreground of the lower object area
votek'＝votek'+1votek' = votek' +1
步骤S4-10：如果yoi＝0，表示上方目标j'遮挡下方目标k'，即上方目标j'为靠近摄像机的前景区域，则对上方目标区域的前景进行一次投票Step S4-10: If yoi = 0, it means that the upper object j' blocks the lower object k', that is, the upper object j' is the foreground area close to the camera, then perform a vote on the foreground of the upper object area
votej'＝votej'+1votej' = votej' +1
步骤S4-11：重复步骤S4-4到步骤S4-10，完成对共享边界boundj,k的多次等间隔采样，并完成多次前景区域投票，获得分割区域前景投票向量Votej,k＝{votej,votek}；Step S4-11: Repeat Step S4-4 to Step S4-10 to complete multiple equal-spaced sampling of the shared boundary boundj,k , and complete multiple foreground area voting, and obtain the divided area foreground voting vector Votej,k = { votej , votek };
步骤S4-12：比较投票数，投票数较大的分割区域遮挡投票数较小的分割区域，记录遮挡关系元素occj,k；如果分割区域j遮挡分割区域k，则记录occj,k＝1；如果分割区域k遮挡分割区域j，则记录occj,k＝-1；Step S4-12: Compare the number of votes, the segmented area with a larger number of votes covers the segmented area with a smaller number of votes, and record the occlusion relationship element occj,k ; if the segmented area j blocks the segmented area k, then record occj,k = 1; if segmented area k blocks segmented area j, then record occj,k =-1;
步骤S4-13：重复步骤S4-2到步骤S4-12，完成对图像中所有共享边界的分析，获得所有共享边界两侧区域对之间的边缘线索的遮挡关系矩阵Occ＝{occj,k}。Step S4-13: Repeat Step S4-2 to Step S4-12 to complete the analysis of all shared boundaries in the image, and obtain the occlusion relationship matrix Occ={occj,k }.
所述的目标深度估计，包括以下步骤：The target depth estimation includes the following steps:
步骤S5-1：根据根据图像表面标记集合YL＝{yli},i＝1,...,nr，获取垂直目标区域的编号集合VYL＝{vi}，其中ylvi＝vertical,vi∈{1,...,nr}；Step S5-1: According to the image surface label set YL={yli },i=1,...,nr , obtain the number set VYL={vi} of the vertical target area, where ylvi =vertical,vi∈ {1,...,nr };
步骤S5-2：垂直目标区域的编号集合VYL＝{vi}，对共享边界Bound＝{boundj,k}进行筛选，找出共享边界两侧都是垂直目标区域的垂直目标间共享边界GBound＝{gboundj,k}，其中，j,k∈VYL；Step S5-2: The number set VYL={vi} of the vertical target area, screen the shared boundary Bound={boundj, k }, and find out the shared boundary GBound= between vertical targets with vertical target areas on both sides of the shared boundary {gboundj,k }, where j,k∈VYL;
步骤S5-3：对垂直目标间共享边界GBound＝{gboundj,k}，其中j,k∈VYL，选择一条共享边界gboundj,k进行地面线索传递；Step S5-3: For the shared boundary GBound={gboundj,k } between vertical targets, where j,k∈VYL, select a shared boundary gboundj,k for ground clue transmission;
步骤S5-3-1：根据共享边界gboundj,k，其中j,k∈VYL，获取共享边界两侧区域编号j,k；Step S5-3-1: According to the shared boundary gboundj,k , where j,k∈VYL, obtain the area numbers j,k on both sides of the shared boundary;
步骤S5-3-2：针对共享边界两侧区域编号j,k，获取其地面线索高斯密度函数均值，其中，区域编号j的地面线索高斯密度函数均值为uj；区域编号k的地面线索高斯密度函数均值为uk；Step S5-3-2: For the area numbers j and k on both sides of the shared boundary, obtain the mean value of the Gaussian density function of the ground clues, where the mean value of the Gaussian density function of the ground clues of the area number j is uj ; the Gaussian density function of the ground clues of the area number k is u j ; The mean value of the density function is uk ;
步骤S5-3-3：根据共享边界两侧区域的地面线索高斯密度函数均值，记录区域间的遮挡关系，具体操作为：如果区域编号j的均值uj小于区域编号k的均值uk，即uj<uk，说明区域编号k距离摄像头更远，区域编号k被区域编号j遮挡，记录地面线索遮挡关系为goccj,k＝1；如果区域编号j的均值uj大于区域编号k的均值uk，即uj>uk，说明区域编号j距离摄像头更远，区域编号j被区域编号k遮挡，记录地面线索遮挡关系为goccj,k＝-1；如果区域编号j的均值uj等于区域编号k的均值uk，即uj＝uk，说明区域编号k与区域编号j无遮挡关系，记录地面线索遮挡关系为goccj,k＝0；Step S5-3-3: According to the mean value of the ground clue Gaussian density function of the areas on both sides of the shared boundary, record the occlusion relationship between the areas. The specific operation is: if the mean value uj of the area number j is smaller than the mean value uk of the area number k, that is uj < uk , indicating that the area number k is farther away from the camera, and the area number k is blocked by the area number j, and the occlusion relation of the ground clue is recorded as goccj,k = 1; if the mean value uj of the area number j is greater than that of the area number k The mean value uk , that is, uj >uk , indicates that the area number j is farther away from the camera, and the area number j is blocked by the area number k, and the occlusion relation of recorded ground clues is goccj,k = -1; if the mean value u of the area number jj is equal to the average value uk of area number k, that is, uj =uk , indicating that there is no occlusion relationship between area number k and area number j, and the occlusion relationship of ground clues is recorded as goccj,k =0;
步骤S5-3-4：根据共享边界两侧区域编号j,k，记录边缘线索的遮挡关系occj,k；Step S5-3-4: According to the area numbers j, k on both sides of the shared border, record the occlusion relationship occj, k of the edge clue;
步骤S5-3-5：根据地面线索遮挡关系goccj,k，以及边缘线索的遮挡关系occj,k，如果occj,k·goccj,k＝-1，说明地面线索和边缘线索得到的遮挡关系相关矛盾，则对区域编号j的地面线索的高斯密度函数pdfj＝Gaussian(uj,σj)进行更新，获得未归一化的分布函数unpdfj，具体方式为：Step S5-3-5: According to the occlusion relationship goccj,kof the ground clues and the occlusion relationship occ j,k of the edge cues, if occ j,k ·goccj,k = -1, it means that the ground clues and the edge cues are obtained If there is a contradiction related to the occlusion relationship, update the Gaussian density function pdfj =Gaussian(uj ,σj ) of the ground clue of area number j to obtain the unnormalized distribution function unpdfj , the specific method is as follows:
unpdfj＝pdfj-pdfkunpdfj =pdfj -pdfk
步骤S5-3-6：计算未归一化的分布函数unpdfj中的最小值minun，和最大值maxun；Step S5-3-6: Calculating the minimum value minun and the maximum value maxun in the unnormalized distribution function unpdfj ;
步骤S5-3-7：根据未归一化的分布函数unpdfj中的最小值minun，和最大值maxun，重新归一化区域编号j的地面线索的非高斯密度函数updfj，重新归一化的公式为：Step S5-3-7: According to the minimum value minun and the maximum value maxun in the unnormalized distribution function unpdfj , renormalize the non-Gaussian density function updfj of the ground clue of the area number j, and renormalize The formula is:
步骤S5-3-8,：根据非高斯密度函数updfj，重新计算并更新非高斯密度函数updfj的均值和方差Step S5-3-8,: According to the non-Gaussian density function updfj , recalculate and update the mean value of the non-Gaussian density function updfj and variance
步骤S5-3-9：根据非高斯密度函数updfj的均值和方差重新估计并更新区域编号j的地面线索的高斯密度函数Step S5-3-9: According to the mean value of the non-Gaussian density function updfj and variance Re-estimate and update the Gaussian density function of the ground cues for area number j
步骤S5-3-10：根据地面线索遮挡关系goccj,k，以及边缘线索的遮挡关系occj,k，如果occj,k·goccj,k＝-1，说明地面线索和边缘线索得到的遮挡关系相关矛盾，对区域编号k的地面线索的高斯密度函数进行更新，获得未归一化的分布函数unpdfj，具体方式为：Step S5-3-10: According to the occlusion relationship goccj,kof the ground clues and the occlusion relationship occ j,k of the edge cues, if occ j,k ·goccj,k = -1, it means that the ground clues and the edge cues are obtained The occlusion relationship is related to the contradiction, and the Gaussian density function of the ground clue of the area number k is updated to obtain the unnormalized distribution function unpdfj , the specific method is:
unpdfk＝pdfk-pdfjunpdfk =pdfk -pdfj
步骤S5-3-11：重复步骤S5-3-6到步骤5-3-9，重新归一化密度函数，并计算并更新均值和方差获得更新后的高斯密度函数Step S5-3-11: Repeat Step S5-3-6 to Step 5-3-9, renormalize the density function, and calculate and update the mean and variance Get the updated Gaussian density function
步骤S5-4：重复步骤S5-3，对所有垂直目标间共享边界GBound＝{gboundj,k}，进行处理更新，更新所有垂直目标区域的高斯密度函数，完成一次地面线索传播的更新过程。Step S5-4: Repeat step S5-3, update the shared boundary GBound={gboundj,k } between all vertical targets, update the Gaussian density function of all vertical target areas, and complete an update process of ground clue propagation.
所述的深度稳定性判断，包括以下步骤：The described depth stability judgment includes the following steps:
步骤S6-1：获得步骤S3-1中的原始地面线索高斯密度函数均值为uj，和更新后的地面线索高斯密度函数均值为Step S6-1: Obtain the mean value of the Gaussian density function of the original ground clues in step S3-1 as uj , and the mean value of the updated Gaussian density function of the ground clues as
步骤S6-2：如果更新前后地面线索高斯密度函数均值不发生变化，即则认为该目标区域稳定；否则，如果更新前后地面线索高斯密度函数均值发生变化，即则认为该目标区域不稳定；Step S6-2: If the mean value of the ground clue Gaussian density function does not change before and after the update, that is Then it is considered that the target area is stable; otherwise, if the mean value of the Gaussian density function of ground clues changes before and after the update, that is Then the target area is considered unstable;
步骤S6-3：重复步骤S6-2，判断是否所有垂直目标区域稳定，区域稳定的计算方式为：Step S6-3: Repeat step S6-2 to judge whether all vertical target areas are stable, and the calculation method of area stability is:
步骤S6-4：如果stab>0，则说明不是所有垂直目标区域稳定，即至少处在一个垂直目标区域的高斯密度函数均值发生变化，则重复步骤S5，对所有垂直目标区域进行一次地面线索传播；Step S6-4: If stab>0, it means that not all vertical target areas are stable, that is, the mean value of the Gaussian density function in at least one vertical target area changes, then repeat step S5, and conduct ground clue propagation for all vertical target areas ;
步骤S6-5：重复步骤S6-3，判断是否所有垂直目标区域稳定，如果不是所有垂直目标区域稳定，则重复执行步骤S6-4，直到所有垂直目标区域稳定，获得收敛状态；Step S6-5: Repeat step S6-3 to determine whether all vertical target areas are stable, if not all vertical target areas are stable, then repeat step S6-4 until all vertical target areas are stable and a convergence state is obtained;
步骤S6-6：根据收敛状态的高斯密度函数均值作为垂直目标区域的目标深度估计值，像中目标的深度估计值Step S6-6: Gaussian density function mean according to convergence state As the target depth estimate of the vertical target area, the depth estimate of the target in the image
本发明克服了局部遮挡、全局场景线索之间的深度估计不一致等问题，提出了一种基于全局布局约束的道路目标深度估计方法，联合考虑局部遮挡、地面接触与水平消失线的垂直距离获取全局目标间的深度关系。与现有技术相比，本发明的主要优势如下：The present invention overcomes the problems of local occlusion and inconsistency in depth estimation among global scene clues, and proposes a road target depth estimation method based on global layout constraints, which jointly considers local occlusion, ground contact and the vertical distance of horizontal disappearing lines to obtain the global Deep relationships between objects. Compared with prior art, main advantage of the present invention is as follows:
(1)目标的场景深度受到透视假设的约束，但是，目前场景深度估计中没有充分利用该透视假设，本发明首先完成了目标区域的提取，并利用地面接触线和消失点初步估计目标的场景深度，并对场景深度跨度较大的特点，采用概率分布的方式描述目标的场景深度，同时具有可靠性和鲁棒性。(1) The scene depth of the target is constrained by the perspective assumption. However, the perspective assumption is not fully utilized in the current scene depth estimation. The present invention first completes the extraction of the target area, and uses the ground contact line and the vanishing point to initially estimate the scene of the target Depth, and for the characteristics of a large scene depth span, the probability distribution method is used to describe the scene depth of the target, which is reliable and robust at the same time.
(2)目标之间的轮廓能够指示目标之间的遮挡关系，是一种重要的场景深度估计的约束，能够有效发现错误的场景深度估计，但是，目前方法中没有充分使用边缘朝向关系作为深度估计的线索，本发明在共享边界检测的基础上，构建基于支持向量机的训练器，在训练器的基础上，实现可靠的边缘遮挡关系检测。(2) The contours between objects can indicate the occlusion relationship between objects, which is an important constraint for scene depth estimation, and can effectively detect wrong scene depth estimates. However, the current method does not fully use the edge orientation relationship as depth Estimated clues, the present invention builds a support vector machine-based trainer on the basis of shared boundary detection, and realizes reliable edge occlusion relationship detection on the basis of the trainer.
(3)目标的场景深度冲突不仅仅是局部的，但是，目前方法中并没有考虑基于图模型的场景深度估计方法，因此，无法描述深度传播过程，已经利用深度传播过程解决更多的场景深度冲突问题，本发明根据边缘遮挡关系，将地面线索进行传播，并实现场景深度迭代推理，从而实现可靠的场景深度估计。(3) The scene depth conflict of the target is not only local. However, the current method does not consider the scene depth estimation method based on the graph model. Therefore, the depth propagation process cannot be described. The depth propagation process has been used to solve more scene depths. For the conflict problem, the present invention propagates ground clues according to the edge occlusion relationship, and realizes scene depth iterative reasoning, thereby realizing reliable scene depth estimation.
附图说明Description of drawings
图1为本发明提出的场景深度估计方法流程图；Fig. 1 is the scene depth estimation method flowchart that the present invention proposes;
图2为本发明实施例中超像素分割和区域合并，其中：Fig. 2 is superpixel segmentation and region merging in the embodiment of the present invention, wherein:
图2a为超像素分割结果，图2b为区域合并结果；Figure 2a is the result of superpixel segmentation, and Figure 2b is the result of region merging;
图3为本发明实施例中边缘检测和地面消失点，其中：Fig. 3 is edge detection and ground vanishing point in the embodiment of the present invention, wherein:
图3a为边缘检测结果，图3b为直线检测结果，图3c为地面消失点估计结果；Figure 3a is the edge detection result, Figure 3b is the line detection result, and Figure 3c is the ground vanishing point estimation result;
图4为本发明实施例中分割区域深度估计，其中：Fig. 4 is the segmentation area depth estimation in the embodiment of the present invention, wherein:
图4a为区域表面标记结果，图4b为列深度图像，图4c为区域深度图像，均值，Figure 4a is the result of area surface labeling, Figure 4b is the column depth image, Figure 4c is the area depth image, mean,
图4d为高斯概率密度函数，区域j；Figure 4d is a Gaussian probability density function, region j;
图5为本发明实施例中边缘遮挡关系检测，其中：Fig. 5 is the edge occlusion relation detection in the embodiment of the present invention, wherein:
图5a为共享边界结果，图5b为边缘遮挡关系检测结果；Figure 5a is the result of shared boundary, and Figure 5b is the detection result of edge occlusion relationship;
图6为本发明实施例道路目标深度估计，其中：Fig. 6 is the road object depth estimation of the embodiment of the present invention, wherein:
图6a为初始的高斯概率密度函数，图6b为更新后的高斯概率密度函数，图6c为最终的目标深度估计结果。Figure 6a is the initial Gaussian probability density function, Figure 6b is the updated Gaussian probability density function, and Figure 6c is the final target depth estimation result.
具体实施方式Detailed ways
下面结合附图及具体实施方式详细介绍本发明。本发明为基于地面线索传播的道路目标深度估计，具体流程如图1所示，本发明的实现方案分为以下步骤：The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The present invention is a road target depth estimation based on ground clue propagation. The specific process is shown in Figure 1. The implementation of the present invention is divided into the following steps:
步骤S1：对输入道路图像，提取过分割超像素，合并生成目标区域，具体操作步骤包括：Step S1: For the input road image, extract over-segmented superpixels and combine them to generate the target area. The specific operation steps include:
步骤S1-1：对输入图像，使用分水岭算法，获得过分割超像素集合SP，记作SP＝{spi},i＝1,...,nsp，其中，spi是编号为i的超像素，nsp是道路图像中超像素的个数；Step S1-1: For the input image, use the watershed algorithm to obtain the over-segmented superpixel set SP, denoted as SP={spi }, i=1,...,nsp , where spi is the number i Superpixels, nsp is the number of superpixels in the road image;
步骤S1-2：计算所有超像素的平均RGB颜色，提取所有超像素的平均RGB颜色C＝{ci},i＝1,...,nsp；Step S1-2: Calculate the average RGB color of all superpixels, and extract the average RGB color C={ci },i=1,...,nsp of all superpixels;
步骤S1-3：将所有超像素，标记为未遍历状态，记录未标记超像素集合Vset＝{spi},i＝1,...,nsp；Step S1-3: Mark all superpixels as untraversed state, and record the unmarked superpixel set Vset={spi },i=1,...,nsp ;
步骤S1-4：对所有超像素，初始化超像素合并编号为labeli＝i，获得超像素合并编号集合Label＝{labeli},i＝1,...,nsp；Step S1-4: For all superpixels, initialize the superpixel merging number as labeli =i, and obtain the superpixel merging number set Label={labeli }, i=1,...,nsp ;
步骤S1-5：如果未标记超像素集合Vset不为空，选择遍历超像素spj，对遍历超像素和周围超像素，进行区域合并；Step S1-5: If the unmarked superpixel set Vset is not empty, select the traversal superpixel spj , and perform region merging on the traversal superpixel and surrounding superpixels;
步骤S1-5-1：从未遍历的超像素中，选择遍历超像素spj，并将该超像素标记为已遍历状态vj＝0；Step S1-5-1: Select a traversed superpixel spj from the untraversed superpixels, and mark this superpixel as traversed state vj =0;
步骤S1-5-2：从未标记超像素集合Vset去除遍历超像素spj编号，更新未标记超像素集合Vset；Step S1-5-2: remove the traversed superpixel spj number from the unmarked superpixel set Vset, and update the unmarked superpixel set Vset;
步骤S1-5-3：对遍历像素spj，统计其周围的超像素编号，获得邻接超像素编号，找出其中未遍历的邻接超像素编号neighj；Step S1-5-3: For the traversable pixel spj , count the superpixel numbers around it, obtain the adjacent superpixel numbers, and find out the adjacent superpixel numbers neighj that have not been traversed;
步骤S1-5-4：如果未遍历的邻接超像素编号neighj为空集合，则停止对遍历超像素spj的遍历；Step S1-5-4: If the untraversed adjacent superpixel number neighj is an empty set, stop traversing the traversed superpixel spj ;
步骤S1-5-5：如果未遍历的邻接超像素编号neighj不为空集合，则选择一个其中未遍历的邻接超像素spk；Step S1-5-5: If the untraversed adjacent superpixel number neighj is not an empty set, select an untraversed adjacent superpixel spk among them;
步骤S1-5-6：根据RGB平均颜色集合C＝{ci},i＝1,...,nsp，使用2范数，计算遍历像素spj和邻接超像素spk的颜色距离Step S1-5-6: Calculate the color distance between the traversed pixel spj and the adjacent superpixel spk according to the RGB average color set C={ci },i=1,...,nsp , using the 2-norm
djk＝||ci-ck||2djk ＝||ci -ck ||2
步骤S1-5-7：如果遍历像素spj和邻接超像素spk的颜色距离djk＝||ci-ck||2≥T，其中T为合并阈值，则不进行区域合并；Step S1-5-7: If the color distance djk = ||ci -ck ||2≥ T between the traversed pixel spj and the adjacent superpixel spk , where T is the merging threshold, region merging is not performed;
步骤S1-5-8：如果遍历像素spj和邻接超像素spk的颜色距离，djk＝||ci-ck||2<T，其中T为合并阈值，则进行区域合并；合并方式为：将遍历超像素spj编号，赋值给邻接超像素spk编号，具体操作方式为：Step S1-5-8: If the color distance between the traversal pixel spj and the adjacent superpixel spk , djk = ||ci -c k||2 <T, where T is the merge threshold, perform region merge; merge The method is: assign the number of the traversed superpixel spj to the number of the adjacent superpixel spk . The specific operation method is:
labelk＝labeljlabelk = labelj
步骤S1-5-9：将超像素spk，从未遍历的超像素编号neighj去除，获得更新后的未遍历的邻接超像素编号neighj；Step S1-5-9: remove the superpixel spk from the untraversed superpixel number neighj to obtain the updated untraversed adjacent superpixel number neighj ;
步骤S1-5-10：如果未遍历的邻接超像素编号neighj不为空集合，重复步骤S1-5-4到步骤S1-5-10，直到完成对邻接超像素编号neighj的遍历；Step S1-5-10: If the untraversed adjacent superpixel number neighj is not an empty set, repeat steps S1-5-4 to step S1-5-10 until the traversal of the adjacent superpixel number neighj is completed;
步骤S1-6：如果未标记超像素集合Vset不为空，则重复执行步骤S1-5，直到完成所有超像素的遍历，最终获得合并后的超像素合并编号Label＝{labeli},i＝1,...,nsp；Step S1-6: If the unlabeled superpixel set Vset is not empty, repeat step S1-5 until the traversal of all superpixels is completed, and finally obtain the merged superpixel number Label={labeli }, i= 1,...,nsp ;
步骤S1-7：根据合并后的超像素合并编号Label＝{labeli},i＝1,...,nsp，将相同编号的超像素进行合并，获得目标区域集合R，记作R＝{ri},i＝1,...,nr，其中，ri是编号为i的目标区域，nr是道路图像中目标的个数。Step S1-7: According to the merged superpixel number Label={labeli },i=1,...,nsp , merge the superpixels with the same number to obtain the target region set R, denoted as R= {ri },i=1,...,nr , where ri is the target area numbered i, and nr is the number of targets in the road image.
步骤S2：识别道路图像中的垂直物体区域，利用高斯概率分布估计，提取分割区域深度，具体操作步骤包括：Step S2: Identify the vertical object area in the road image, and use Gaussian probability distribution estimation to extract the depth of the segmented area. The specific operation steps include:
步骤S2-1：采集图像块数据，构建图像表面检测器，获取图像表面标记；Step S2-1: collecting image block data, constructing an image surface detector, and obtaining image surface markers;
步骤S2-1-1：提取分割区域的平均颜色特征xi,i＝1,...,nr；Step S2-1-1: Extracting the average color feature xi of the segmented area, i=1,...,nr ;
步骤S2-1-2：使用地面检测器模型进行地面区域判断如果ygi＝1，则该区域标记为地面yli＝groud；如果ygi＝0，则不更新该区域标记；Step S2-1-2: Use the ground detector model Make ground area judgments If ygi =1, the area is marked as ground yli =groud; if ygi =0, the area mark is not updated;
步骤S2-1-3：使用天空检测器模型进行天空区域判断如果ysi＝1，则该区域标记为天空yli＝sky；如果ysi＝0，则不更新该区域标记；Step S2-1-3: Use the sky detector model Make Sky Region Judgment If ysi =1, the area is marked as sky yli =sky; if ysi =0, the area mark is not updated;
步骤S2-1-4：重复步骤步骤S2-1-2和步骤步骤S2-1-3，对所有分割区域进行标记更新，将非地面、天空区域记做垂直物体区域，获得地面、天空、垂直物体区域检测的标记集合，作为图像表面标记集合，记作YL＝{yli},i＝1,...,nr；Step S2-1-4: Repeat step S2-1-2 and step S2-1-3 to update all the segmentation areas, record the non-ground and sky areas as vertical object areas, and obtain the ground, sky, vertical object area The mark set of object region detection, as the image surface mark set, is recorded as YL={yli }, i=1,...,nr ;
步骤S2-1-5：根据图像表面标记集合YL＝{yli},i＝1,...,nr，对道路图像各像素进行图像表面标记赋值，具体方式为，查找图像分割区域ri，对该区域中的所有像素，赋值对应的表面标记集合yli，最终获得表面标记图像Isurf；Step S2-1-5: According to the image surface label set YL={yli },i=1,...,nr , assign the image surface label value to each pixel of the road image. The specific method is to search for the image segmentation region ri , assign the corresponding surface marker set yli to all the pixels in the area, and finally obtain the surface marker image Isurf ;
步骤S2-2：根据道路图像，提取地面消失点，具体操作步骤包括：Step S2-2: According to the road image, extract the ground vanishing point, the specific operation steps include:
步骤S2-2-1：对道路图像进行hough直线检测，获得直线集合H＝{hi},i＝1,...,nh，其中hi是编号为i的直线，nh是道路图像中直线的个数；Step S2-2-1: Perform hough straight line detection on the road image to obtain a set of straight lines H={hi },i=1,...,nh , where hi is the straight line numbered i, and nh is the road The number of straight lines in the image;
步骤S2-2-2：根据直线hj和直线hk的方程，计算直线hj和直线hk的交叉点，依次遍历直线集合H＝{hi},i＝1,...,nh，计算两两直线的交叉点，获得交叉点集合PT＝{ptjk},j＝1,...,nh,k＝1,...,nh,j≠k；Step S2-2-2: According to the equations of straight line hj and straight line hk , calculate the intersection point of straight line hj and straight line hk , and traverse the set of straight lines H={hi }, i=1,...,n in sequenceh , calculate the intersection point of two straight lines, and obtain the intersection point set PT={ptjk },j=1,...,nh ,k=1,...,nh ,j≠k;
步骤S2-2-3：在交叉点图像中，选择以交叉点ptjk为圆心，半径为rad的圆形区域Circlejk，将圆形区域Circlejk内出现的交叉点个数，作为交叉点作为地面消失点的投票数votejk；Step S2-2-3: In the intersection point image, select the circle area Circlejk with the intersection point ptjk as the center and radius rad, and use the number of intersection points appearing in the circle area Circlejk as the intersection point as The number of votes at the ground vanishing point votejk ;
步骤S2-2-4：根据步骤S2-2-3，计算所有交叉点的投票数，将投票数最高的交叉点，作为道路图像的地面消失点vp＝{vpx,vpy}，其中，vpx是消失点的水平方向的坐标，vpy是消失点垂直方向的坐标；Step S2-2-4: According to step S2-2-3, calculate the number of votes for all intersections, and use the intersection with the highest number of votes as the ground vanishing point of the road image vp={vpx,vpy}, where vpx is The horizontal coordinates of the vanishing point, vpy is the vertical coordinates of the vanishing point;
步骤S2-3：对道路图像的垂直物体区域，利用高斯概率分布估计，提取分割区域深度；Step S2-3: For the vertical object area of the road image, the Gaussian probability distribution is used to estimate the depth of the segmented area;
步骤S2-3-1：根据图像表面标记集合YL＝{yli},i＝1,...,nr，查找其中地面标记的区域，获取所有地面区域Rg；Step S2-3-1: According to the image surface marker set YL={yli }, i=1,...,nr , find the area marked by the ground, and obtain all the ground areas Rg;
步骤S2-3-2：对地面区域Rg进行轮廓分析，获得地面区域边界Bg；Step S2-3-2: Perform contour analysis on the ground area Rg to obtain the ground area boundary Bg;
步骤S2-3-3：对地面区域边界Bg，找出第kv列中最低的地面区域边界像素gpkv＝{kv,gpykv}，其中，kv表示地面区域边界像素点的水平方向的坐标，即第kv列，gpykv是地面区域边界像素点垂直方向的坐标；Step S2-3-3: For the ground area boundary Bg, find out the lowest ground area boundary pixel gpkv ={kv, gpykv } in the kv column, where kv represents the horizontal direction coordinates of the ground area boundary pixel point, That is, the kv column, gpykv is the coordinate in the vertical direction of the boundary pixel of the ground area;
步骤S2-3-4：估计第kv列的最小场景深度depthkv，具体方式为：Step S2-3-4: Estimate the minimum scene depth depthkv of the kv-th column, the specific method is:
其中，A是摄像机高度和摄像机焦距的乘积，gpykv是地面区域边界像素点垂直方向的坐标，vpy是消失点垂直方向的坐标；Among them, A is the product of the height of the camera and the focal length of the camera, gpykv is the coordinate of the vertical direction of the boundary pixel of the ground area, and vpy is the coordinate of the vertical direction of the vanishing point;
步骤S2-3-5：表面标记图像Isurf，查找出第kv列中垂直表面标记的像素，并赋值第kv列的最小场景深度depthkv；Step S2-3-5: surface mark image Isurf , find out the pixels of the vertical surface mark in column kv, and assign the minimum scene depth depth kv in columnkv ;
步骤S2-3-6：重复步骤S2-3-4到步骤S2-3-5，依次计算每列的最小场景深度depthkv，并赋值给对应列的垂直表面标记的像素，获得道路图像的列深度图像Dvertical；Step S2-3-6: Repeat step S2-3-4 to step S2-3-5, calculate the minimum scene depth depthkv of each column in turn, and assign it to the pixel of the vertical surface mark of the corresponding column, and obtain the column of the road image Depth image Dvertical ;
步骤S2-3-7：根据图像表面标记集合YL＝{yli},i＝1,...,nr，找出第i个垂直物体区域的所有像素集合；Step S2-3-7: According to the image surface label set YL={yli }, i=1,...,nr , find out all the pixel sets of the i-th vertical object region;
步骤S2-3-8：根据道路图像的列深度图像Dvertical，提取第i个垂直物体区域的所有像素集合对应的场景深度，获得第i个垂直物体区域的场景深度值集合dseti,i＝1,...,nr；Step S2-3-8: According to the column depth image Dvertical of the road image, extract the scene depth corresponding to all pixel sets of the i-th vertical object area, and obtain the scene depth value set dseti of the i-th vertical object area, i= 1,...,nr ;
步骤S2-3-9：对第i个垂直物体区域的场景深度值集合dseti,i＝1,...,nr，求解第i个垂直物体区域的场景深度均值ui和方差σi，其中i＝1,...,nr表示区域编号；Step S2-3-9: For the scene depth value set dseti of the i-th vertical object area, i=1,...,nr , calculate the scene depth mean ui and variance σi of the i-th vertical object area , where i=1,...,nr represents the area number;
步骤S2-3-10：根据第i个垂直物体区域的场景深度均值ui和方差σi，构建高斯形式的概率密度函数，作为该分割区域的场景深度估计Step S2-3-10: According to the scene depth mean ui and variance σi of the i-th vertical object area, construct a Gaussian probability density function as the scene depth estimation of the segmented area
pdfi＝Gaussian(ui,σi)pdfi ＝Gaussian(ui ,σi )
其中i＝1,...,nr，Gaussian(ui,σi)表示以ui为均值，以σi为方差的高斯概率密度函数；Where i=1,...,nr , Gaussian(ui ,σi ) represents the Gaussian probability density function with ui as the mean and σi as the variance;
步骤S2-3-11：重复步骤S2-3-8到步骤S2-3-10，提取所有垂直物体区域的场景深度均值ui和方差σi，以及其对应的场景深度估计pdfi＝Gaussian(ui,σi),i＝1,...,nr。Step S2-3-11: Repeat steps S2-3-8 to S2-3-10 to extract the scene depth mean value ui and variance σi of all vertical object regions, and the corresponding scene depth estimation pdfi =Gaussian( ui ,σi ), i=1,...,nr .
步骤S3：提取共享边界，构建边缘遮挡朝向检测器，具体操作步骤包括：Step S3: Extract the shared boundary and construct an edge occlusion orientation detector. The specific operation steps include:
步骤S3-1：针对道路图像的分割区域，根据邻接关系矩阵，提取分割区域间的共享边界；Step S3-1: For the segmented area of the road image, extract the shared boundary between the segmented areas according to the adjacency matrix;
步骤S3-1-1：道路图像的分割区域R＝{ri},i＝1,...,nr，提取邻接关系矩阵，记录邻接关系矩阵Adjacent＝{adjj,k}，其中将邻接关系矩阵中的元素赋值为adjj,k＝1，表示分割区域ri和分割区域rk存在邻接关系；Step S3-1-1: In the segmented area R={ri },i=1,...,nr of the road image, extract the adjacency matrix and record the adjacency matrix Adjacent={adjj,k }, where The elements in the adjacency matrix are assigned adjj,k = 1, which means that there is an adjacency relationship between the segmented region ri and the segmented region rk ;
步骤S3-1-2：根据邻接关系矩阵Adjacent＝{adjj,k}，找出两个邻接区域，分割区域ri和分割区域rk，将两个区域之间的共享边界，记为道路图像中所有的共享边界Bound＝{boundj,k}，其中，j＝1,...,nh,k＝1,...,nh,j≠k；Step S3-1-2: According to the adjacency matrix Adjacent={adjj,k }, find two adjacent regions, split region ri and split region rk , record the shared boundary between the two regions as a road All shared boundaries Bound={boundj,k } in the image, where j=1,...,nh ,k=1,...,nh ,j≠k;
步骤S3-1-3：根据道路图像中所有的共享边界Bound＝{boundj,k}，将共享边界对应像素标记为边界，获得共享边界图像Ibound；Step S3-1-3: According to all shared boundaries Bound={boundj,k } in the road image, mark the pixels corresponding to the shared boundaries as boundaries, and obtain the shared boundary image Ibound ;
步骤S3-2：对共享边界区域正则化，并进行共享边界朝向标记；Step S3-2: Regularize the shared boundary area and mark the shared boundary orientation;
步骤S3-2-1：根据道路图像中所有的共享边界Bound＝{boundj,k}，选择一条共享边界boundj,k；Step S3-2-1: Select a shared boundary boundj,k according to all shared boundaries Bound={boundj,k } in the road image;
步骤S3-2-2：根据共享边界boundj,k像素下标，选择第m个像素作为参考像素prefm；Step S3-2-2: Select the mth pixel as the reference pixel prefm according to the shared boundary boundj,k pixel index;
步骤S3-2-3：计算参考像素prefm处的切线，将切线和图像的水平线夹角，作为共享边界的切线方向θrefm；Step S3-2-3: Calculate the tangent at the reference pixel prefm , and use the angle between the tangent and the horizontal line of the image as the tangent direction θrefm of the shared boundary;
步骤S3-2-4：以参考像素prefm为中心旋转，将共享边界图像Ibound，旋转θrefm角度，获得旋转后图像，以保证旋转后的图像中，参考像素prefm处的切线为水平方向；Step S3-2-4: Rotate around the reference pixel prefm , rotate the shared boundary image Ibound by the angle θrefm , and obtain the rotated image, so as to ensure that the tangent at the reference pixel prefm in the rotated image is horizontal direction;
步骤S3-2-5：在旋转后图像中，以共享边界的参考像素prefm，以半径为15，提取参考像素的图像块，作为正则参考像素图像块patchm；Step S3-2-5: In the rotated image, take the reference pixel prefm sharing the boundary and take the radius of 15 to extract the image block of the reference pixel as the regular reference pixel image block patchm ;
步骤S3-2-6：对正则参考像素图像块patchm进行边缘遮挡朝向标记，具体操作为，如果正则参考像素图像块patchm中下方目标遮挡上方目标，则遮挡标记，记为yom＝1；如果正则参考像素图像块patchm中上方目标遮挡下方目标，则遮挡标记，记为yom＝0；Step S3-2-6: mark the edge occlusion direction of the regular reference pixel image block patchm , the specific operation is, if the lower object in the regular reference pixel image block patchm blocks the upper object, then the occlusion mark is recorded as yom = 1 ; If the upper object in the regular reference pixel image block patchm blocks the lower object, then the occlusion mark is recorded as yom = 0;
步骤S3-3：构建边缘遮挡朝向检测器，训练边缘遮挡朝向检测器模型；Step S3-3: building an edge occlusion orientation detector, and training an edge occlusion orientation detector model;
步骤S3-3-1：重复步骤S3-2-1到步骤S3-2-5，可以对不同共享边界上进行多次采样，获得多个遮挡朝向样本，作为遮挡朝向的训练样本集合；Step S3-3-1: Repeat steps S3-2-1 to S3-2-5, multiple samplings can be performed on different shared boundaries, and multiple occlusion orientation samples can be obtained as a training sample set of occlusion orientations;
步骤S3-3-2：对遮挡朝向的训练样本集合，重复步骤S3-2-6，获得集合中各样本的遮挡朝向标记yogt；Step S3-3-2: For the training sample set of occlusion orientation, repeat step S3-2-6 to obtain the occlusion orientation mark yogt of each sample in the set;
步骤S3-3-3：对遮挡朝向样本的图像块，依据行遍历方式，转化为边界行向量特征xo；Step S3-3-3: For the image block occluded towards the sample, convert it into the boundary row vector feature xo according to the row traversal method;
步骤S3-3-4：构建边缘遮挡朝向检测器，预测待检测区域的边缘遮挡朝向标记yotest，具体方式为Step S3-3-4: Build an edge occlusion orientation detector to predict the edge occlusion orientation mark yotest of the area to be detected, the specific method is
yotest＝wo·xotestyotest = wo xotest
其中，xotest是待检测区域的边界行向量，wo是边缘遮挡朝向检测器的参数；Among them, xotest is the boundary row vector of the area to be detected, and wo is the parameter of the edge occlusion orientation detector;
步骤S3-3-5：根据支持向量机，设计边缘遮挡朝向检测器目标函数Step S3-3-5: According to the support vector machine, design the objective function of the edge occlusion orientation detector
步骤S3-3-6：根据支持向量机，训练边缘遮挡朝向检测器模型，求解出最佳的边缘遮挡朝向检测器参数具体方式为Step S3-3-6: According to the support vector machine, train the edge occlusion orientation detector model, and solve the best edge occlusion orientation detector parameters The specific method is
步骤S4：对图像中的共享边界，提取分割区域间边缘遮挡关系，具体操作步骤包括：Step S4: For the shared boundary in the image, extract the edge occlusion relationship between the segmented regions, and the specific operation steps include:
步骤S4-1：初始化图像中所有区域的边缘线索遮挡关系矩阵Occ＝{occj,k}，其中，j＝1,...,nh,k＝1,...,nh,j≠k，其中初始值为occj,k＝0，表示分割区域j和分割区域k无遮挡关系；Step S4-1: Initialize the edge clue occlusion relationship matrix Occ={occj,k } of all regions in the image, where j=1,...,nh ,k=1,...,nh ,j ≠ k, where the initial value is occj, k = 0, indicating that there is no occlusion relationship between the segmented area j and the segmented area k;
步骤S4-2：根据道路图像中所有的共享边界Bound＝{boundj,k}，选择一条共享边界boundj,k；Step S4-2: Select a shared boundary boundj,k according to all shared boundaries Bound={boundj,k } in the road image;
步骤S4-3：对共享边界两侧的分割区域j和分割区域k，初始化分割区域前景投票向量Votej,k＝{votej,votek}，其中初始值为votej＝0，votek＝0；Step S4-3: For the segmented area j and segmented area k on both sides of the shared boundary, initialize the segmented area foreground voting vector Votej,k ={votej ,votek }, where the initial value is votej =0, votek = 0;
步骤S4-4：根据共享边界boundj,k像素个数，等间隔采样，选择候选像素集合Pref＝{prefm}；Step S4-4: According to the number of shared boundary boundj,k pixels, sampling at equal intervals, select the candidate pixel set Pref={prefm };
步骤S4-5：从候选像素集合Pref＝{prefm}中选择一个候选像素prefm，重复步骤S3-2-3到步骤S3-2-5，获得候选像素的正则图像块patchm；Step S4-5: Select a candidate pixel prefm from the candidate pixel set Pref={prefm }, repeat step S3-2-3 to step S3-2-5, and obtain the regular image block patchm of the candidate pixel;
步骤S4-6：根据原始共享边界两侧的分割区域j和分割区域k，记录候选像素的正则图像块patchm的上下区域编号，上方区域分割编号为j'，下方区域分割编号为k'；Step S4-6: According to the segmentation area j and the segmentation area k on both sides of the original shared boundary, record the upper and lower area numbers of the regular image block patchm of the candidate pixel, the upper area segmentation number is j', and the lower area segmentation number is k';
步骤S4-7：对候选像素的正则图像块patchm，依据行遍历方式，转化为边界行向量特征xotest；Step S4-7: For the regular image block patchm of the candidate pixel, convert it into the boundary row vector feature xotest according to the row traversal method;
步骤S4-8：使用训练完成的边缘遮挡朝向检测器模型对正则图像块判断Step S4-8: Use the trained edge occlusion orientation detector model Judgment on regular image blocks
步骤S4-9：如果yoi＝1，表示下方目标k'遮挡上方目标j'，即下方目标k'为靠近摄像机的前景区域，则对下方目标区域的前景进行一次投票Step S4-9: If yoi = 1, it means that the lower object k' blocks the upper object j', that is, the lower object k' is the foreground area close to the camera, then conduct a vote on the foreground of the lower object area
votek'＝votek'+1votek' = votek' +1
步骤S4-10：如果yoi＝0，表示上方目标j'遮挡下方目标k'，即上方目标j'为靠近摄像机的前景区域，则对上方目标区域的前景进行一次投票Step S4-10: If yoi = 0, it means that the upper object j' blocks the lower object k', that is, the upper object j' is the foreground area close to the camera, then perform a vote on the foreground of the upper object area
votej'＝votej'+1votej' = votej' +1
步骤S4-11：重复步骤S4-4到步骤S4-10，完成对共享边界boundj,k的多次等间隔采样，并完成多次前景区域投票，获得分割区域前景投票向量Votej,k＝{votej,votek}；Step S4-11: Repeat Step S4-4 to Step S4-10 to complete multiple equal-spaced sampling of the shared boundary boundj,k , and complete multiple foreground area voting, and obtain the divided area foreground voting vector Votej,k = { votej , votek };
步骤S4-12：比较投票数，投票数较大的分割区域遮挡投票数较小的分割区域，记录遮挡关系元素occj,k；如果分割区域j遮挡分割区域k，则记录occj,k＝1；如果分割区域k遮挡分割区域j，则记录occj,k＝-1；Step S4-12: Compare the number of votes, the segmented area with a larger number of votes covers the segmented area with a smaller number of votes, and record the occlusion relationship element occj,k ; if the segmented area j blocks the segmented area k, then record occj,k = 1; if segmented area k blocks segmented area j, then record occj,k =-1;
步骤S4-13：重复步骤S4-2到步骤S4-12，完成对图像中所有共享边界的分析，获得所有共享边界两侧区域对之间的边缘线索的遮挡关系矩阵Occ＝{occj,k}。Step S4-13: Repeat Step S4-2 to Step S4-12 to complete the analysis of all shared boundaries in the image, and obtain the occlusion relationship matrix Occ={occj,k }.
步骤S5：利用地面线索传播，估计目标深度，具体操作步骤包括：Step S5: Estimate the depth of the target by using ground clues, and the specific steps include:
步骤S5-1：根据根据图像表面标记集合YL＝{yli},i＝1,...,nr，获取垂直目标区域的编号集合VYL＝{vi}，其中ylvi＝vertical,vi∈{1,...,nr}；Step S5-1: According to the image surface label set YL={yli },i=1,...,nr , obtain the number set VYL={vi} of the vertical target area, where ylvi =vertical,vi∈ {1,...,nr };
步骤S5-2：垂直目标区域的编号集合VYL＝{vi}，对共享边界Bound＝{boundj,k}进行筛选，找出共享边界两侧都是垂直目标区域的垂直目标间共享边界GBound＝{gboundj,k}，其中，j,k∈VYL；Step S5-2: The number set VYL={vi} of the vertical target area, screen the shared boundary Bound={boundj, k }, and find out the shared boundary GBound= between vertical targets with vertical target areas on both sides of the shared boundary {gboundj,k }, where j,k∈VYL;
步骤S5-3：对垂直目标间共享边界GBound＝{gboundj,k}，其中j,k∈VYL，选择一条共享边界gboundj,k进行地面线索传递；Step S5-3: For the shared boundary GBound={gboundj,k } between vertical targets, where j,k∈VYL, select a shared boundary gboundj,k for ground clue transmission;
步骤S5-3-1：根据共享边界gboundj,k，其中j,k∈VYL，获取共享边界两侧区域编号j,k；Step S5-3-1: According to the shared boundary gboundj,k , where j,k∈VYL, obtain the area numbers j,k on both sides of the shared boundary;
步骤S5-3-2：针对共享边界两侧区域编号j,k，获取其地面线索高斯密度函数均值，其中，区域编号j的地面线索高斯密度函数均值为uj；区域编号k的地面线索高斯密度函数均值为uk；Step S5-3-2: For the area numbers j and k on both sides of the shared boundary, obtain the mean value of the Gaussian density function of the ground clues, where the mean value of the Gaussian density function of the ground clues of the area number j is uj ; the Gaussian density function of the ground clues of the area number k is u j ; The mean value of the density function is uk ;
步骤S5-3-3：根据共享边界两侧区域的地面线索高斯密度函数均值，记录区域间的遮挡关系，具体操作为：如果区域编号j的均值uj小于区域编号k的均值uk，即uj<uk，说明区域编号k距离摄像头更远，区域编号k被区域编号j遮挡，记录地面线索遮挡关系为goccj,k＝1；如果区域编号j的均值uj大于区域编号k的均值uk，即uj>uk，说明区域编号j距离摄像头更远，区域编号j被区域编号k遮挡，记录地面线索遮挡关系为goccj,k＝-1；如果区域编号j的均值uj等于区域编号k的均值uk，即uj＝uk，说明区域编号k与区域编号j无遮挡关系，记录地面线索遮挡关系为goccj,k＝0；Step S5-3-3: According to the mean value of the ground clue Gaussian density function of the areas on both sides of the shared boundary, record the occlusion relationship between the areas. The specific operation is: if the mean value uj of the area number j is smaller than the mean value uk of the area number k, that is uj < uk , indicating that the area number k is farther away from the camera, and the area number k is blocked by the area number j, and the occlusion relation of the ground clue is recorded as goccj,k = 1; if the mean value uj of the area number j is greater than that of the area number k The mean value uk , that is, uj >uk , indicates that the area number j is farther away from the camera, and the area number j is blocked by the area number k, and the occlusion relation of recorded ground clues is goccj,k = -1; if the mean value u of the area number jj is equal to the average value uk of area number k, that is, uj =uk , indicating that there is no occlusion relationship between area number k and area number j, and the occlusion relationship of ground clues is recorded as goccj,k =0;
步骤S5-3-4：根据共享边界两侧区域编号j,k，记录边缘线索的遮挡关系occj,k；Step S5-3-4: According to the area numbers j, k on both sides of the shared border, record the occlusion relationship occj, k of the edge clue;
步骤S5-3-5：根据地面线索遮挡关系goccj,k，以及边缘线索的遮挡关系occj,k，如果occj,k·goccj,k＝-1，说明地面线索和边缘线索得到的遮挡关系相关矛盾，则对区域编号j的地面线索的高斯密度函数pdfj＝Gaussian(uj,σj)进行更新，获得未归一化的分布函数unpdfj，具体方式为：Step S5-3-5: According to the occlusion relationship goccj,kof the ground clues and the occlusion relationship occ j,k of the edge cues, if occ j,k ·goccj,k = -1, it means that the ground clues and the edge cues are obtained If there is a contradiction related to the occlusion relationship, update the Gaussian density function pdfj =Gaussian(uj ,σj ) of the ground clue of area number j to obtain the unnormalized distribution function unpdfj , the specific method is as follows:
unpdfj＝pdfj-pdfkunpdfj =pdfj -pdfk
步骤S5-3-6：计算未归一化的分布函数unpdfj中的最小值minun，和最大值maxun；Step S5-3-6: Calculating the minimum value minun and the maximum value maxun in the unnormalized distribution function unpdfj ;
步骤S5-3-7：根据未归一化的分布函数unpdfj中的最小值minun，和最大值maxun，重新归一化区域编号j的地面线索的非高斯密度函数updfj，重新归一化的公式为：Step S5-3-7: According to the minimum value minun and the maximum value maxun in the unnormalized distribution function unpdfj , renormalize the non-Gaussian density function updfj of the ground clue of the area number j, and renormalize The formula is:
步骤S5-3-8,：根据非高斯密度函数updfj，重新计算并更新非高斯密度函数updfj的均值和方差Step S5-3-8,: According to the non-Gaussian density function updfj , recalculate and update the mean value of the non-Gaussian density function updfj and variance
步骤S5-3-9：根据非高斯密度函数updfj的均值和方差重新估计并更新区域编号j的地面线索的高斯密度函数Step S5-3-9: According to the mean value of the non-Gaussian density function updfj and variance Re-estimate and update the Gaussian density function of the ground cues for area number j
步骤S5-3-10：根据地面线索遮挡关系goccj,k，以及边缘线索的遮挡关系occj,k，如果occj,k·goccj,k＝-1，说明地面线索和边缘线索得到的遮挡关系相关矛盾，对区域编号k的地面线索的高斯密度函数进行更新，获得未归一化的分布函数unpdfj，具体方式为：Step S5-3-10: According to the occlusion relationship goccj,kof the ground clues and the occlusion relationship occ j,k of the edge cues, if occ j,k ·goccj,k = -1, it means that the ground clues and the edge cues are obtained The occlusion relationship is related to the contradiction, and the Gaussian density function of the ground clue of the area number k is updated to obtain the unnormalized distribution function unpdfj , the specific method is:
unpdfk＝pdfk-pdfjunpdfk =pdfk -pdfj
步骤S5-3-11：重复步骤S5-3-6到步骤5-3-9，重新归一化密度函数，并计算并更新均值和方差获得更新后的高斯密度函数Step S5-3-11: Repeat Step S5-3-6 to Step 5-3-9, renormalize the density function, and calculate and update the mean and variance Get the updated Gaussian density function
步骤S5-4：重复步骤S5-3，对所有垂直目标间共享边界GBound＝{gboundj,k}，进行处理更新，更新所有垂直目标区域的高斯密度函数，完成一次地面线索传播的更新过程。Step S5-4: Repeat step S5-3, update the shared boundary GBound={gboundj,k } between all vertical targets, update the Gaussian density function of all vertical target areas, and complete an update process of ground clue propagation.
步骤S6：对地面线索传播进行迭代推理，判断深度是否稳定，输出目标深度，具体操作步骤包括：Step S6: Carry out iterative reasoning on the propagation of ground clues, judge whether the depth is stable, and output the target depth. The specific operation steps include:
步骤S6-1：获得步骤S3-1中的原始地面线索高斯密度函数均值为uj，和更新后的地面线索高斯密度函数均值为Step S6-1: Obtain the mean value of the Gaussian density function of the original ground clues in step S3-1 as uj , and the mean value of the updated Gaussian density function of the ground clues as
步骤S6-2：如果更新前后地面线索高斯密度函数均值不发生变化，即则认为该目标区域稳定；否则，如果更新前后地面线索高斯密度函数均值发生变化，即则认为该目标区域不稳定；Step S6-2: If the mean value of the ground clue Gaussian density function does not change before and after the update, that is Then it is considered that the target area is stable; otherwise, if the mean value of the Gaussian density function of ground clues changes before and after the update, that is Then the target area is considered unstable;
步骤S6-3：重复步骤S6-2，判断是否所有垂直目标区域稳定，区域稳定的计算方式为：Step S6-3: Repeat step S6-2 to judge whether all vertical target areas are stable, and the calculation method of area stability is:
步骤S6-4：如果stab>0，则说明不是所有垂直目标区域稳定，即至少处在一个垂直目标区域的高斯密度函数均值发生变化，则重复步骤S5，对所有垂直目标区域进行一次地面线索传播；Step S6-4: If stab>0, it means that not all vertical target areas are stable, that is, the mean value of the Gaussian density function in at least one vertical target area changes, then repeat step S5, and conduct ground clue propagation for all vertical target areas ;
步骤S6-5：重复步骤S6-3，判断是否所有垂直目标区域稳定，如果不是所有垂直目标区域稳定，则重复执行步骤S6-4，直到所有垂直目标区域稳定，获得收敛状态；Step S6-5: Repeat step S6-3 to determine whether all vertical target areas are stable, if not all vertical target areas are stable, then repeat step S6-4 until all vertical target areas are stable and a convergence state is obtained;
步骤S6-6：根据收敛状态的高斯密度函数均值作为垂直目标区域的目标深度估计值，像中目标的深度估计值Step S6-6: Gaussian density function mean according to convergence state As the target depth estimate of the vertical target area, the depth estimate of the target in the image
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810396565.4ACN108629782B (en) | 2018-04-28 | 2018-04-28 | Road target depth estimation method based on ground clue propagation |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810396565.4ACN108629782B (en) | 2018-04-28 | 2018-04-28 | Road target depth estimation method based on ground clue propagation |
| Publication Number | Publication Date |
|---|---|
| CN108629782Atrue CN108629782A (en) | 2018-10-09 |
| CN108629782B CN108629782B (en) | 2021-09-28 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810396565.4AActiveCN108629782B (en) | 2018-04-28 | 2018-04-28 | Road target depth estimation method based on ground clue propagation |
| Country | Link |
|---|---|
| CN (1) | CN108629782B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110738665A (en)* | 2019-09-17 | 2020-01-31 | 浙江大学 | object contact identification method based on depth image information |
| CN110749861A (en)* | 2019-10-24 | 2020-02-04 | 哈尔滨工程大学 | A three-dimensional localization method for underwater fixed targets based on multi-hypothesis depth |
| CN111680702A (en)* | 2020-05-28 | 2020-09-18 | 杭州电子科技大学 | A method for weakly supervised image saliency detection using detection boxes |
| CN113129306A (en)* | 2021-05-10 | 2021-07-16 | 电子科技大学成都学院 | Occlusion object segmentation solving method based on deep learning |
| CN116433698A (en)* | 2021-12-29 | 2023-07-14 | 中国移动通信有限公司研究院 | Image information determining method, device and computer readable storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110142289A1 (en)* | 2009-12-11 | 2011-06-16 | Nxp B.V. | System and method for motion estimation using image depth information |
| CN102223553A (en)* | 2011-05-27 | 2011-10-19 | 山东大学 | Method for converting two-dimensional video into three-dimensional video automatically |
| CN104598744A (en)* | 2015-01-27 | 2015-05-06 | 北京工业大学 | Depth estimation method based on optical field |
| CN106204572A (en)* | 2016-07-06 | 2016-12-07 | 合肥工业大学 | The road target depth estimation method mapped based on scene depth |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110142289A1 (en)* | 2009-12-11 | 2011-06-16 | Nxp B.V. | System and method for motion estimation using image depth information |
| CN102223553A (en)* | 2011-05-27 | 2011-10-19 | 山东大学 | Method for converting two-dimensional video into three-dimensional video automatically |
| CN104598744A (en)* | 2015-01-27 | 2015-05-06 | 北京工业大学 | Depth estimation method based on optical field |
| CN106204572A (en)* | 2016-07-06 | 2016-12-07 | 合肥工业大学 | The road target depth estimation method mapped based on scene depth |
| Title |
|---|
| TING-CHUN WANG等: "Depth Estimation with Occlusion Modeling Using Light-Field Cameras", 《IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE》* |
| 李乐 等: "基于内容理解的单幅静态街景图像深度估计", 《机器人》* |
| 牛玉婷: "单目图像中的场景深度结构分析和估计", 《中国优秀硕士学位论文全文数据库信息科技辑》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110738665A (en)* | 2019-09-17 | 2020-01-31 | 浙江大学 | object contact identification method based on depth image information |
| CN110749861A (en)* | 2019-10-24 | 2020-02-04 | 哈尔滨工程大学 | A three-dimensional localization method for underwater fixed targets based on multi-hypothesis depth |
| CN110749861B (en)* | 2019-10-24 | 2022-06-17 | 哈尔滨工程大学 | A three-dimensional localization method for underwater fixed targets based on multi-hypothesis depth |
| CN111680702A (en)* | 2020-05-28 | 2020-09-18 | 杭州电子科技大学 | A method for weakly supervised image saliency detection using detection boxes |
| CN111680702B (en)* | 2020-05-28 | 2022-04-01 | 杭州电子科技大学 | Method for realizing weak supervision image significance detection by using detection frame |
| CN113129306A (en)* | 2021-05-10 | 2021-07-16 | 电子科技大学成都学院 | Occlusion object segmentation solving method based on deep learning |
| CN116433698A (en)* | 2021-12-29 | 2023-07-14 | 中国移动通信有限公司研究院 | Image information determining method, device and computer readable storage medium |
| Publication number | Publication date |
|---|---|
| CN108629782B (en) | 2021-09-28 |
| Publication | Publication Date | Title |
|---|---|---|
| US11971726B2 (en) | Method of constructing indoor two-dimensional semantic map with wall corner as critical feature based on robot platform | |
| CN108629782B (en) | Road target depth estimation method based on ground clue propagation | |
| US20210183080A1 (en) | Interior photographic documentation of architectural and industrial environments using 360 panoramic videos | |
| CN107093205B (en) | A kind of three-dimensional space building window detection method for reconstructing based on unmanned plane image | |
| CN101657825B (en) | Modeling Human Figures from Depth Maps | |
| Wang et al. | Window detection from mobile LiDAR data | |
| CN107357427A (en) | A kind of gesture identification control method for virtual reality device | |
| Xu et al. | csboundary: City-scale road-boundary detection in aerial images for high-definition maps | |
| CN109934847A (en) | The method and apparatus of weak texture three-dimension object Attitude estimation | |
| CN115388902A (en) | Indoor positioning method and system, AR indoor positioning navigation method and system | |
| Demantké et al. | Streamed vertical rectangle detection in terrestrial laser scans for facade database production | |
| CN112699834A (en) | Traffic identification detection method and device, computer equipment and storage medium | |
| CN116188417B (en) | Slit detection and three-dimensional positioning method based on SLAM and image processing | |
| CN114387576B (en) | Lane line recognition method, system, medium, device and information processing terminal | |
| CN104851089A (en) | Static scene foreground segmentation method and device based on three-dimensional light field | |
| CN112396701B (en) | Satellite image processing method, device, electronic device and computer storage medium | |
| Yan et al. | Depth map generation for 2d-to-3d conversion by limited user inputs and depth propagation | |
| CN110634160B (en) | 3D Keypoint Extraction Model Construction and Pose Recognition Method of Target in 2D Graphics | |
| KR102824305B1 (en) | Method and System for change detection and automatic updating of road marking in HD map through IPM image and HD map fitting | |
| CN113920254B (en) | Monocular RGB (Red Green blue) -based indoor three-dimensional reconstruction method and system thereof | |
| CN115115535A (en) | Depth map denoising method, device, medium and device | |
| Kiran et al. | Automatic hump detection and 3D view generation from a single road image | |
| CN117333548B (en) | Method, system and storage medium for estimating camera pose based on quasi-torus | |
| CN117788503A (en) | A lidar-based road boundary detection method for underground mines | |
| CN118116032A (en) | Data enhancement method and system for pedestrian detection data |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | Effective date of registration:20241223 Address after:Buildings E1 and E2, Dongguan Street Industrial Park, Xixiu District, Anshun City, Guizhou Province, 561000 Patentee after:Guizhou Yourongchuang Communication Technology Co.,Ltd. Country or region after:China Address before:Tunxi road in Baohe District of Hefei city of Anhui Province, No. 193 230009 Patentee before:Hefei University of Technology Country or region before:China | |
| TR01 | Transfer of patent right |