CN110689578A - An obstacle recognition method for UAV based on monocular vision - Google Patents
An obstacle recognition method for UAV based on monocular visionDownload PDFInfo
- Publication number
- CN110689578A CN110689578ACN201910963529.6ACN201910963529ACN110689578ACN 110689578 ACN110689578 ACN 110689578ACN 201910963529 ACN201910963529 ACN 201910963529ACN 110689578 ACN110689578 ACN 110689578A
- Authority
- CN
- China
- Prior art keywords
- point
- image
- matching
- points
- unmanned aerial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30261—Obstacle
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明属于无人机图像处理技术领域，具体涉及一种基于单目视觉的无人机障碍物识别方法。The invention belongs to the technical field of unmanned aerial vehicle image processing, and particularly relates to a method for identifying obstacles of unmanned aerial vehicles based on monocular vision.
背景技术Background technique
近年来随着无人机的发展，无人机自动巡航技术的应用范围越来越广泛。自动巡航的过程中要求无人机能够自动识别并躲避障碍物，传统的避障方法大多基于各类传感器，无人机必须搭载专用的模块，增加了成本和功耗，减少了巡航时间。近年来随着图像处理技术的发展，基于机器视觉的避障方法开始兴起；另一方面，现在的无人机大多会搭载摄像头，此类方法无需额外增加设备。目前常用的视觉避障方法分为单目视觉和双目视觉，双目视觉硬件成本高，同一时间处理数据量大，效率低；单目视觉的各类方法也各有各的缺陷，仍需进一步的改进和完善。In recent years, with the development of UAVs, the application scope of UAV automatic cruise technology has become more and more extensive. In the process of automatic cruise, the UAV is required to be able to automatically identify and avoid obstacles. Most of the traditional obstacle avoidance methods are based on various sensors. The UAV must be equipped with a dedicated module, which increases the cost and power consumption and reduces the cruise time. In recent years, with the development of image processing technology, obstacle avoidance methods based on machine vision have begun to emerge; on the other hand, most of the current drones are equipped with cameras, and such methods do not require additional equipment. At present, the commonly used visual obstacle avoidance methods are divided into monocular vision and binocular vision. The hardware cost of binocular vision is high, the amount of data processed at the same time is large, and the efficiency is low; various methods of monocular vision also have their own shortcomings, and still need to be Further improvements and refinements.
目前，常用的单目视觉障碍物识别方法多为提取前后两幅连续图像特征点并匹配，通过对比匹配点对的位置变化，以及前后两幅图像提取时间间隔内无人机的移动距离，根据三角形相似定理计算距离。常用的图像特征算法包括SIFT(尺度不变特征变换)算法、SURF(加速稳健特征)算法等，但以上两种算法计算用时较长，而无人机避障对实时性要求较高，因此并不适合。2011年，Ethan Rublee等人提出了ORB(Oriented FAST and RotationBRIEF)算法，较之SIFT算法和SURF算法，速度分别提高了100倍和10倍之多。但是ORB算法亦有缺点：不具备尺度不变性，而无人机在接近障碍物的过程中必然伴随着图像尺寸的变化，因此ORB算法若想应用于无人机避障，必须加以改进。At present, the commonly used monocular visual obstacle recognition methods are mostly to extract the feature points of two consecutive images before and after and match them. By comparing the position changes of the matched point pairs, and the moving distance of the UAV within the extraction time interval of the two images before and after, according to the The triangle similarity theorem calculates distance. Commonly used image feature algorithms include SIFT (scale invariant feature transformation) algorithm, SURF (accelerated robust feature) algorithm, etc., but the above two algorithms take a long time to calculate, and UAV obstacle avoidance requires higher real-time performance, so it is not necessary Not suitable. In 2011, Ethan Rublee et al. proposed the ORB (Oriented FAST and RotationBRIEF) algorithm. Compared with the SIFT algorithm and the SURF algorithm, the speed was increased by 100 times and 10 times respectively. However, the ORB algorithm also has shortcomings: it does not have scale invariance, and the UAV must be accompanied by changes in the image size in the process of approaching obstacles. Therefore, if the ORB algorithm is to be applied to UAV obstacle avoidance, it must be improved.
发明内容SUMMARY OF THE INVENTION
本发明的目的在于克服现有技术中的不足，提出了一种基于单目视觉的无人机障碍物识别方法，解决了原ORB算法不具备尺度不变性的缺点，使其可以应用于无人机避障。The purpose of the present invention is to overcome the deficiencies in the prior art, and proposes a method for identifying obstacles of unmanned aerial vehicles based on monocular vision, which solves the disadvantage that the original ORB algorithm does not have scale invariance, so that it can be applied to unmanned aerial vehicles. Machine avoidance.
为解决上述技术问题，本发明提供了一种基于单目视觉的无人机障碍物识别方法，其特征是，包括以下过程：In order to solve the above-mentioned technical problem, the present invention provides a kind of UAV obstacle recognition method based on monocular vision, it is characterized in that, comprises the following process:
获取单目相机采集到的无人机正前方图像，并将此图像转换成灰度图像；并从灰度图像中提取各特征点；Obtain the image of the front of the UAV collected by the monocular camera, and convert the image into a grayscale image; and extract each feature point from the grayscale image;
获取单目相机采集到的另一幅无人机正前方图像，将此图像转换为灰度图像，并从灰度图像中提取各特征点；Obtain another image of the front of the UAV collected by the monocular camera, convert this image into a grayscale image, and extract each feature point from the grayscale image;
将两幅图像中的各特征点进行匹配，将相匹配的两特征点构成匹配点对；Match the feature points in the two images, and form a matching point pair from the matched two feature points;
遍历每对匹配点对，判断无人机与特征点对应障碍物是否超出安全距离。Traverse each pair of matching points to determine whether the obstacle corresponding to the UAV and the feature point exceeds the safe distance.
进一步的，图像转换成灰度图像后，先对灰度图像进行高斯滤波。Further, after the image is converted into a grayscale image, Gaussian filtering is first performed on the grayscale image.
进一步的，所述对灰度图像进行高斯滤波包括：Further, the performing Gaussian filtering on the grayscale image includes:
采用盒子滤波器对灰度图像进行高斯滤波。The grayscale image is Gaussian filtered using a box filter.
进一步的，所述从灰度图像中提取各特征点包括以下过程：Further, the extraction of each feature point from the grayscale image includes the following process:
获取灰度图像在不同尺度下的响应图用以构建尺度空间金字塔，Obtain the response maps of grayscale images at different scales to build a scale space pyramid,
通过尺度空间金字塔中各响应图中点的响应值的对比，提取特征点。Feature points are extracted by comparing the response values of points in each response map in the scale space pyramid.
进一步的，获取灰度图像在不同尺度下的响应图用以构建尺度空间金字塔包括：Further, obtaining response maps of grayscale images at different scales to construct a scale space pyramid includes:
用若干不同尺寸的盒子滤波器对灰度图像进行滤波，滤波后产生的响应图构成一组，若干组响应图构成尺度空间金字塔。The grayscale image is filtered with several box filters of different sizes, and the response maps generated after filtering form a group, and several groups of response maps form a scale space pyramid.
进一步的，通过尺度空间金字塔中各响应图中点的响应值的对比提取特征点包括：Further, extracting feature points by comparing the response values of points in each response map in the scale space pyramid includes:
尺度空间金字塔采用3组，每组4层；在每一组中选取相邻的三层响应图像，即第一组的第一二三层和第二三四层、第二组的第一二三层和第二三四层、第三组的第一二三层和第二三四层，对于中间一层的每一个点，在空间中选取该点周围的26个点与之比较响应值大小，若该点响应值大于其他26个点，则该点为特征点。The scale space pyramid adopts 3 groups, each group has 4 layers; in each group, select the adjacent three-layer response images, that is, the first, second, third, and second, third, and fourth layers of the first group, and the first and second layers of the second group. The third and third four layers, the first two, three, and the third and third floors of the third and third groups. For each point of the middle layer, select the 26 points around the point in the space to compare the response value If the response value of this point is greater than the other 26 points, the point is a feature point.
进一步的，将两幅图像中的各特征点进行匹配，将相匹配的两特征点构成匹配点对包括：Further, each feature point in the two images is matched, and the matched two feature points are formed into a matching point pair including:
获取各特征点的BRIEF描述子；Obtain the Brief descriptor of each feature point;
计算各特征点BRIEF描述子的Hamming相似性；Calculate the Hamming similarity of the BRIEF descriptor of each feature point;
取Hamming相似性最高的特征点作为匹配点，这两个特征点构成匹配点对。The feature point with the highest Hamming similarity is taken as the matching point, and these two feature points constitute a matching point pair.
进一步的，获取各特征点的BRIEF描述子包括：Further, obtaining the Brief descriptor of each feature point includes:
在某特征点邻域范围内随机选取N对点对{(Xi,Yi)，1<i≤N}，定义一个2×n的矩阵
用该特征点方向θ，生成旋转矩阵
矩阵S顺时针旋转θ角度后的Sθ＝RθS，Sθ =Rθ S after the matrix S is rotated clockwise by an angle of θ,
对Sθ进行τ测试，生成n位二进制编码作为特征点的BRIEF描述子。Perform τ test on Sθ to generate n-bit binary code as the Brief descriptor of feature points.
进一步的，匹配完成后，统计匹配点对欧式距离的最大值和最小值，根据最大值和最小值设置阈值，剔除欧氏距离大于阈值的匹配点对；再由RANSAC(随机抽样一致)算法对匹配对进行优化，剔除误匹配。Further, after the matching is completed, the maximum and minimum values of the Euclidean distance of the matching point pairs are counted, the threshold is set according to the maximum value and the minimum value, and the matching point pairs whose Euclidean distance is greater than the threshold are eliminated; Matching pairs are optimized to eliminate false matches.
进一步的，判断无人机与特征点对应障碍物是否超出安全距离包括：Further, judging whether the obstacle corresponding to the UAV and the feature point exceeds the safe distance includes:
遍历每对匹配点对KP(i，inedx(j))，KP(i+1，index(j))，分别表示Pi和Pi+1两幅图像的第j对匹配点对，inedx(j)表示在图像中特征点的序号；Pi中以KP(i，inedx(j))为中心的k×k邻域范围的局部图像作为模板tmp1，然后以scale为比例扩大tmp1的尺度；Pi+1中以KP(i+1，inedx(j))为中心的k×k邻域范围的局部图像作为模板tmp2，用模板匹配算法对tmp1、tmp2进行匹配获得相似度TMscale，TMscale值越小则tmp1、tmp2越相似；逐渐增大scale，重复上述匹配步骤，获得TMscale最小时的scale值，记为scalemin，该值可视为前后两图像中该匹配点对代表障碍物的尺寸比(即multiple)；若scalemin大于阈值，则表示该特征点对应的物体距离单目相机小于d。Traverse each pair of matching point pairs KP(i , inedx(j)), KP(i+1, index(j)), which respectively represent the jth matching point pair of the two images Pi and Pi+1 , inedx( j) Represents the serial number of the feature points in the image; the local image of the k×k neighborhood range centered on KP(i, inedx(j)) in Pi is used as the template tmp1, and then the scale of tmp1 is enlarged by scale; P Ini+1 , the local image of the k×k neighborhood range with KP(i+1, inedx(j)) as the center is used as the template tmp2, and the template matching algorithm is used to match tmp1 and tmp2 to obtain the similarity TMscale. The smaller is the more similar tmp1 and tmp2; the scale is gradually increased, and the above matching steps are repeated to obtain the scale value when the TMscale is the smallest, denoted as scalemin, and this value can be regarded as the matching point pair in the front and rear images. The size ratio of the obstacle ( That is, multiple); if scalemin is greater than the threshold, it means that the distance of the object corresponding to the feature point to the monocular camera is less than d.
与现有技术相比，本发明所达到的有益效果是：本发明在计算量上低于SIFT算法和SURF算法，可以集成于飞控芯片性能较高的飞控系统中，而无需将图像传到地面站处理之后再将控制信息发回飞控，或者额外安装计算设备；同时由于计算量低，计算速度也大大加快，实时性强，可以在无人机飞行速度较快时仍然有效，可以减少在遇到突发障碍物时所需的反应时间。Compared with the prior art, the beneficial effects achieved by the present invention are: the present invention is less computationally intensive than the SIFT algorithm and the SURF algorithm, and can be integrated into the flight control system with higher performance of the flight control chip, without the need to transmit images. After processing at the ground station, the control information is sent back to the flight control, or additional computing equipment is installed; at the same time, due to the low amount of calculation, the calculation speed is also greatly accelerated, and the real-time performance is strong, which can still be effective when the drone is flying at a fast speed. Reduce the reaction time required when encountering sudden obstacles.
附图说明Description of drawings
图1为本发明一种基于单目视觉的无人机障碍物识别方法的流程图；Fig. 1 is the flow chart of a kind of UAV obstacle identification method based on monocular vision of the present invention;
图2为本发明用于实现近似高斯滤波的盒子滤波器模板(9×9)；2 is a box filter template (9×9) for realizing approximate Gaussian filtering according to the present invention;
图3为本发明用于实现近似高斯滤波的盒子滤波器模板(15×15)；FIG. 3 is a box filter template (15×15) for realizing approximate Gaussian filtering according to the present invention;
图4为本发明相机移动过程中障碍物的针孔成像示意图。FIG. 4 is a schematic diagram of pinhole imaging of obstacles during the movement of the camera of the present invention.
具体实施方式Detailed ways
下面结合附图对本发明作进一步描述。以下实施例仅用于更加清楚地说明本发明的技术方案，而不能以此来限制本发明的保护范围。The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to illustrate the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.
本发明一种基于单目视觉的无人机障碍物识别方法，参见图1所示，包括以下过程：A monocular vision-based UAV obstacle identification method of the present invention, as shown in FIG. 1 , includes the following processes:
步骤1)：无人机通过固定于机体正前方的单目相机(单目相机光线入路与无人机前进方向一致)采集无人机正前方图像，选取采集到的第i幅图像，并对此图像进行灰度化处理，转换成灰度图Pi，Pi表示第i幅灰度图。Step 1): The UAV collects the image directly in front of the UAV through the monocular camera fixed in front of the body (the light path of the monocular camera is consistent with the advancing direction of the UAV), selects the i-th image collected, and This image is subjected to grayscale processing and converted into a grayscale image Pi, wherePi represents theith grayscale image.
步骤2)：对步骤1)获取的灰度图像Pi进行高斯滤波，目的是减少噪声，并将结果通过步骤3)的处理使获取的特征点具备尺度不变性。Step 2): Gaussian filtering is performed on the grayscale imagePi obtained in step 1) to reduce noise, and the result is processed in step 3) to make the obtained feature points have scale invariance.
高斯滤波的具体方法为：计算步骤1)获取的灰度图像Pi中每一个像素点的Hessian矩阵行列式值。为简化计算提高计算速度，本发明中采用SURF算法使用的盒子滤波器来近似实现高斯滤波，滤波计算后结果作为该像素点的响应值用于下一步计算。The specific method of Gaussian filtering is: calculating the determinant value of theHessian matrix of each pixel point in the grayscale image Pi obtained in step 1). In order to simplify the calculation and improve the calculation speed, in the present invention, the box filter used by the SURF algorithm is used to approximate Gaussian filtering, and the result of the filtering calculation is used as the response value of the pixel point for the next calculation.
Hessian矩阵行列式值的近似计算公式为det(H)＝DxxDyy-(0.9Dxy)2。其中0.9为根据实验所得的经验值。图2三个网格图依次表示Dxx、Dyy和Dxy对应的盒子滤波器(具体为9×9矩阵)，各网格的颜色表示盒子滤波器表示的矩阵中该位置处的权值，Dxx和Dyy对应的盒子滤波器表示的矩阵中，白色部分为1，黑色部分为-2，灰色为0；Dxy对应的盒子滤波器表示的矩阵中，白色部分为1，黑色部分为-1，灰色为0。近似公式中的Dxx、Dyy和Dxy表示相对应的盒子滤波器与灰度图像Pi该像素点进行卷积后的结果。灰度图像Pi全部计算完成后产生的图像即为响应图。The approximate calculation formula of the determinant value of the Hessian matrix is det(H)=Dxx Dyy -(0.9Dxy )2 . Among them, 0.9 is the empirical value obtained according to the experiment. The three grid diagrams in Fig. 2 represent the box filters corresponding to Dxx , Dyy and Dxy in turn (specifically, a 9×9 matrix), and the color of each grid represents the weight at that position in the matrix represented by the box filter. , in the matrix represented by the box filter corresponding to Dxx and Dyy , the white part is 1, the black part is -2, and the gray part is 0; in the matrix represented by the box filter corresponding to Dxy , the white part is 1, and the black part is 1 is -1 and gray is 0. Dxx , Dyy and Dxy in the approximation formula represent the result of convolution of the corresponding box filter and the pixel point of the grayscale image Pi . The image generated after all the calculations of the grayscale imagePi are completed is the response map.
步骤3)：构建尺度空间并提取特征点。Step 3): Construct the scale space and extract feature points.
无人机在接近障碍物的过程中，障碍物在单目相机中的成像尺寸会发生改变，也就是尺度会改变。而想要在不同尺度仍能提取到同一障碍物的特征点并匹配，必须建立图像的尺度空间金字塔。本发明采用不断增大盒子滤波器尺寸的间接方法，求取图像在不同尺度下的响应图，用以构建尺度空间金字塔，通过不同尺度下响应图像的对比，提取特征点。When the UAV approaches the obstacle, the imaging size of the obstacle in the monocular camera will change, that is, the scale will change. In order to extract and match the feature points of the same obstacle at different scales, the scale space pyramid of the image must be established. The invention adopts the indirect method of increasing the size of the box filter continuously, obtains the response maps of the images at different scales, and uses it to construct a scale space pyramid, and extracts feature points by comparing the response images at different scales.
构建尺度空间金字塔的具体方法为：用若干不同尺寸的盒子滤波器对Pi进行滤波后产生的响应图构成一组，若干组响应图构成尺度空间金字塔。The specific method of constructing the scale space pyramid is: use several box filters of different sizes to filter Pi to form a group of response maps, and several groups of response maps form a scale space pyramid.
本发明的尺度空间金字塔采用3组，每组4层。第一组四层图像的盒子滤波器尺寸分别为：9×9，15×15，21×21，27×27。15×15的盒子滤波器黑色、白色区域的长度增加2个像素，以保证一个中心像素的存在，15×15的盒子滤波器可参考图3，其他尺寸的盒子滤波器可参考图2、3的变化规律获得。采用类似的方法来处理其他几组的盒子滤波器尺寸，其方法是将滤波器尺寸增加量翻倍(6，12，24，36)。第二组的滤波器尺寸为15×15，27×27，39×39，51×51。第三组的滤波器尺寸为27×27，51×51，75×75，99×99。(注：各像素点响应值的计算可通过积分图像转化为时间复杂度O(1)的有限次加减运算，盒子滤波器尺寸的增加不会导致计算量增加。有关积分图像请自行查阅本发明不做赘述)。The scale space pyramid of the present invention adopts 3 groups, and each group has 4 layers. The box filter sizes of the first group of four-layer images are: 9×9, 15×15, 21×21, 27×27. The length of the black and white areas of the 15×15 box filter is increased by 2 pixels to ensure For the existence of a central pixel, the 15×15 box filter can be obtained by referring to Figure 3, and the box filters of other sizes can be obtained by referring to the variation rules of Figures 2 and 3. A similar approach is used for the other sets of box filter sizes by doubling the filter size increase (6, 12, 24, 36). The filter sizes of the second group are 15×15, 27×27, 39×39, 51×51. The filter sizes of the third group are 27×27, 51×51, 75×75, 99×99. (Note: The calculation of the response value of each pixel can be converted into a finite number of addition and subtraction operations with time complexity O(1) through the integral image. The invention will not be repeated).
在每一组中选取相邻的三层响应图像，即第一组的第一二三层和第二三四层、第二组的第一二三层和第二三四层、第三组的第一二三层和第二三四层，对于中间一层(第二和三层，第一和四层仅有一层与之相邻可与之比较因此弃之不用)的每一个点，在空间中选取该点周围的26个点(即同一层中周围8个点，邻近两层中与之位置相同的点和相邻8个点，总计26个)与之比较响应值大小，若该点响应值大于其他26个点，则该点为特征点，即Pi中具有鲜明特性的点，该点往往为单目相机中物体成像的轮廓边缘、角点等特征明显的点。Select adjacent three-layer response images in each group, namely the first, second, third and second, third and fourth layers of the first group, the first, second, third and second, third and fourth layers of the second group, and the third group. The first, second, third, and second, third, and fourth layers of , for each point in the middle layer (the second and third layers, the first and fourth layers have only one adjacent layer that can be compared with them, so they are discarded), Select 26 points around the point in the space (that is, 8 points around the same layer, adjacent two layers with the same position and 8 adjacent points, a total of 26 points) and compare the response value with it. The response value of this point is greater than the other 26 points, then this point is a feature point, that is, a point with distinct characteristics in Pi, which is often a point with obvious features such as contour edges and corners of objects imaged by a monocular camera.
步骤4)：计算特征点的方向。Step 4): Calculate the orientation of the feature points.
首先计算步骤3)选出的各个特征点的矩：mpq＝∑x,y∈BxpyqI(x,y)，p,q＝{0,1}，其中B为以特征点为中心半径r范围内的图像区域，半径r具体取值需根据图像大小、计算速度的要求来设定，x,y为区域B范围内特征点的纵横坐标，I(x,y)为该点处图像灰度。然后通过矩找到图像质心：特征点到质心的向量方向即为该特征点的方向θ＝arctan(m01/m10)。First calculate the moment of each feature point selected in step 3): mpq =∑x,y∈B xp yq I(x,y), p,q={0,1}, where B is the feature point is the image area within the range of the center radius r. The specific value of the radius r needs to be set according to the requirements of the image size and calculation speed. Image grayscale at the point. Then find the image centroid by moments: The vector direction from the feature point to the centroid is the direction of the feature point θ=arctan(m01 /m10 ).
步骤5)：采用BRIEF方法计算各个特征点的BRIEF描述子。Step 5): using the Brief method to calculate the Brief descriptor of each feature point.
BRIEF(Binary Robust Independent Elementary Features)方法通过从特征点邻域内随机选择2×n个像素点，两两比较灰度值，根据比较结果生成一个长度为n的二进制字符串，即为该特征点的BRIEF描述子。The BRIEF (Binary Robust Independent Elementary Features) method randomly selects 2 × n pixels from the feature point neighborhood, compares the gray values in pairs, and generates a binary string of length n according to the comparison result, which is the feature point's BRIEF descriptor.
定义在以特征点为中心的s×s邻域(s×s的图像范围内，邻域的范围大小根据图像大小与BRIEF描述子长度设置)P内的τ测试(即比较灰度值产生结果的方法)，并将点对依次两两进行τ测试生成BRIEF描述子：
为了具有旋转不变性，需对BRIEF方法进行一些改进：在某特征点邻域范围内随机选取N对点对{(Xi,Yi)，1<i≤N}，定义一个2×n的矩阵用步骤4计算所得该特征点方向θ，生成旋转矩阵
步骤6)：再次获取单目相机采集到的第i+1幅图像，并转换成灰度图Pi+1，对Pi+1重复上述步骤2)至步骤5)，获得灰度图Pi+1中各特征点以及各特征点BRIEF描述子，进入步骤7)。Step 6): Obtain the i+1 th image collected by the monocular camera again, and convert it into a grayscale image Pi+1 , repeat the above steps 2) to 5) for Pi+1 , and obtain a grayscale image P Each feature point ini+1 and each feature point BRIEF descriptor, go to step 7).
步骤7)：特征点匹配，即找出同一物体在两幅图像中的特征点。Step 7): feature point matching, that is, finding the feature points of the same object in two images.
将步骤3)所获得的Pi中所有特征点作为待匹配特征点队列1，步骤6)所获得的Pi+1中所有特征点作为待匹配特征点队列2。从待匹配特征点队列1取一特征点，将待匹配特征点队列2中各个特征点分别与该特征点对比，对比方法为：Take all feature points in Pi obtained in step 3) as the feature point queue 1 to be matched, and all feature points in Pi+1 obtained in step 6) as the feature point queue 2 to be matched. Take a feature point from the feature point queue 1 to be matched, and compare each feature point in the feature point queue 2 to be matched with the feature point. The comparison method is:
将该特征点BRIEF描述子与待匹配特征点队列2中所有特征点的BRIEF描述子分别按位比较，计算该特征点BRIEF描述子与待匹配特征点队列2中各个特征点BRIEF描述子的Hamming相似性(按位遍历两个特征点的BRIEF描述子，若相同位上两个特征点BRIEF描述子字符相等则计数，遍历完后统计相同位数占总位数的百分比即Hamming相似性)，取待匹配特征点队列2中与该特征点Hamming相似性最高的特征点作为该特征点的匹配点，这两个特征点构成匹配点对，若该匹配点对的Hamming相似性高于阈值(根据图像大小、特征点数目等条件自行设置，目的是减少误匹配，减少下一步骤的计算量)，则匹配成功，否则匹配失败，该特征点无与之匹配的特征点，无论失败还是成功都将两个特征点分别从匹配特征点队列1和2中剔除，然后待匹配特征点队列1再取一特征点，重复上述步骤，直至待匹配特征点队列1和2有一个为空停止。Compare the feature point Brief descriptor with the Brief descriptors of all feature points in the feature point queue 2 to be matched bit by bit, and calculate the Hamming of the feature point Brief descriptor and each feature point Brief descriptor in the feature point queue 2 to be matched. Similarity (traversing the Brief descriptors of two feature points by bit, if the two feature points on the same bit are equal to the Brief descriptor characters, count them. After traversing, count the percentage of the same number of bits to the total number of bits, that is, Hamming similarity), Take the feature point with the highest Hamming similarity with the feature point in the feature point queue 2 to be matched as the matching point of the feature point, these two feature points constitute a matching point pair, if the Hamming similarity of the matching point pair is higher than the threshold ( According to the image size, the number of feature points and other conditions, the purpose is to reduce false matching and reduce the amount of calculation in the next step), then the matching is successful, otherwise the matching fails, the feature point has no matching feature point, whether it fails or succeeds Both feature points are removed from the matching feature point queues 1 and 2 respectively, and then another feature point is taken from the feature point queue 1 to be matched, and the above steps are repeated until one of the feature point queues 1 and 2 to be matched is empty.
匹配完成后，统计匹配点对欧式距离的最大值和最小值，根据最大值和最小值设置阈值，剔除欧氏距离大于阈值的匹配点对(欧式距离大于阈值，即两者相距过远，可认为是误匹配)；再由RANSAC(随机抽样一致)算法(可直接调用openCV的相关方法实现)对匹配对进行优化，剔除误匹配。After the matching is completed, the maximum and minimum values of the Euclidean distance between the matching point pairs are counted, the threshold is set according to the maximum value and the minimum value, and the matching point pairs whose Euclidean distance is greater than the threshold value are eliminated (the Euclidean distance is greater than the threshold value, that is, the distance between the two is too far. It is considered to be a mismatch); then the matching pair is optimized by the RANSAC (Random Sampling Consistency) algorithm (which can be implemented by directly calling the relevant methods of openCV) to eliminate the mismatch.
步骤8)：判断单目相机与特征点对应障碍物是否超出安全距离。Step 8): Determine whether the obstacle corresponding to the monocular camera and the feature point exceeds the safe distance.
在摄像机针孔成像模型下，Pi和Pi+1两帧图像中成像关系如图3所示，其中f表示相机焦距，l1，l2分别表示第一幅和第二幅图像中障碍物成像尺寸，d1，d2分别表示Pi，Pi+1中单目相机与障碍物的距离，L表示障碍物实际尺寸。Under the camera pinhole imaging model, the imaging relationship in the two frames of images Pi and Pi+1 is shown in Figure 3, where f represents the focal length of the camera, and l1 and l2 represent the obstacles in the first and second images, respectively The imaging size of the object, d1 and d2 represent the distance between the monocular camera and the obstacle in Pi and Pi+1 respectively, and L represents the actual size of the obstacle.
由三角形相似定理，可得推得
假设需要检测出距离无人机d(单位m)以内的障碍物，即筛选出距离无人机d(单位m)以内的特征点(特征点往往为物体的轮廓边缘、角点等，因此筛选出符合条件的匹配点对，即可大致获取障碍物的轮廓)，则前后两幅图像障碍物在成像中的尺寸比multiple需满足：
具体判断过程如下：遍历每对匹配点对KP(i，inedx(j))，KP(i+1，index(j))，分别表示Pi和Pi+1两幅图像的第j对匹配点对，inedx(j)表示在图像中特征点的序号；以KP(i，inedx(j))为中心创建模板tmp1，即Pi中以KP(i，inedx(j))为中心的k×k邻域范围的局部图像，k的大小请根据图像大小与特征点的数目设定，建议设置为10的整倍数(非奇数也可以，大致处于中心即可)，然后以scale(初始为1.1)为比例扩大tmp1的尺度(可直接用openCV中相关方法实现)；以KP(i+1，inedx(j))为中心创建模板tmp2(方法同上)，用模板匹配算法对tmp1、tmp2进行匹配获得相似度TMscale(可直接调用openCV的相关方法实现)，TMscale值越小则tmp1、tmp2越相似；逐渐增大scale(每次增加0.1，到1.5为止)，重复上述匹配步骤，获得TMscale最小时的scale值，记为scalemin，该值可视为前后两图像中该匹配点对代表障碍物的尺寸比(即multiple)。若scalemin大于阈值(本例为1.2)，则表示该特征点对应的物体距离单目相机小于d(单位m，本例为5m)。The specific judgment process is as follows: traverse each pair of matching point pairs KP(i, inedx(j)), KP(i+1, index(j)), respectively representing the jth pair of matching of the two images Pi and Pi+1 Point pair, inedx(j) represents the serial number of the feature points in the image; create a template tmp1 with KP(i , inedx(j)) as the center, that is, k in Pi with KP(i, inedx(j)) as the center For local images in the neighborhood of ×k, the size of k should be set according to the size of the image and the number of feature points. It is recommended to set it to an integer multiple of 10 (non-odd numbers are also acceptable, it can be roughly in the center), and then scale (initially is 1.1) To scale up the scale of tmp1 (which can be directly implemented by the related methods in openCV); create a template tmp2 with KP(i+1, inedx(j)) as the center (the method is the same as above), and use the template matching algorithm to tmp1 and tmp2. Match to obtain the similarity TMscale (which can be realized by directly calling the related methods of openCV), the smaller the TMscale value, the more similar tmp1 and tmp2; gradually increase the scale (increase by 0.1 each time until 1.5), repeat the above matching steps to obtain the most TMscale. The hourly scale value, denoted as scalemin, can be regarded as the size ratio (ie multiple) of the obstacles represented by the matching point pair in the two images before and after. If scalemin is greater than the threshold (1.2 in this example), it means that the distance to the object corresponding to the feature point is less than d (unit m, 5m in this example) from the monocular camera.
若无人机飞行速度较慢，用于计算的芯片性能较高导致两帧图像的摄取间隔较短，或者设置的安全距离较长，multiple数值过低，则可采取以下改进：获取一副图像Pk并提取特征点，然后连续获取Pk+1，...，Pk+m图像的特征点；将Pk和Pk+m的特征点进行匹配。要检测出距离无人机dm以内的障碍物，需满足其中m根据v、t、d的大小设置，例如速度v为1m/s，前后两幅摄取间隔t为0.5s，安全距离d为10m，则m可设置为4，最终将Pk和Pk+4作为输入，通过步骤7、步骤8，当scalemin大于1.2时，特征点对应障碍物距离在10m范围内；然后获取Pk+m+1图像特征点，与Pk+1的特征点进行匹配，再通过步骤7、8，当scalemin大于1.2时，特征点对应障碍物距离在10m范围内，之后以此类推。If the flying speed of the drone is slow, the performance of the chip used for calculation is high, resulting in a short capturing interval between two frames of images, or the set safe distance is long and the multiple value is too low, the following improvements can be taken: Obtain an image Pk and extract feature points, and then continuously obtain the feature points of Pk+1 , . . . , Pk+m images; match the feature points of Pk and Pk+m . To detect obstacles within dm from the UAV, it is necessary to meet the Among them, m is set according to the size of v, t, and d. For example, the speed v is 1m/s, the interval t between the front and rear images is 0.5s, and the safety distance d is 10m, then m can be set to 4, and finally Pk and Pk +4 is used as input, through steps 7 and 8, whenscalemin is greater than1.2 , the distance between the corresponding obstacles of the feature points is within 10m; After matching, go through steps 7 and 8. When the scalemin is greater than 1.2, the distance between the corresponding obstacle of the feature point is within 10m, and so on.
以上所述仅是本发明的优选实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明技术原理的前提下，还可以做出若干改进和变型，这些改进和变型也应视为本发明的保护范围。The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the technical principles of the present invention, several improvements and modifications can also be made. These improvements and modifications It should also be regarded as the protection scope of the present invention.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910963529.6ACN110689578A (en) | 2019-10-11 | 2019-10-11 | An obstacle recognition method for UAV based on monocular vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910963529.6ACN110689578A (en) | 2019-10-11 | 2019-10-11 | An obstacle recognition method for UAV based on monocular vision |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110689578Atrue CN110689578A (en) | 2020-01-14 |
Family
ID=69112116
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910963529.6APendingCN110689578A (en) | 2019-10-11 | 2019-10-11 | An obstacle recognition method for UAV based on monocular vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110689578A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112171675A (en)* | 2020-09-28 | 2021-01-05 | 深圳市丹芽科技有限公司 | Obstacle avoidance method and device for mobile robot, robot and storage medium |
| CN112363528A (en)* | 2020-10-15 | 2021-02-12 | 北京理工大学 | Unmanned aerial vehicle anti-interference cluster formation control method based on airborne vision |
| CN113391642A (en)* | 2021-05-28 | 2021-09-14 | 西南交通大学 | Unmanned aerial vehicle autonomous obstacle avoidance method and system based on monocular vision |
| CN113486907A (en)* | 2021-07-12 | 2021-10-08 | 深圳慧源创新科技有限公司 | Unmanned equipment obstacle avoidance method and device and unmanned equipment |
| WO2021217451A1 (en)* | 2020-04-28 | 2021-11-04 | 深圳市大疆创新科技有限公司 | Unmanned aerial vehicle control method, motion information determination method and device, and unmanned aerial vehicle |
| CN114715390A (en)* | 2022-06-07 | 2022-07-08 | 西华大学 | Auxiliary drone, emergency rescue system and emergency rescue method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104850851A (en)* | 2015-04-22 | 2015-08-19 | 福州大学 | ORB feature point matching method with scale invariance |
| US20180314903A1 (en)* | 2017-05-01 | 2018-11-01 | Intel Corporation | Optimized image feature extraction |
| CN108805812A (en)* | 2018-06-04 | 2018-11-13 | 东北林业大学 | Multiple dimensioned constant ORB algorithms for image mosaic |
| CN109903338A (en)* | 2019-03-14 | 2019-06-18 | 中国计量大学 | A Mobile Robot Localization Method Based on Improved ORB Algorithm |
- 2019
- 2019-10-11CNCN201910963529.6Apatent/CN110689578A/enactivePending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104850851A (en)* | 2015-04-22 | 2015-08-19 | 福州大学 | ORB feature point matching method with scale invariance |
| US20180314903A1 (en)* | 2017-05-01 | 2018-11-01 | Intel Corporation | Optimized image feature extraction |
| CN108805812A (en)* | 2018-06-04 | 2018-11-13 | 东北林业大学 | Multiple dimensioned constant ORB algorithms for image mosaic |
| CN109903338A (en)* | 2019-03-14 | 2019-06-18 | 中国计量大学 | A Mobile Robot Localization Method Based on Improved ORB Algorithm |
Non-Patent Citations (2)
| Title |
|---|
| 卢键 等: "结合尺度不变特征的ORB算法改进", 《测控技术》* |
| 王新东: "基于机器视觉的无人机避障技术研究", 《中国优秀硕士学位论文全文数据库 工程科技II辑》* |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021217451A1 (en)* | 2020-04-28 | 2021-11-04 | 深圳市大疆创新科技有限公司 | Unmanned aerial vehicle control method, motion information determination method and device, and unmanned aerial vehicle |
| CN112171675A (en)* | 2020-09-28 | 2021-01-05 | 深圳市丹芽科技有限公司 | Obstacle avoidance method and device for mobile robot, robot and storage medium |
| CN112171675B (en)* | 2020-09-28 | 2022-06-10 | 深圳市丹芽科技有限公司 | Obstacle avoidance method and device for mobile robot, robot and storage medium |
| CN112363528A (en)* | 2020-10-15 | 2021-02-12 | 北京理工大学 | Unmanned aerial vehicle anti-interference cluster formation control method based on airborne vision |
| CN113391642A (en)* | 2021-05-28 | 2021-09-14 | 西南交通大学 | Unmanned aerial vehicle autonomous obstacle avoidance method and system based on monocular vision |
| CN113486907A (en)* | 2021-07-12 | 2021-10-08 | 深圳慧源创新科技有限公司 | Unmanned equipment obstacle avoidance method and device and unmanned equipment |
| CN114715390A (en)* | 2022-06-07 | 2022-07-08 | 西华大学 | Auxiliary drone, emergency rescue system and emergency rescue method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110020651B (en) | License plate detection and positioning method based on deep learning network | |
| CN111898621B (en) | A Contour Shape Recognition Method | |
| CN110689578A (en) | An obstacle recognition method for UAV based on monocular vision | |
| CN109308483B (en) | Dual-source image feature extraction and fusion identification method based on convolutional neural network | |
| CN107851195B (en) | Object Detection Using Neural Networks | |
| CN109903299B (en) | Registration method and device for heterogenous remote sensing image of conditional generation countermeasure network | |
| CN108334881B (en) | License plate recognition method based on deep learning | |
| CN116342894B (en) | GIS infrared feature recognition system and method based on improved YOLOv5 | |
| CN104809443A (en) | Convolutional neural network-based license plate detection method and system | |
| Lai et al. | Traffic Signs Recognition and Classification based on Deep Feature Learning. | |
| CN107464252A (en) | A kind of visible ray based on composite character and infrared heterologous image-recognizing method | |
| CN113888461B (en) | Small hardware defect detection method, system and equipment based on deep learning | |
| CN109948457B (en) | Real-time object recognition method based on convolutional neural network and CUDA acceleration | |
| CN108681718A (en) | A kind of accurate detection recognition method of unmanned plane low target | |
| CN105469039A (en) | Target identification system based on AER image sensor | |
| Hu et al. | A video streaming vehicle detection algorithm based on YOLOv4 | |
| CN112417931A (en) | A method for detection and classification of water surface objects based on visual saliency | |
| CN115690401A (en) | Improved rotating frame target detection method fusing CIOU adaptive sample allocation | |
| Khellal et al. | Pedestrian classification and detection in far infrared images | |
| CN103971096A (en) | Multi-pose face recognition method based on MB-LBP features and face energy diagram | |
| CN108614996A (en) | A kind of military ships based on deep learning, civilian boat automatic identifying method | |
| CN110188646A (en) | Human Ear Recognition Method Based on Fusion of Gradient Orientation Histogram and Local Binary Pattern | |
| Watanabe et al. | Distance to center of mass encoding for instance segmentation | |
| CN111079585B (en) | Pedestrian re-identification method combining image enhancement with pseudo-twin convolutional neural network | |
| CN112464982A (en) | Target detection model, method and application based on improved SSD algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | Address after:No.66 Xinfan Road, Gulou District, Nanjing City, Jiangsu Province Applicant after:NANJING University OF POSTS AND TELECOMMUNICATIONS Address before:210023 No.1 Xichun Road, Yuhuatai District, Nanjing City, Jiangsu Province Applicant before:NANJING University OF POSTS AND TELECOMMUNICATIONS | |
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20200114 | |
| RJ01 | Rejection of invention patent application after publication |