CN108243623B - Automobile anti-collision early warning method and system based on binocular stereo vision - Google Patents
Automobile anti-collision early warning method and system based on binocular stereo visionDownload PDFInfo
- Publication number
- CN108243623B CN108243623BCN201680001426.6ACN201680001426ACN108243623BCN 108243623 BCN108243623 BCN 108243623BCN 201680001426 ACN201680001426 ACN 201680001426ACN 108243623 BCN108243623 BCN 108243623B
- Authority
- CN
- China
- Prior art keywords
- straight line
- map
- image
- points
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Description
Technical Field
The present invention relates generally to automotive autopilot technology, and more particularly to automotive anti-collision technology based on binocular stereo vision.
Background
The accurate real-time anti-collision early warning has important application significance, and especially plays a decisive role in assisting driving safety warning and automatic control of automatic driving, for example, in the automatic driving, the anti-collision early warning can reduce accidents as much as possible and avoid personal and property loss; in automatic driving, the more accurate the anti-collision early warning is, the higher the safety is.
At present, the anti-collision early warning method mainly comprises the steps that firstly, calibration is carried out on the basis of a laser radar sensor or a millimeter wave radar, and an area lower than a certain threshold value is judged to be the ground, the method needs the laser radar with high cost and is difficult to popularize and use, and the millimeter wave precision is far lower than that of the laser radar; secondly, a monocular color camera is used for detecting a front obstacle through a method of machine learning and computer vision, the method depends heavily on training samples and characteristics of artificial design, driving areas are different, the obstacle cannot be detected under the condition that the training samples do not exist, expansibility and universality are not strong, on the other hand, the monocular camera cannot accurately acquire depth information, the obtained result does not accord with a real scene, and finally real-time performance of the method is difficult to guarantee.
In recent years, some automobile safe driving technologies based on machine vision (including monocular vision and stereoscopic vision) have been proposed.
Patent document 1CN101135558B discloses an automobile anti-collision early warning technology based on machine vision, in which a machine vision method is used to collect the characteristics of the license plate of the front vehicle and the lane line information, the distance between the front vehicle and the front vehicle is calculated according to the size of the projection imaging pixel points of the license plate of the front vehicle in the machine vision, the driving state of the front vehicle is calculated by combining the state information of the vehicle speed, steering and the like of the front vehicle, and whether the front vehicle is driven in a safe lane range or not is judged according to the relative distance between the vehicle and the lane line boundary.
Patent document 1 uses a machine learning method for detection, which depends heavily on training samples and characteristics of manual design, and the scene areas encountered during driving are very different, and cannot be detected when no training sample exists, so that the expansibility and the universality are not strong; on the other hand, the monocular camera cannot accurately acquire depth information and speed information, and the obtained result often does not conform to a real scene; finally, the real-time performance of the method is difficult to guarantee.
Patent document 2CN102685516A discloses an active safety type driving assistance method based on a stereoscopic vision technique. The active safety type auxiliary driving system comprehensively utilizes an optical-electro-mechanical information technology, consists of a stereoscopic vision subsystem, an image rapid processing subsystem and a safety auxiliary driving subsystem, and comprises two high-resolution CCD cameras, an ambient light illumination sensor, a two-channel video acquisition card, a synchronous controller, a data transmission circuit, a power supply circuit, an image rapid processing algorithm library, a voice reminding module, a screen display module, an active safety driving control module and the like. Under various weather conditions, parameters such as a lane line, relative distance, relative speed, relative acceleration and the like of dangerous targets such as front vehicles, bicycles, pedestrians and the like are identified in real time, and the response measures taken by a driver are prompted through voice, so that automatic deceleration and emergency braking are realized under an emergency condition, and the driving safety is ensured all day long.
Patent document 2 also detects by using a recognition method based on machine learning, and depends heavily on training samples and features of manual design, so that scene areas encountered during driving are very different, and cannot be detected when training samples do not exist, and thus, expansibility and universality are not strong, but a more accurate result can be achieved by binocular in distance calculation.
The automobile anti-collision early warning technology based on the machine vision is required to be stronger in universality and real-time performance and less dependent on training data and artificial design features.
Disclosure of Invention
The present invention has been made in view of the above circumstances.
According to an aspect of the present invention, there is provided an automobile anti-collision early warning method based on binocular stereo vision, which may include: shooting through a binocular camera carried on an automobile body to obtain a left gray image and a right gray image in front of the automobile along the automobile advancing direction, and calculating to obtain a parallax image; converting the disparity map into a V disparity map; carrying out binarization on the V disparity map; fitting points of the binarized V disparity map by using a RANSAC method to obtain a segmented straight line;
smoothing a filtering straight line according to the multi-frame image; obtaining a travelable region in the original gray image through the extracted straight line; calculating the three-dimensional coordinates of points belonging to the ground in a real world coordinate system according to the original image and the disparity map, assuming that the ground is a plane model, and fitting the plane by using RANSAC to obtain a ground model; converting the whole scene in the original gray level image from camera coordinates to world coordinates, generating a plan view at the same time, and solving an occupation map from the plan view; the position of each obstacle is obtained by dividing the occupied map through a connected domain mark detection algorithm, the obstacle is converted into an original image and marked, and the distance from the obstacle to the vehicle is calculated through a disparity map; and when the distance between the current vehicles is less than a certain threshold value, alarming or transmitting the distance into a decision module to participate in decision.
According to the above-mentioned anti-collision warning method for an automobile, the obtaining of the occupancy map from the plan view may include: firstly, extracting points higher than a first threshold height of the ground according to the world coordinates of each point, and converting the points into a ground coordinate system; marking points above a first threshold height and below a second threshold height in an occupancy map; the value of each pixel in the occupancy map is the sum of the heights of its corresponding points, thereby resulting in the occupancy map.
According to the above anti-collision warning method for an automobile, the obtaining the position of each obstacle by dividing the occupied map through a connected domain mark detection algorithm may include: and converting each pixel value in the occupied map into an image with a mark in color according to the size of the pixel value, wherein the larger the pixel value is, the more red the pixel value is, and the smaller the pixel value is, the more blue the pixel value is, and different objects are segmented by a connected domain mark detection algorithm, so that the position of each obstacle is obtained.
According to the above automobile anti-collision warning method, the binarizing the V disparity map may include: and (4) solving the maximum value of the pixel values of each row, setting the gray value of the pixel where only the maximum value is located in each row to be 255, and setting the gray values of the rest pixels to be 0.
According to the automobile anti-collision warning method, fitting a segment of a straight line by using a RANSAC method may include: the following sequence of operations is repeatedly performed until a predetermined end criterion is reached: selecting a group of random subsets in the maximum value points in the V disparity map to perform straight line fitting to obtain a straight line model; using the obtained linear model to test all other data, if a certain point is suitable for the estimated linear model, considering it as an intra-office point, if more than a predetermined number of points are classified as intra-office points, then considering the estimated model as reasonable, then using all intra-office points to re-estimate the model, and estimating the error rate of the intra-office points and the model; if the error rate of the model is lower than that of the best model, replacing the best model with the model; and taking the best model obtained finally as the segmentation straight line.
According to the above anti-collision warning method for an automobile, the fitting a multi-segment straight line by using the RANSAC method may include: first, a first straight line is extracted according to the method, after extraction is completed, points belonging to the first straight line are removed from the V disparity map, then, a second straight line is extracted according to the method of claim 5 for the rest points, and the steps are repeated until the number of the remaining points is smaller than a preset threshold value.
According to the automobile anti-collision early warning method, smoothing the filtering straight line according to the multi-frame image may include: setting a time window, assuming that a linear model is represented as ax + by + c being 0, obtaining linear model parameters for each frame of image, accumulating each frame of image according to each parameter, subtracting the linear model parameters of the initial frame of image from the accumulated parameter result when a new image comes, adding the linear model parameters of the current frame of image, and then averaging to obtain the linear model parameters of the frame.
According to the automobile anti-collision early warning method, the step of obtaining the travelable area in the original gray level image through the extracted straight line comprises the following steps: and selecting a point with a parallax value d on the extracted straight line aiming at each row in the V parallax map, comparing the parallax value of each pixel with the difference value of d in the row corresponding to the parallax map, and judging the corresponding position of the original map as a safe travelable area when the difference value is less than a certain threshold value.
According to another aspect of the present invention, there is provided an automobile anti-collision warning system based on binocular stereo vision, which may include: the binocular camera is used for continuously shooting to obtain a left gray image and a right gray image in front of the automobile along the automobile driving direction; the computing device comprises a memory, a processor, a communication interface and a bus, wherein the memory, the communication interface and the processor are connected to the bus, computer-executable instructions are stored in the memory, the computing device can obtain left and right gray-scale images shot by the binocular camera through the communication interface, and when the processor executes the computer-executable instructions, the following method is executed: calculating to obtain a disparity map based on the left and right gray level images; converting the disparity map into a V disparity map; carrying out binarization on the V disparity map; fitting points of the binarized V disparity map by using a RANSAC method to obtain a segmented straight line; smoothing a filtering straight line according to the multi-frame image; obtaining a travelable region in the original gray image through the extracted straight line; calculating the three-dimensional coordinates of points belonging to the ground in a real world coordinate system according to the original image and the disparity map, assuming that the ground is a plane model, and fitting the plane by using RANSAC to obtain a ground model; converting the whole scene in the original gray level image from camera coordinates to world coordinates, generating a plan view at the same time, and solving an occupation map from the plan view; the position of each obstacle is obtained by dividing the occupied map through a connected domain mark detection algorithm, the obstacle is converted into an original image and marked, the distance from the obstacle to the vehicle is calculated through a disparity map, and when the distance between the vehicle and the obstacle is smaller than a certain threshold value, an alarm is given or the distance is transmitted into a decision module to participate in decision.
According to the above-mentioned anti-collision warning system for an automobile, the obtaining of the occupancy map from the plan view may include: firstly, extracting points higher than a first threshold height of the ground according to the world coordinates of each point, and converting the points into a ground coordinate system; marking points above a first threshold height and below a second threshold height in an occupancy map; the value of each pixel in the occupancy map is the sum of the heights of its corresponding points, thereby resulting in the occupancy map.
According to the above vehicle anti-collision warning system, the segmenting the occupied map by the connected component detection algorithm to obtain the position of each obstacle may include: and converting each pixel value in the occupied map into an image with a mark in color according to the size of the pixel value, wherein the larger the pixel value is, the more red the pixel value is, and the smaller the pixel value is, the more blue the pixel value is, and different objects are segmented by a connected domain mark detection algorithm, so that the position of each obstacle is obtained.
According to the automobile anti-collision early warning system, the binarization of the V disparity map comprises the following steps: and (4) solving the maximum value of the pixel values of each row, setting the gray value of the pixel where only the maximum value is located in each row to be 255, and setting the gray values of the rest pixels to be 0.
According to the above anti-collision warning system for an automobile, fitting a segment of a straight line by using the RANSAC method may include: the following sequence of operations is repeatedly performed until a predetermined exit criterion is reached: selecting a group of random subsets in the maximum value points in the V disparity map to perform straight line fitting to obtain a straight line model; using the obtained linear model to test all other data, if a certain point is suitable for the estimated linear model, considering it as an intra-office point, if more than a predetermined number of points are classified as intra-office points, then considering the estimated model as reasonable, then using all intra-office points to re-estimate the model, and estimating the error rate of the intra-office points and the model; if the error rate of the model is lower than that of the best model, replacing the best model with the model; and taking the best model obtained finally as the segmentation straight line.
According to the above anti-collision warning system for an automobile, the fitting a multi-segment straight line by using the RANSAC method may include: and extracting a first straight line, removing points belonging to the first straight line from the V disparity map after extraction is finished, extracting a second straight line aiming at the rest points, and repeating the steps until the number of the rest points is less than a preset threshold value.
According to the above automobile anti-collision warning system, smoothing the filtering line according to the multi-frame image may include: setting a time window, assuming that a linear model is represented as ax + by + c being 0, obtaining linear model parameters for each frame of image, accumulating each frame of image according to each parameter, subtracting the linear model parameters of the initial frame of image from the accumulated parameter result when a new image comes, adding the linear model parameters of the current frame of image, and then averaging to obtain the linear model parameters of the frame.
According to the above anti-collision warning system for an automobile, the obtaining of the travelable region in the original gray level image from the extracted straight line may include: and selecting the parallax value on the extracted straight line as d for each line in the V parallax image, comparing the parallax value of each pixel with the difference value of d in the corresponding line in the parallax image, and judging the corresponding position of the original image as a safe drivable area when the difference value is smaller than a certain threshold value.
According to still another aspect of the present invention, an anti-collision warning system for an automobile may include: the binocular camera is configured to shoot a left gray image and a right gray image in front of the automobile along the automobile traveling direction; a parallax map calculation unit which calculates a parallax map from the left and right two gray images; the V disparity map conversion module is used for converting the disparity map to obtain a V disparity map; the binarization module is used for binarizing the V disparity map; the RANSAC straight line fitting module is used for fitting points of the binarized V disparity map by using an RANSAC method to obtain a segmented straight line; the multi-frame image filtering module is used for smoothing filtering straight lines according to multi-frame images; the original image travelable area determining module is used for obtaining travelable areas in the original gray level image through the extracted straight lines; the ground model fitting module is used for calculating the three-dimensional coordinates of points belonging to the ground in a real world coordinate system according to the original image and the disparity map, assuming that the ground is a plane model, and fitting the plane by using RANSAC to obtain a ground model; the occupied map calculation module converts the whole scene in the original gray level image from camera coordinates to world coordinates, generates a plane graph at the same time, and calculates the occupied map from the plane graph; the obstacle segmentation and distance calculation module is used for obtaining the position of each obstacle from the occupied map through segmentation of a connected domain mark detection algorithm, converting the position of each obstacle into an original image to be marked, and calculating the distance from the obstacle to the vehicle through a parallax map; and the alarm module is used for giving an alarm when the distance between the current vehicles is less than a certain threshold value or transmitting the distance into the decision module to participate in decision making.
According to the automobile anti-collision early warning method and system based on the stereoscopic vision, the ground model is obtained based on the binocular vision image, the occupied map of the scene is obtained through the ground model, the anti-collision information is obtained through the occupied map, the method and system can adapt to various road surfaces and road conditions, the algorithm has low requirement on the disparity map precision, the front-end calculation amount is reduced, the anti-interference capability is strong, and the influence caused by data and artificial design characteristics is not relied on.
Drawings
These and/or other aspects and advantages of the present invention will become more apparent and more readily appreciated from the following detailed description of the embodiments of the invention, taken in conjunction with the accompanying drawings of which:
fig. 1 shows a schematic diagram of an in-vehicle automotive collisionavoidance warning system 100, according to an embodiment of the present invention;
fig. 2 is a general flowchart illustrating a binocular stereo vision-based automobile anti-collision warning method according to an embodiment of the present invention;
FIG. 3 is a diagram showing a case where a least square method erroneously extracts straight lines in the presence of large noise;
FIG. 4 shows a flow diagram of amethod 240 of fitting a straight line from points of a V disparity map, according to an embodiment of the invention;
fig. 5 is a block diagram illustrating a collisionavoidance warning system 300 for a vehicle according to another embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the invention, further details are provided below in conjunction with the accompanying drawings and the detailed description of the invention.
An explanation is first given of the terms used herein.
Parallax map: the disparity map is an image whose element value is a disparity value, and whose size is the size of the reference image, with reference to any one of the pair of images. The disparity map contains distance information of the scene. The disparity map may be calculated from left and right images captured by a binocular camera. The coordinate of a certain point in the common two-dimensional disparity map is represented by (u, v), wherein u is an abscissa, and v is an ordinate; the pixel value of the pixel at the point (u, v) is denoted by d (u, v), and the pixel value represents the parallax at the point (u, v). Since the disparity map includes distance information of a scene, image matching for extracting the disparity map from a stereo image pair has been the most active field in binocular vision research.
V disparity map: the V disparity map is obtained by converting the disparity map, and the gray value of any point (d, V) in the V disparity map is the number of points with the disparity value equal to d in a line with the vertical coordinate V corresponding to the disparity map. In a pictographic sense, the V disparity map can be considered as a side view of the disparity map. And projecting the planes in the original image into a straight line by accumulating the number of the same parallax values of the same line.
RANSAC, an abbreviation of RANdom Sample Consensus, which is an algorithm for calculating mathematical model parameters of data according to a group of Sample data sets containing abnormal data to obtain effective Sample data.
Occupancy map (occupancy map): in early vision systems, knowledge of the environment was represented by a grid in which 2D projection information of each object in the environment onto the ground was preserved, such a representation being referred to as an occupancy map.
Binocular stereoscopic vision: binocular Stereo Vision (Binocular Stereo Vision) is an important form of machine Vision, and is a method for acquiring three-dimensional geometric information of an object by acquiring two images of the object to be measured from different positions by using imaging equipment based on a parallax principle and calculating position deviation between corresponding points of the images. Images obtained by two eyes are fused and the difference between the images is observed, so that the user can obtain obvious depth feeling, the corresponding relation between the characteristics is established, mapping points of the same space physical point in different images are corresponded, and the difference is called as a parallax (Disparity) image. The binocular stereo vision measuring method has the advantages of high efficiency, proper precision, simple system structure, low cost and the like. In the measurement of moving objects (including animal and human bodies), the stereoscopic vision method is a more effective measurement method because image acquisition is completed in a moment. The binocular stereo vision system is one of key technologies of computer vision, and the distance information for acquiring the spatial three-dimensional scene is also the most basic content in computer vision research.
Fig. 1 shows a schematic diagram of a vehicle-mountedsystem 100 for detecting a drivable area of a vehicle according to an exemplary embodiment of the present disclosure, comprising a binocular camera 110 and a computing device 120.
The binocular camera 110 continuously captures left and right two gray images in front of the vehicle in the traveling direction of the vehicle.
The binocular camera 110 is mounted, for example, in front of the top of the vehicle so that its imaging range is focused on the road surface in front of the vehicle.
Computing device 120 includes memory 121, processor 122, communication interface 123,bus 124. The memory 121, the communication interface 123 and the processor 122 are all connected to thebus 124, the memory stores computer executable instructions, the computing device can obtain the left and right two gray images shot by the binocular camera through the communication interface, and when the processor executes the computer executable instructions, the method for the anti-collision warning of the automobile is executed.
An alarm 125 may also be included in the computing device 120 for giving alarm signals or sending out notifications when a dangerous or emergency situation is discovered.
The structure shown in fig. 1 is merely an example, and addition, subtraction, replacement, and the like may be performed as necessary.
In addition, it should be noted that some functions or some of the functions may be implemented by different components as needed, for example, calculating the disparity map from the left and right images is described as being implemented by a computing device in the embodiment, but software, hardware or firmware for calculating the disparity map may be added to the binocular camera as needed, or a special component for calculating the disparity map based on the left and right images may be disposed in the vehicle, which is within the scope of the present invention.
The method for detecting the driving area of the vehicle in real time according to the embodiment of the invention is described in detail with reference to fig. 2.
The technology for detecting the automobile drivable area in real time comprises the steps of obtaining a left image and a right image through a binocular camera sensor, obtaining a disparity map (disparity map) from the left image and the right image, constructing a V-disparity map (V-disparity map) by using the disparity map, obtaining segmented straight lines on the V disparity map by using RANSAC, performing smooth filtering on the straight lines according to multi-frame images, and finally obtaining a drivable safety area in an original image by using the disparities corresponding to the straight lines.
Fig. 2 shows a general flowchart of amethod 200 for detecting a drivable area of a motor vehicle in real time according to an exemplary embodiment of the invention.
In step S210, two left and right grayscale images in front of the vehicle in the vehicle traveling direction are captured by a binocular camera mounted on the vehicle body, and a parallax map is calculated.
Specifically, for example, according to a binocular stereo matching correlation algorithm, a corresponding relationship between each pair of images is first found, and a disparity map of a current scene is obtained according to a triangulation principle.
Here, some denoising processing and the like may also be performed on the disparity map.
In step S220, a V disparity map is converted from the disparity map.
Specifically, for example, in a disparity map, the relative distance of an object with respect to a lens is represented by a change in grayscale depth, and the disparity on the ground is continuously changed according to depth information included in the disparity map, approximating a piecewise straight line. Assume using MdThe pixel value representing a point on the disparity map is represented by MvdRepresenting the pixel values of the corresponding points on the V-disparity map. Using function f (M)d)＝MvdTo represent the conversion relationship between the disparity map and the V disparity map, and the function f represents the number P of pixels with the same disparity on each line of the cumulative disparity mapnumThus, the horizontal axis represents parallax, and the vertical axis represents parallax map, PnumIs the gray value of the corresponding pixel, thus obtaining a gray V disparity map.
In step S230, the V disparity map is binarized.
In one example, binarization is performed using the following method: the principle of binarization is to first find the maximum value of each line, the gray value of the pixel where only the maximum value is located in each line is set to be 255, and the gray values of the rest pixels are set to be 0.
In step S240, a segment straight line is fitted from the points of the binarized V disparity map using the RANSAC method.
The following explains why the embodiment of the present invention selects to use the RANSAC method to perform the straight line fitting of the points of the binarized V disparity map among numerous straight line fitting algorithms.
Data in real life often has a certain bias, or noise, which makes mathematical fitting difficult. For example, we know that there is a linear relationship between two variables X and Y, Y ═ aX + b, and we want to determine the specific values of parameters a and b. Through experiments, a set of test values of X and Y can be obtained. Although theoretically the equations of two unknowns only need two sets of values to be confirmed, due to systematic errors, the values of a and b calculated by taking any two points are different. It is desirable that the final calculated theoretical model have minimal error from the test values.
Typically the prior art uses a least squares method or hough transform to fit a straight line.
The disadvantages of the hough transform are: the detection speed is too slow to realize real-time control; the accuracy is not high enough, and the expected information cannot be detected but an error judgment is made, so that a large amount of redundant data is generated. This is mainly due to:
1. a large amount of memory space is occupied, the time is long, and the real-time performance is poor;
2. in reality, images are generally interfered by external noise, the signal to noise ratio is low, the performance of conventional Hough transformation is sharply reduced at the moment, and the problems of 'false peak' and 'missing detection' often occur due to the fact that a proper threshold value is difficult to determine when the maximum value of the parameter space is searched.
The least squares method calculates the value at which the partial derivative of the minimum mean square error with respect to the parameters a, b is zero. In fact, in many cases, the least squares method is a synonym for linear regression. Unfortunately, the least squares method is only suitable for small errors. In an attempt to try this, the least squares method is not satisfactory if the model needs to be extracted from a noisy data set (say, only 20% of the data is fit). For example, in fig. 3, a straight line (pattern) can be easily seen by the naked eye, but the least squares method is wrong.
The road surface is detected by extracting straight lines from the V disparity map, the disparity map has large noise, and in the case that the straight lines are extracted by the least square method, wrong fitting is likely to be obtained.
The RANSAC algorithm can estimate the parameters of a mathematical model from a group of observation data sets comprising 'local outliers' in an iterative manner, and is very suitable for model parameter estimation of observation data containing more noise. In practical applications, the acquired data often contains noise data which can interfere with model construction, the noise data points are called outliers (local points), the noise data points which play a positive role in model construction are called inliers (local points), one thing which is done by RANSAC is to randomly select some points, use the points to obtain a model (if a straight line is fitted, the so-called model is actually a slope), then use the model to test the rest points, if the tested data points are within an error tolerance range, the data points are judged as local points, otherwise, the data points are judged as local points. If the number of the local points reaches a certain set threshold value, the selected data point sets reach an acceptable degree, otherwise, all the steps after the random selection of the point sets are continued, the process is continuously repeated until the selected data point sets reach the acceptable degree, and the obtained model can be regarded as the optimal model construction of the data points.
Fig. 4 shows a flowchart of amethod 240 of fitting a straight line from points of a V-disparity map according to an embodiment of the invention. The method may be used in step S240 in fig. 2.
In step S241, a group of random subsets of the points in the binarized V disparity map is selected for line fitting, so as to obtain a line model.
All other data are tested with the obtained line model in step S242, and if a certain point is suitable for the estimated line model, it is considered to be a local point, and the number of local points is counted.
In step S243, it is determined whether the number of local points is greater than a threshold value, and if the determination result is yes, the process proceeds to step S245, otherwise, the process proceeds to step S244.
In step S244, it is determined that the estimated model is not reasonable, the model is discarded, and the process proceeds to step S249.
In step S245, it is determined that the estimated model is reasonable, then the model is re-estimated with all the intra-office points, and the error rate of the intra-office points and the model is estimated, and then it proceeds to step S246.
In step S246, it is determined whether the error rate of the current estimation model is smaller than that of the optimum model, and if the result is affirmative, it proceeds to step S247, otherwise it proceeds to step S248.
In step S247, the best model is replaced with the currently estimated model, that is, because the currently estimated model has a lower error rate and better performance than the best model according to the determination of step S246, the replaced best model becomes a new best model, and then proceeds to step S249.
In step S248, the estimated model is discarded, and then the process proceeds to step S249.
In step S249, it is determined whether a termination condition is reached, and if the termination condition is reached, the process is terminated, otherwise, it returns to step S241 to be repeatedly executed. The termination condition may be, for example, that the number of iterations reaches a threshold number, that the error rate is lower than a predetermined threshold, or the like.
The method for extracting a straight line from the V-disparity map by using the RANSAC method is described above with reference to fig. 4, where the ground is not a plane and is therefore reflected in the V-disparity map as a plurality of continuous piecewise straight lines, and the piecewise straight line extracting method may be, for example, as follows: first, a first straight line is extracted according to a method such as that described in conjunction with fig. 4, after extraction is completed, a point belonging to the first straight line is removed from the V-disparity map, and then a second straight line is extracted according to the same method for the remaining points, and so on until the number of remaining points is less than a predetermined threshold.
Returning to fig. 2, after completion of step S240, the process proceeds to step S250.
In step S250, the filter straight line is smoothed from the multi-frame image.
As described above, in patent document CN 103489175B, kalman filtering is performed on the fitted straight line.
The inventor considers through experimental analysis that the kalman filtering method performs filtering based on the fact that the change of the processing object is gaussian distribution, but actually the change of the road surface is not gaussian distribution, and in addition, the kalman filtering method is slow to operate, and cannot meet the real-time requirement of detecting the automobile travelable area in the field of automatic driving.
According to the pavement detection technology provided by the embodiment of the invention, a method for smoothing the filtering straight line according to the multi-frame image, which meets the real-time requirement, is designed. Since the gradient of the road surface on which the automobile runs does not change greatly, and the change is uniform and slow, the change of the fitted straight line according to the embodiment of the invention is also uniform. On the other hand, because the disparity map obtained by the binocular camera has much noise, the obtained straight line generates unnecessary jitter. In order to reduce the jitter and take the property that the straight line obtained by the above fitting is uniform and slow into consideration, the embodiment of the invention proposes to perform smoothing filtering on the multi-frame image to obtain a smooth and uniform straight line model.
Specifically, the smoothing filtering of the straight line may be performed from the multi-frame image as follows: setting a time window, assuming that a linear model is represented as ax + by + c being 0, obtaining linear model parameters for each frame of image, accumulating each frame of image according to each parameter, subtracting the linear model parameters of the initial frame of image from the accumulated parameter result when a new image comes, adding the linear model parameters of the current frame of image, and then averaging to obtain the linear model parameters of the frame. For example, when an automobile runs on a road surface and the current time is tc, a current image is obtained by new shooting, at this time, a first frame is removed from a window for a fixed window, then a new image frame is added, and the linear model parameters of the images in the window are averaged to be used as the linear model parameters of the new image frame, namely the estimated mathematical model parameters of the road surface in the V disparity map; then, as time progresses, this operation continues, corresponding to sliding the window forward as time progresses.
In step S260, a travelable region in the original gradation image is obtained from the extracted straight line.
In one example, the travelable region in the original grayscale image can be obtained by the straight line extracted in the V-disparity map as follows: and selecting a point with a parallax value d on the extracted straight line aiming at each row in the V parallax map, comparing the parallax value of each pixel with the difference value of d in the row corresponding to the parallax map, and judging the corresponding position of the original map as a safe travelable area when the difference value is less than a certain threshold value.
The information of the safe driving area is obtained in the gray-scale image, so that key decision information can be provided for auxiliary driving, automatic driving and unmanned driving, collision is prevented, and safety is guaranteed.
Returning to fig. 2, in step S270, three-dimensional coordinates of points belonging to the ground in the real world coordinate system are calculated from the original image and the disparity map, and assuming that the ground is a plane model, the plane is fitted using RANSAC to obtain a ground model.
Specifically, in one example, the ground model is derived by fitting: and calculating the three-dimensional coordinates of the points belonging to the ground in the real world coordinate system according to the original image and the parallax map. In the real world coordinate system, the ground is assumed to be a plane model, denoted as ax + by + cz + d as 0, and then RANSAC is used to fit the plane. RANSAC performs straight line fitting by repeatedly selecting a group of random subsets in maximum value points in ground candidate points, and because a disparity map contains a lot of noise, after RANSAC is performed once, RANSAC is performed again on the obtained point set belonging to the ground to fit a plane, and finally a ground model is obtained.
In step S280, the entire scene in the original grayscale image is converted from the camera coordinates to the world coordinates, and a plan view is generated, from which an occupancy map is obtained.
In one example, the occupancy map is derived by: firstly, extracting points higher than the ground by a certain height, and converting the points into a ground coordinate system; we mark points above and below a certain height in the occupancy map; the value of each pixel in the occupancy map is the sum of the heights of its corresponding points, thereby resulting in the occupancy map.
In step S290, the position of each obstacle is obtained by dividing the occupied map by the connected component detection algorithm, and the image is converted into the original image and marked, and the distance from the obstacle to the host vehicle is calculated by the disparity map.
In one example, each pixel value in the map is converted to a color tagged image according to the size of the pixel value occupying it, with larger pixel values favoring red and smaller pixel values favoring blue. Different objects () are then segmented out by a connected component marker detection algorithm, which may be referred to in the literature, Di Stefano, Luigi, and Andrea burgarelli. "a simple and effective connected components labeling algorithm," Image Analysis and Processing,1999.proceedings. international Conference on. ieee,1999. The specific position of each obstacle is obtained, the specific position is converted into an original image and marked, and the distance from the obstacle to the vehicle is calculated through the parallax map. The relevant distance may be found based on the relationship between disparity and depth.
In step S2901, when the distance between the current vehicle and the vehicle is less than a certain threshold, an alarm is given or the vehicle is transmitted to a decision module to participate in a decision.
Fig. 5 is a block diagram illustrating a real-time automobile travelableregion detection system 300 for detecting a travelable region of an automobile in real time according to another embodiment of the present invention. Thesystem 300 is installed on a vehicle for real-time detection of the drivable area of the vehicle, providing critical support for assisted driving, autonomous driving, and unmanned driving of the vehicle.
As shown in fig. 5, the real-time detection system 300 for a travelable area of an automobile may include: the image processing device comprises abinocular camera 310, a disparitymap calculation part 320, a V disparitymap conversion part 330, abinarization part 340, a RANSAC straight linefitting part 350, a multi-frameimage filtering part 360, an original image travelablearea determination part 370, a ground modelfitting module 380, an occupancymap calculation module 390, an obstacle segmentation anddistance calculation module 391 and analarm module 392.
Thebinocular camera 310 is configured to capture left and right two grayscale images in front of the automobile in the automobile traveling direction. The parallaxmap calculation unit 320 calculates a parallax map from the left and right two grayscale images. The V disparitymap converting section 330 converts the disparity map into a V disparity map. Thebinarization section 340 binarizes the V disparity map. The RANSAC straight linefitting section 350 fits a piecewise straight line from points of the binarized V-disparity map using the RANSAC method. The multi-frameimage filtering section 360 smoothes the filtering straight line from the multi-frame image. The original image travelableregion determining section 370 obtains a travelable region in the original gradation image from the extracted straight line. The ground modelfitting module 380 calculates three-dimensional coordinates of points belonging to the ground in a real world coordinate system according to the original image and the disparity map, assumes that the ground is a plane model, and fits the plane by using RANSAC to obtain a ground model. The occupancymap extraction module 390 converts the entire scene in the original grayscale image from camera coordinates to world coordinates while generating a plan from which the occupancy map is extracted. The obstacle segmentation anddistance calculation module 391 segments the occupied map by a connected domain mark detection algorithm to obtain the position of each obstacle, converts the position of each obstacle into an original image to be marked, and calculates the distance from each obstacle to the vehicle by a parallax map. And thealarm module 392 is used for giving an alarm when the current vehicle distance is less than a certain threshold value or transmitting the current vehicle distance into the decision module to participate in decision making.
For the functions and specific implementation of the disparitymap calculation component 320, the V disparitymap conversion component 330, thebinarization component 340, the RANSAC straight linefitting component 350, the multi-frameimage filtering component 360, the original image travelableregion determination component 370, the ground modelfitting module 380, the occupationmap calculation module 390, the obstacle segmentation anddistance calculation module 391, and thealarm module 392, reference may be made to the description of the corresponding steps in fig. 2, and details are not repeated here.
It should be noted that the binocular camera herein should be understood in a broad sense, and any camera or device having an image capturing function capable of obtaining left and right images may be considered as the binocular camera herein.
It should be understood that the disparitymap calculating unit 320, the V disparitymap converting unit 330, thebinarizing unit 340, the RANSAC linefitting unit 350, the multi-frameimage filtering unit 360, the original image travelableregion determining unit 370, the ground modelfitting module 380, the occupancymap obtaining module 390, the obstacle segmenting and distance calculatingmodule 391, and thealarm module 392 are also broadly implemented, and these components may be implemented by software, firmware, or hardware, or a combination of these, and each component may be combined with each other, sub-combined with each other, or further separated, and the like, and these all fall within the scope of the present disclosure.
The method and the system for detecting the automobile drivable area in real time can adapt to various road surfaces and road conditions, have low requirement on the precision of a parallax map, reduce the front-end calculation amount, have strong anti-interference capability and improve the real-time property, and are very critical to the automatic safe driving of the automobile.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (15)
1. An automobile anti-collision early warning method based on binocular stereo vision comprises the following steps:
shooting through a binocular camera carried on an automobile body to obtain a left gray image and a right gray image in front of the automobile along the automobile advancing direction, and calculating to obtain a parallax image;
converting the disparity map into a V disparity map;
carrying out binarization on the V disparity map;
fitting points of the binarized V disparity map by using a RANSAC method to obtain a segmented straight line;
smoothing filtering straight lines according to multiple frames of images, wherein a time window is set, a straight line model is assumed to be represented as ax + by + c being 0, straight line model parameters are obtained for each frame of image, each frame is accumulated aiming at each parameter, when a new image comes from each frame, the straight line model parameters of the image of the initial frame are subtracted from the accumulated parameter result, the straight line model parameters of the image of the current frame are added, and then the average is obtained to be used as the straight line model parameters of the frame; and
obtaining a travelable region in the original gray image through the extracted straight line;
calculating the three-dimensional coordinates of points belonging to the ground in a real world coordinate system according to the original image and the disparity map, assuming that the ground is a plane model, and fitting the plane by using RANSAC to obtain a ground model;
converting the whole scene in the original gray level image from camera coordinates to world coordinates, generating a plan view at the same time, and solving an occupation map from the plan view;
the position of each obstacle is obtained by dividing the occupied map through a connected domain mark detection algorithm, the obstacle is converted into an original image and marked, and the distance from the obstacle to the vehicle is calculated through a disparity map;
and when the distance between the current vehicles is less than a certain threshold value, alarming or transmitting the distance into a decision module to participate in decision.
2. The automotive collision avoidance warning method of claim 1, wherein the step of determining the occupancy map from the plan view comprises:
firstly, extracting points higher than a first threshold height of the ground according to the world coordinates of each point, and converting the points into a ground coordinate system; marking points above a first threshold height and below a second threshold height in an occupancy map; the value of each pixel in the occupancy map is the sum of the heights of its corresponding points, thereby resulting in the occupancy map.
3. The automobile anti-collision warning method according to claim 1, wherein the segmenting the position of each obstacle from the occupancy map by the connected component marker detection algorithm comprises:
and converting each pixel value in the occupied map into an image with a mark in color according to the size of the pixel value, wherein the larger the pixel value is, the more red the pixel value is, and the smaller the pixel value is, the more blue the pixel value is, and different objects are segmented by a connected domain mark detection algorithm, so that the position of each obstacle is obtained.
4. The automobile anti-collision early warning method according to claim 1, wherein the binarizing the V disparity map comprises:
and (4) solving the maximum value of the pixel values of each row, setting the gray value of the pixel where only the maximum value is located in each row to be 255, and setting the gray values of the rest pixels to be 0.
5. The automotive anti-collision warning method of claim 1, wherein fitting a piecewise straight line using the RANSAC method comprises:
the following sequence of operations is repeatedly performed until a predetermined end criterion is reached:
selecting a group of random subsets in the maximum value points in the V disparity map to perform straight line fitting to obtain a straight line model;
using the obtained linear model to test all other data, if a certain point is suitable for the estimated linear model, considering it as an intra-office point, if more than a predetermined number of points are classified as intra-office points, then considering the estimated model as reasonable, then using all intra-office points to re-estimate the model, and estimating the error rate of the intra-office points and the model;
if the error rate of the model is lower than that of the best model, replacing the best model with the model;
and taking the best model obtained finally as the segmentation straight line.
6. The automobile anti-collision warning method according to claim 5, wherein the fitting the multi-segment linear line by using RANSAC method comprises:
firstly, a first straight line is extracted according to the method of claim 5, after the extraction is finished, the points belonging to the first straight line are removed from the V disparity map, then, a second straight line is extracted according to the method of claim 5 for the rest points, and the steps are repeated until the number of the rest points is less than a preset threshold value.
7. The automobile anti-collision warning method according to any one of claims 1 to 6, wherein the obtaining of the travelable region in the original gray image from the extracted straight lines includes:
and selecting a point with a parallax value d on the extracted straight line aiming at each row in the V parallax map, comparing the parallax value of each pixel with the difference value of d in the row corresponding to the parallax map, and judging the corresponding position of the original map as a safe travelable area when the difference value is less than a certain threshold value.
8. An automobile anti-collision early warning system based on binocular stereo vision, comprising:
the binocular camera is used for continuously shooting to obtain left and right gray level images in front of the automobile along the automobile running direction;
the computing device comprises a memory, a processor, a communication interface and a bus, wherein the memory, the communication interface and the processor are connected to the bus, computer-executable instructions are stored in the memory, the computing device can obtain left and right gray-scale images shot by the binocular camera through the communication interface, and when the processor executes the computer-executable instructions, the following method is executed:
calculating to obtain a disparity map based on the left and right gray level images;
converting the disparity map into a V disparity map;
carrying out binarization on the V disparity map;
fitting points of the binarized V disparity map by using a RANSAC method to obtain a segmented straight line;
smoothing filtering straight lines according to multiple frames of images, wherein a time window is set, a straight line model is assumed to be represented as ax + by + c being 0, straight line model parameters are obtained for each frame of image, each frame is accumulated aiming at each parameter, when a new image comes from each frame, the straight line model parameters of the image of the initial frame are subtracted from the accumulated parameter result, the straight line model parameters of the image of the current frame are added, and then the average is obtained to be used as the straight line model parameters of the frame;
obtaining a travelable region in the original gray image through the extracted straight line;
calculating the three-dimensional coordinates of points belonging to the ground in a real world coordinate system according to the original image and the disparity map, assuming that the ground is a plane model, and fitting the plane by using RANSAC to obtain a ground model;
converting the whole scene in the original gray level image from camera coordinates to world coordinates, generating a plan view at the same time, and solving an occupation map from the plan view;
the position of each obstacle is obtained by dividing the occupied map through a connected domain mark detection algorithm, the obstacle is converted into an original image and marked, the distance from the obstacle to the vehicle is calculated through a disparity map,
and when the distance between the current vehicles is less than a certain threshold value, alarming or transmitting the distance into a decision module to participate in decision.
9. The automotive collision avoidance warning system of claim 8, wherein the obtaining of the occupancy map from the plan view comprises:
firstly, extracting points higher than a first threshold height of the ground according to the world coordinates of each point, and converting the points into a ground coordinate system; marking points above a first threshold height and below a second threshold height in an occupancy map; the value of each pixel in the occupancy map is the sum of the heights of its corresponding points, thereby resulting in the occupancy map.
10. The automotive collision avoidance warning system of claim 8, wherein the segmenting from the occupancy map the location of each obstacle by the connected component marker detection algorithm comprises:
and converting each pixel value in the occupied map into an image with a mark in color according to the size of the pixel value, so that the larger the pixel value is, the more red the pixel value is, the smaller the pixel value is, the more blue the pixel value is, and the different objects are segmented by a connected domain mark detection algorithm, thereby obtaining the position of each obstacle.
11. The automobile anti-collision warning system according to claim 8, wherein the binarizing the V disparity map comprises:
the maximum value of the pixel values of each row is obtained, the gray value of the pixel where only the maximum value is located in each row is set to be 255, and the gray values of the rest pixels are set to be 0.
12. The automotive collision avoidance warning system of claim 8, wherein the fitting a piecewise straight line using the RANSAC method comprises:
the following sequence of operations is repeatedly performed until a predetermined exit criterion is reached:
selecting a group of random subsets in the maximum value points in the V disparity map to perform straight line fitting to obtain a straight line model;
using the obtained line model to test all other data, if a certain point is suitable for the estimated line model, considering it as a local point, if more than a predetermined number of points are classified as local points, then the estimated model is considered as reasonable, then using all local points to re-estimate the model, and estimating the error rate of the local points and the model;
if the error rate of the model is lower than that of the best model, replacing the best model with the model;
and taking the best model obtained finally as the segmentation straight line.
13. The automotive collision avoidance warning system of claim 12, wherein the fitting of the multi-segment piecewise straight line using the RANSAC method comprises:
and extracting a first straight line, removing points belonging to the first straight line from the V disparity map after extraction is finished, extracting a second straight line aiming at the rest points, and repeating the steps until the number of the rest points is less than a preset threshold value.
14. The anti-collision warning system for automobiles according to claim 8, wherein said deriving the travelable region in the original gray image from the extracted straight line comprises:
and selecting the parallax value on the extracted straight line as d for each line in the V parallax image, comparing the parallax value of each pixel with the difference value of d in the corresponding line in the parallax image, and judging the corresponding position of the original image as a safe drivable area when the difference value is smaller than a certain threshold value.
15. An automotive collision avoidance early warning system, comprising:
the binocular camera is configured to shoot a left gray image and a right gray image in front of the automobile along the automobile traveling direction;
a parallax map calculation unit which calculates a parallax map from the left and right two gray images;
the V disparity map conversion module is used for converting the disparity map to obtain a V disparity map;
the binarization module is used for binarizing the V disparity map;
the RANSAC straight line fitting module is used for fitting points of the binarized V disparity map by using an RANSAC method to obtain a segmented straight line;
the multi-frame image filtering module is used for smoothing filtering straight lines according to multi-frame images, and comprises a time window, wherein a straight line model is assumed to be represented as ax + by + c which is 0, straight line model parameters are obtained for each frame of image, each frame is accumulated aiming at each parameter, when a new image comes from each frame, the straight line model parameters of the initial frame of image are subtracted from the accumulated parameter result, the straight line model parameters of the current frame of image are added, and then the average is obtained to be used as the straight line model parameters of the frame;
the original image travelable area determining module is used for obtaining travelable areas in the original gray level image through the extracted straight lines;
the ground model fitting module is used for calculating the three-dimensional coordinates of points belonging to the ground in a real world coordinate system according to the original image and the disparity map, assuming that the ground is a plane model, and fitting the plane by using RANSAC to obtain a ground model;
the occupied map calculation module converts the whole scene in the original gray level image from camera coordinates to world coordinates, generates a plane graph at the same time, and calculates the occupied map from the plane graph;
the obstacle segmentation and distance calculation module is used for obtaining the position of each obstacle from the occupied map through segmentation of a connected domain mark detection algorithm, converting the position into an original image to be marked, and calculating the distance from the obstacle to the vehicle through a disparity map;
and the alarm module is used for giving an alarm when the distance between the current vehicles is less than a certain threshold value or transmitting the distance into the decision module to participate in decision making.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2016/100521WO2018058356A1 (en) | 2016-09-28 | 2016-09-28 | Method and system for vehicle anti-collision pre-warning based on binocular stereo vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108243623A CN108243623A (en) | 2018-07-03 |
| CN108243623Btrue CN108243623B (en) | 2022-06-03 |
Family
ID=61762982
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201680001426.6AActiveCN108243623B (en) | 2016-09-28 | 2016-09-28 | Automobile anti-collision early warning method and system based on binocular stereo vision |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108243623B (en) |
| WO (1) | WO2018058356A1 (en) |
Families Citing this family (58)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113610923B (en)* | 2018-07-26 | 2025-07-22 | 上海联影智能医疗科技有限公司 | Scanning positioning method, scanning positioning device, computer equipment and computer readable storage medium |
| CN110909569B (en)* | 2018-09-17 | 2022-09-23 | 深圳市优必选科技有限公司 | Road condition information identification method and terminal equipment |
| CN110969064B (en)* | 2018-09-30 | 2023-10-27 | 北京四维图新科技股份有限公司 | An image detection method, device and storage device based on monocular vision |
| CN109444916B (en)* | 2018-10-17 | 2023-07-04 | 上海蔚来汽车有限公司 | Unmanned driving drivable area determining device and method |
| CN109461308B (en)* | 2018-11-22 | 2020-10-16 | 东软睿驰汽车技术（沈阳）有限公司 | Information filtering method and image processing server |
| CN109601109A (en)* | 2018-12-07 | 2019-04-12 | 江西洪都航空工业集团有限责任公司 | A kind of unmanned grass-cutting vehicle collision-proof method based on binocular vision detection |
| CN111382591B (en)* | 2018-12-27 | 2023-09-29 | 海信集团有限公司 | Binocular camera ranging correction method and vehicle-mounted equipment |
| CN111469759A (en)* | 2019-01-24 | 2020-07-31 | 海信集团有限公司 | Scratch and rub early warning method for vehicle, vehicle and storage medium |
| CN109993060B (en)* | 2019-03-01 | 2022-11-22 | 长安大学 | Vehicle omnidirectional obstacle detection method of depth camera |
| CN110569704B (en)* | 2019-05-11 | 2022-11-22 | 北京工业大学 | A Multi-strategy Adaptive Lane Line Detection Method Based on Stereo Vision |
| CN110110682B (en)* | 2019-05-14 | 2023-04-18 | 西安电子科技大学 | Semantic stereo reconstruction method for remote sensing image |
| CN110298330B (en)* | 2019-07-05 | 2023-07-18 | 东北大学 | A monocular detection and positioning method for a power transmission line inspection robot |
| CN110285793B (en)* | 2019-07-08 | 2020-05-15 | 中原工学院 | Intelligent vehicle track measuring method based on binocular stereo vision system |
| CN110472508B (en)* | 2019-07-15 | 2023-04-28 | 天津大学 | Lane line distance measurement method based on deep learning and binocular vision |
| CN112767818B (en)* | 2019-11-01 | 2022-09-27 | 北京初速度科技有限公司 | Map construction method and device |
| CN111241979B (en)* | 2020-01-07 | 2023-06-23 | 浙江科技学院 | Real-time obstacle detection method based on image feature calibration |
| CN111275698B (en)* | 2020-02-11 | 2023-05-09 | 西安汇智信息科技有限公司 | Method for detecting visibility of road in foggy weather based on unimodal offset maximum entropy threshold segmentation |
| WO2021174118A1 (en)* | 2020-02-26 | 2021-09-02 | Nvidia Corporation | Object detection using image alignment for autonomous machine applications |
| WO2021174539A1 (en)* | 2020-03-06 | 2021-09-10 | 深圳市大疆创新科技有限公司 | Object detection method, mobile platform, device and storage medium |
| CN111626095B (en)* | 2020-04-06 | 2023-07-28 | 连云港市港圣开关制造有限公司 | Power distribution inspection system based on Ethernet |
| CN111580131B (en)* | 2020-04-08 | 2023-07-07 | 西安邮电大学 | Method for 3D lidar smart car to recognize vehicles on the highway |
| CN111612760B (en)* | 2020-05-20 | 2023-11-17 | 阿波罗智联(北京)科技有限公司 | Method and device for detecting obstacles |
| CN111890358B (en)* | 2020-07-01 | 2022-06-14 | 浙江大华技术股份有限公司 | Binocular obstacle avoidance method and device, storage medium and electronic device |
| CN112097732A (en)* | 2020-08-04 | 2020-12-18 | 北京中科慧眼科技有限公司 | Binocular camera-based three-dimensional distance measurement method, system, equipment and readable storage medium |
| CN111985436B (en)* | 2020-08-29 | 2024-03-12 | 浙江工业大学 | Workshop ground marking recognition fitting method based on LSD |
| CN112200771B (en)* | 2020-09-14 | 2024-08-16 | 浙江大华技术股份有限公司 | Height measurement method, device, equipment and medium |
| CN112233136B (en)* | 2020-11-03 | 2021-10-22 | 上海西井信息科技有限公司 | Method, system, equipment and storage medium for alignment of container trucks based on binocular recognition |
| CN112288791B (en)* | 2020-11-06 | 2024-04-30 | 浙江中控技术股份有限公司 | Parallax image obtaining method, three-dimensional model obtaining method and device based on fisheye camera |
| CN112418103B (en)* | 2020-11-24 | 2022-10-11 | 中国人民解放军火箭军工程大学 | A safety anti-collision system and method for bridge crane hoisting based on dynamic binocular vision |
| CN114661039A (en)* | 2020-12-22 | 2022-06-24 | 郑州宇通客车股份有限公司 | Logistics vehicle and its trailer pose determination, pre-collision detection and automatic driving method |
| CN112669362B (en)* | 2021-01-12 | 2024-03-29 | 四川深瑞视科技有限公司 | Depth information acquisition method, device and system based on speckles |
| CN112634359B (en)* | 2021-01-14 | 2024-09-03 | 深圳市一心视觉科技有限公司 | Vehicle anti-collision early warning method and device, terminal equipment and storage medium |
| CN112818766A (en)* | 2021-01-18 | 2021-05-18 | 深圳英飞拓科技股份有限公司 | High-altitude parabolic detection alarm method and system based on computer vision |
| CN113370977B (en)* | 2021-05-06 | 2022-11-18 | 上海大学 | Intelligent vehicle forward collision early warning method and system based on vision |
| CN113112553B (en)* | 2021-05-26 | 2022-07-29 | 北京三快在线科技有限公司 | Parameter calibration method and device for binocular camera, electronic equipment and storage medium |
| CN113341391B (en)* | 2021-06-01 | 2022-05-10 | 电子科技大学 | Radar target multi-frame joint detection method in unknown environment based on deep learning |
| CN113658240B (en)* | 2021-07-15 | 2024-04-19 | 北京中科慧眼科技有限公司 | Main obstacle detection method and device and automatic driving system |
| CN113900443B (en)* | 2021-09-28 | 2023-07-18 | 合肥工业大学 | A method and device for UAV obstacle avoidance warning based on binocular vision |
| CN113706622B (en)* | 2021-10-29 | 2022-04-19 | 北京中科慧眼科技有限公司 | Road surface fitting method and system based on binocular stereo vision and intelligent terminal |
| CN114155257B (en)* | 2021-11-04 | 2025-03-25 | 浙江建木智能系统有限公司 | Industrial vehicle early warning and obstacle avoidance method and system based on binocular camera |
| CN114119700B (en)* | 2021-11-26 | 2024-03-29 | 山东科技大学 | An obstacle distance measurement method based on U-V disparity map |
| CN114494427B (en)* | 2021-12-17 | 2024-09-03 | 山东鲁软数字科技有限公司 | Method, system and terminal for detecting illegal behaviors of person with suspension arm going off station |
| CN113946154B (en)* | 2021-12-20 | 2022-04-22 | 广东科凯达智能机器人有限公司 | Visual identification method and system for inspection robot |
| CN115239742B (en)* | 2022-07-08 | 2025-04-29 | 清驰(济南)智能科技有限公司 | Green belt automatic watering system, device and storage medium based on binocular vision |
| CN115116038B (en)* | 2022-08-30 | 2023-03-24 | 北京中科慧眼科技有限公司 | Obstacle identification method and system based on binocular vision |
| CN115205809B (en)* | 2022-09-15 | 2023-03-24 | 北京中科慧眼科技有限公司 | Method and system for detecting roughness of road surface |
| CN115503723B (en)* | 2022-10-10 | 2024-12-13 | 大连理工大学 | An automatic detection method for highway height restriction facilities based on binocular vision |
| CN115601435B (en)* | 2022-12-14 | 2023-03-14 | 天津所托瑞安汽车科技有限公司 | Vehicle attitude detection method, device, vehicle and storage medium |
| CN115995163B (en)* | 2023-03-23 | 2023-06-27 | 江西通慧科技集团股份有限公司 | Vehicle collision early warning method and system |
| CN116714669A (en)* | 2023-05-30 | 2023-09-08 | 北京鉴智科技有限公司 | A vehicle turning control method and device based on binocular stereo vision |
| CN117079219B (en)* | 2023-10-08 | 2024-01-09 | 山东车拖车网络科技有限公司 | Vehicle running condition monitoring method and device applied to trailer service |
| CN117274939B (en)* | 2023-10-08 | 2024-05-28 | 北京路凯智行科技有限公司 | Safety area detection method and safety area detection device |
| CN117672007B (en)* | 2024-02-03 | 2024-04-26 | 福建省高速公路科技创新研究院有限公司 | Road construction area safety precaution system based on thunder fuses |
| CN117930224B (en)* | 2024-03-19 | 2024-06-18 | 山东科技大学 | Vehicle ranging method based on monocular vision depth estimation |
| CN118230291B (en)* | 2024-04-02 | 2024-10-08 | 山东倍科信息技术有限公司 | Image recognition system and method |
| CN118570257A (en)* | 2024-06-18 | 2024-08-30 | 岚图汽车科技有限公司 | Dummy head offset test method, device, equipment and storage medium |
| CN118840495B (en)* | 2024-06-24 | 2025-08-01 | 元橡科技(苏州)有限公司 | Wheel track elevation extraction method and system and electronic equipment |
| CN119065368A (en)* | 2024-08-16 | 2024-12-03 | 广州工业智能研究院 | A method and system for controlling vehicle straight-line driving |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102520721A (en)* | 2011-12-08 | 2012-06-27 | 北京控制工程研究所 | Autonomous obstacle-avoiding planning method of tour detector based on binocular stereo vision |
| CN103400392A (en)* | 2013-08-19 | 2013-11-20 | 山东鲁能智能技术有限公司 | Binocular vision navigation system and method based on inspection robot in transformer substation |
| CN103679127A (en)* | 2012-09-24 | 2014-03-26 | 株式会社理光 | Method and device for detecting drivable area of road pavement |
| CN105043350A (en)* | 2015-06-25 | 2015-11-11 | 闽江学院 | Binocular vision measuring method |
| CN105225482A (en)* | 2015-09-02 | 2016-01-06 | 上海大学 | Based on vehicle detecting system and the method for binocular stereo vision |
| CN105550665A (en)* | 2016-01-15 | 2016-05-04 | 北京理工大学 | Method for detecting pilotless automobile through area based on binocular vision |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103106651B (en)* | 2012-07-16 | 2015-06-24 | 清华大学深圳研究生院 | Method for obtaining parallax error plane based on three-dimensional hough |
| CN103413313B (en)* | 2013-08-19 | 2016-08-10 | 国家电网公司 | The binocular vision navigation system of electrically-based robot and method |
- 2016
- 2016-09-28WOPCT/CN2016/100521patent/WO2018058356A1/ennot_activeCeased
- 2016-09-28CNCN201680001426.6Apatent/CN108243623B/enactiveActive
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102520721A (en)* | 2011-12-08 | 2012-06-27 | 北京控制工程研究所 | Autonomous obstacle-avoiding planning method of tour detector based on binocular stereo vision |
| CN103679127A (en)* | 2012-09-24 | 2014-03-26 | 株式会社理光 | Method and device for detecting drivable area of road pavement |
| CN103400392A (en)* | 2013-08-19 | 2013-11-20 | 山东鲁能智能技术有限公司 | Binocular vision navigation system and method based on inspection robot in transformer substation |
| CN105043350A (en)* | 2015-06-25 | 2015-11-11 | 闽江学院 | Binocular vision measuring method |
| CN105225482A (en)* | 2015-09-02 | 2016-01-06 | 上海大学 | Based on vehicle detecting system and the method for binocular stereo vision |
| CN105550665A (en)* | 2016-01-15 | 2016-05-04 | 北京理工大学 | Method for detecting pilotless automobile through area based on binocular vision |
Non-Patent Citations (1)
| Title |
|---|
| 基于立体视觉的非结构化环境下障碍物检测技术研究;张毅;《中国优秀硕士学位论文全文数据库 信息科技辑》;20150715(第07期);第三章* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108243623A (en) | 2018-07-03 |
| WO2018058356A1 (en) | 2018-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108243623B (en) | Automobile anti-collision early warning method and system based on binocular stereo vision | |
| US11093763B2 (en) | Onboard environment recognition device | |
| US10937181B2 (en) | Information processing apparatus, object recognition apparatus, device control system, movable body, image processing method, and computer-readable recording medium | |
| CN107517592B (en) | Real-time detection method and system for automobile driving area | |
| CN108028023B (en) | Information processing apparatus, information processing method, and computer-readable storage medium | |
| US9330320B2 (en) | Object detection apparatus, object detection method, object detection program and device control system for moveable apparatus | |
| US10776946B2 (en) | Image processing device, object recognizing device, device control system, moving object, image processing method, and computer-readable medium | |
| US10885351B2 (en) | Image processing apparatus to estimate a plurality of road surfaces | |
| US9697421B2 (en) | Stereoscopic camera apparatus | |
| CN103679119B (en) | Self adaptation slope road detection method and device | |
| US11514683B2 (en) | Outside recognition apparatus for vehicle | |
| US11691585B2 (en) | Image processing apparatus, imaging device, moving body device control system, image processing method, and program product | |
| KR102507248B1 (en) | Egomotion estimation system and method thereof | |
| US10953885B2 (en) | Road surface detecting apparatus | |
| US10503984B2 (en) | Object detection device | |
| Lion et al. | Smart speed bump detection and estimation with kinect | |
| JPWO2017099199A1 (en) | Image processing apparatus, object recognition apparatus, device control system, image processing method and program | |
| EP3540643A1 (en) | Image processing apparatus and image processing method | |
| EP3336754A2 (en) | Information processing apparatus, photographing apparatus, moving object control system, moving object, information processing method, and program | |
| KR20200027215A (en) | Method for position recognition of vehicle using lane-end-point detection algorithm and method for evaluating performance of the same | |
| Ganesan et al. | An Image Processing Approach to Detect Obstacles on Road | |
| JP7134780B2 (en) | stereo camera device | |
| EP2919191B1 (en) | Disparity value deriving device, equipment control system, movable apparatus, robot, and disparity value producing method | |
| EP3327624A1 (en) | Information processing apparatus, imaging device, device control system, mobile object, information processing method, and carrier means | |
| JP6943092B2 (en) | Information processing device, imaging device, device control system, moving object, information processing method, and information processing program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |