Disclosure of Invention
The invention aims to provide a road width measuring method, system, equipment and medium based on point cloud and images, so as to solve the problems in the prior art.
In order to achieve the above object, the present invention provides a road width measurement method based on a point cloud and an image, including:
acquiring road images and corresponding ground area point cloud data;
Road segmentation is carried out on the road image, road mask fitting is carried out on the basis of a road segmentation result, and a fitted road mask is obtained;
Projecting the fitted road mask into the point cloud data of the ground area and filtering to obtain the point cloud pavement data corresponding to the road mask;
Screening points positioned on the left side and the right side of a road in the point cloud pavement data, and constructing a left edge point set and a right edge point set;
And acquiring a branch road identification result, road width information and a width change curve corresponding to the advancing direction based on the left and right edge point sets.
Optionally, the process for acquiring the ground area point cloud data specifically includes:
Acquiring point cloud data corresponding to a road image;
and extracting ground area point cloud data in the point cloud data based on a ground segmentation algorithm.
Optionally, the performing road mask fitting based on the road segmentation result specifically includes:
Carrying out road semantic segmentation on the road image to obtain an original mask of a road area;
Performing morphological operation on the original mask to obtain a processed mask;
and performing edge detection on the processed mask to generate a road edge curve, fitting the road edge curve based on a weighted curve fitting method, and filling the mask based on the fitted road edge curve to obtain the fitted road mask.
Optionally, the projecting the fitted road mask into the ground area point cloud data and performing filtering processing specifically includes:
Projecting the fitted road mask into the ground area point cloud data based on the external parameter matrix of the road image to obtain a projection transformation result;
and filtering the point cloud data of the non-road area in the projection transformation result, and reserving the point cloud data corresponding to the road area in the road mask to obtain the point cloud pavement data.
Optionally, the construction process of the left and right edge point sets specifically includes:
And screening points positioned on the left side and the right side of the road in the point cloud pavement data based on a preset judging standard and the space position of the point cloud under a plane coordinate system to obtain the left and right edge point sets.
Optionally, the process of obtaining the fork identification result specifically includes:
Analyzing the branch morphology of the road area based on the road segmentation result to obtain a first branch identification result, wherein the first branch identification result comprises the primary position and the primary direction of a branch;
And analyzing the separation degree of the left and right edge point sets to obtain a second branch identification result, and determining a final branch identification result based on the first branch identification result and the second branch identification result.
Optionally, the process of obtaining the road width information specifically includes:
Calculating a central line of the road based on the left and right edge point sets, selecting a central line tangent line of a fixed position at the far end of the road, calculating a normal line of the central line tangent line, and calculating Euclidean distance between the intersection points based on intersection points of the normal line and the two side edge fitting curves to obtain the road width information;
road width information of the road is calculated in real time, and a width change curve corresponding to the traveling direction is output.
A road width measurement system based on a point cloud and an image, comprising:
The data acquisition module is used for acquiring road images and corresponding ground area point cloud data;
the image segmentation and road mask fitting module is used for carrying out road segmentation on the road image and carrying out road mask fitting on the basis of a road segmentation result to obtain a fitted road mask;
The projection conversion module is used for projecting the fitted road mask into the point cloud data of the ground area and filtering the point cloud data to obtain the point cloud pavement data corresponding to the road mask;
the road width measurement module is used for screening points positioned on the left side and the right side of the road in the point cloud road surface data to construct a left edge point set and a right edge point set, and acquiring a branch road identification result, road width information and a width change curve corresponding to the advancing direction based on the left edge point set and the right edge point set.
An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the one point cloud and image based road width measurement method.
A computer readable storage medium storing a computer program which when executed by a processor implements the method of road width measurement based on a point cloud and an image.
The invention has the technical effects that:
The invention can overcome the defects of the prior art, realize more accurate, real-time and stable road width detection, adapt to dynamic road environment, and can accurately identify road boundaries especially under the condition of limited sensor vision, thereby ensuring safe driving and efficient path planning of an automatic driving system.
According to the invention, the multi-dimensional sensing and analysis are carried out on the road environment by combining the laser radar point cloud data and the camera image data, so that the limitation of a single data source is overcome, the accuracy and the robustness of road width measurement are improved, and the risk brought by a complex road section can be reduced.
Detailed Description
Various exemplary embodiments of the invention will now be described in detail, which should not be considered as limiting the invention, but rather as more detailed descriptions of certain aspects, features and embodiments of the invention.
It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In addition, for numerical ranges in this disclosure, it is understood that each intermediate value between the upper and lower limits of the ranges is also specifically disclosed. Every smaller range between any stated value or stated range, and any other stated value or intermediate value within the stated range, is also encompassed within the invention. The upper and lower limits of these smaller ranges may independently be included or excluded in the range.
It will be apparent to those skilled in the art that various modifications and variations can be made in the specific embodiments of the application described herein without departing from the scope or spirit of the application. Other embodiments will be apparent to those skilled in the art from consideration of the specification of the present application. The specification and examples of the present application are exemplary only.
As used herein, the terms "comprising," "including," "having," "containing," and the like are open ended terms, meaning including, but not limited to.
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
1-6, The embodiment provides a road width measurement method based on point clouds and images, which comprises the steps of obtaining road images and corresponding ground area point cloud data, carrying out road segmentation on the road images, carrying out road mask fitting on the basis of road segmentation results to obtain fitted road masks, projecting the fitted road masks into the ground area point cloud data and carrying out filtering processing to obtain point cloud road surface data corresponding to the road masks, screening points positioned on the left side and the right side of a road in the point cloud road surface data, constructing left and right edge point sets, and obtaining a branch road identification result, road width information and a width change curve corresponding to a traveling direction on the basis of the left and right edge point sets.
The embodiment provides a road width measuring method and system based on point cloud and images, which can overcome the defects of the prior art, realize more accurate, real-time and stable road width detection, adapt to dynamic road environments, and can accurately identify road boundaries especially under the condition of limited sensor vision, thereby ensuring safe driving and efficient path planning of an automatic driving system.
The embodiment aims at carrying out multidimensional sensing and analysis on the road environment by combining the laser radar point cloud data and the camera image data, overcoming the limitation of a single data source, improving the accuracy and the robustness of road width measurement and reducing the risks brought by complex road sections.
The technical scheme of the embodiment is as follows:
S1, acquiring point cloud data and extracting the ground, and acquiring the point cloud data of a road scene by using a laser radar. The point cloud data are transmitted through ROS communication, ground segmentation is carried out on the point cloud data, point cloud data of an interested area (namely, a ground area) are extracted, and non-ground point cloud is filtered. Point cloud data for ground areaThe representation P comprises N points, each point comprises (x, y, z) three-dimensional space coordinates, wherein the three-dimensional space coordinates take the center of the laser radar as an origin, the front of the laser radar is in the x positive direction, the left is in the y positive direction, and the upper is in the z positive direction;
s2, acquiring image data and dividing a road, wherein an industrial camera is used for acquiring an image I synchronous with point cloud data, the image I comprises m x n pixel points, and each pixel point is represented by coordinates (u, v). In order to accurately divide the road area, the semantic division model is utilized to divide the road area:
S21, generating a semantic segmentation mask, namely carrying out road semantic segmentation on the image to generate a pavement mask M, wherein the semantic segmentation mask M (u, v) epsilon {0,1} is used for distinguishing road and non-road areas;
S22, mask post-processing, namely performing morphological operation (such as expansion and corrosion) on the generated mask M (u, v) to remove a small noise area and smooth the mask edge so as to obtain a processed mask M' (u, v) in order to eliminate noise and improve the accuracy of a road area;
S23, edge fitting, namely performing edge detection on the processed mask M' (u, v) to generate a road edge curve Cedge, fitting an out-road edge curve, and filling the mask to obtain a fitted mask M "(u, v);
Cedge={(u,v)∣M'(u,v)=1}
s3, road mask fitting is carried out based on an image segmentation result, noise and irregular boundary curves possibly generated due to segmentation and edge detection are subjected to smoothing treatment and curve fitting:
S31, performing edge smoothing treatment, namely performing smoothing treatment on the fitted road edge curve, and reducing the influence of noise on the edge position to obtain a smooth curve Csmooth;
S32, refining the edge curve to ensure the smoothness and continuity of the curve;
S33, edge weighted fitting, namely further fitting edge curves on two sides of the road by using a weighted curve fitting method, so that the fitted curves can reflect the real boundary of the road. On an actual road, some parts of the edges may be more critical, so the edges of different regions may be weighted differently using a weighted curve fitting method, thus better reflecting the actual shape of the road. Weighted curve fitting by assigning weights ωi to the points in the fitting process, the final fitted edge curve equation fleft(u)、fright (u) represents the curve model of the left and right edges, vleft＝fleft(u;ωi),vright＝fright(u;ωi), where ωi is the weight of each point;
s4, projecting the mask to the point cloud to perform point cloud filtering, and obtaining point cloud pavement data. And projecting the processed mask M "(u, v) onto the ground point cloud data P, and filtering the point cloud through the projection mask to obtain the road point cloud Proad. The ground point cloud data P are further divided, and road and non-road parts are distinguished:
S41, projecting pixel points of the image to a point cloud coordinate system by using an external parameter matrix T of the laser radar. Point Pi＝(xi,yi,zi) under the lidar coordinate system) and the corresponding pixel point Mi"(ui,vi) are obtained by projective transformation:
Wherein K is an internal reference matrix of the camera, and T is an external reference matrix of the laser radar reaching the camera;
S42, performing filtering processing on the point cloud data P to obtain a point cloud corresponding to the mask M "(u, v), and obtaining a road point cloud Proad＝{Pi∈P∣Mi"(ui,vi) =1 }, where i=1, 2.
S5, road width measurement, namely, road width measurement is carried out by utilizing the segmented point cloud pavement data Proad:
S51, identifying point clouds on two sides of the road, namely setting a judging standard, and screening out points on the left side and the right side of the road according to the spatial positions of the point clouds under a plane coordinate system (such as an XOY plane). For each point Pi＝(xi,yi,zi)∈Proad, it is determined whether it is to the left or right of the road center line based on the magnitude of its transverse coordinate yi value, if yi>ycenter (assuming ycenter is the transverse position of the road center line), then point Pi is classified as left edge point set Pleft, and if yi<ycenter, point Pi is classified as left edge point set Pright. The edge point sets Pleft and Pright on two sides of the road point cloud can be identified from the road point cloud Proad, and the edge point sets correspond to the point cloud data of the left edge and the right edge of the road respectively;
s52, calculating a center line, namely calculating a center point of the road through the identified road two-edge point clouds Pleft and Pright
And S53, fitting edges and a central line, namely fitting a road left and right edge curve Cleft,Cright and a central curve Ccenter through a third-order curve through left and right edge points and a central point of the road.
S54, measuring the road width, namely selecting a central line tangent T (x) of a fixed position at the far end of the road, and calculating the normal N (x) of the tangent. And calculating the Euclidean distance W (x) = ||P 'left(x)-P'right (x) |between the intersection points by using the intersection points of the normal N (x) and the two side edge fitting curves as P 'left and P 'right, namely the road width.
S6, branch processing:
S61, carrying out branch road identification based on the image segmentation result, wherein the branch morphology of the road region can be analyzed through the image road semantic segmentation result M "(u, v) and the edge fitting curve fleft(u)、fright (u). The edge information in the image can be used for detecting the abrupt change of the road form to judge whether the turnout exists. The position and the direction of the turnout can be primarily judged by detecting the characteristics of sudden increase of the road width, severe change of the edge curve and the like;
S62, the point cloud data assist in further identifying the turnout, wherein the turnout is usually represented as separated smaller point cloud clusters in the point cloud data, and the point cloud clusters can be identified through a clustering method. The divergence is determined based on the degree of separation between the left and right edge point sets Pleft and Pright of the road, and if the edge point set suddenly diverges in a certain area, it can be inferred that the divergence exists here.
And S7, dynamically updating and outputting the road width, namely calculating the road width W (x) in real time and outputting data, wherein the data comprise a width change curve and intersection information.
And S71, outputting the road width data and the change curve of the road width data along with the travelling direction.
S72, outputting information of the turnout according to the turnout identification condition.
Fig. 2 is a schematic diagram of road width calculation based on the combination of image and point cloud in the present embodiment. In the part A, 1 is an original image data graph acquired by a camera, 2 is a pavement mask effect graph based on semantic segmentation, and 3 is an edge fitting and weighting smoothing effect graph based on an image mask. In the section B, 4 is an original point cloud data diagram acquired by a radar, and 5 is a point cloud data diagram after the extraction of the region of interest. In section C, 6 is a visual effect map of the point cloud filtered and projected onto the image, and 7 is a road width measurement map using software. In section D, 8 is an experimental calculation effect diagram of the present embodiment.
Compared with the prior art, the method can dynamically and real-time detect the road width, and by fusing laser radar point cloud and camera image data, the method can sense the road environment in real time and dynamically acquire the passable area of the road. Compared with the traditional map-based labeling mode, the method and the device can adapt to changes of actual road environments, such as temporary obstacles, construction areas or weather changes, and ensure that vehicles can adjust path planning in time.
The method has the advantages of high precision and robustness, namely the laser radar point cloud data has extremely high spatial resolution, and the road boundary and the obstacle position can be accurately identified. The camera image can supplement rich texture information, such as pavement markings, traffic signs and the like. In practical tests, the road width measurement error of the system in various environments is less than 2%.
The embodiment can adapt to complex road conditions, and can process various road conditions including school lanes, highways, rural small roads and the like. In particular, in narrow or complex road sections, the system can accurately calculate the maximum passable width of the vehicle running through joint sensing and data fusion of the sensors, and collision or line crossing risks are avoided.
The embodiment can reduce sensor dependence, although the sensing capability of a single sensor is limited in certain scenes, for example, the accuracy of a laser radar is reduced in strong light or rain and fog, and a camera does not perform well in the condition of night or low visibility, the embodiment compensates a short plate of the single sensor by combining various sensor data, improves the robustness and the stability of a system, and reduces the influence of environment on sensing results.
The method has the economic and social benefits that the automatic driving system can better execute path planning and obstacle avoidance through accurate road width measurement, so that the traffic safety is improved, and the accident rate is reduced. Especially in complex road sections or areas without accurate map data, the application of the embodiment can obviously improve the decision making capability of the automatic driving vehicle, reduce the dependence on high-accuracy maps and reduce the development and maintenance cost. In addition, the wide application of the automatic driving technology is beneficial to reducing traffic jam and improving fuel efficiency, so that remarkable social and economic benefits are brought.
Experiments prove that the road width measurement error of the embodiment in various environments is stably controlled within 10 cm, and the method has higher measurement precision and adaptability.
Specific algorithm implementation examples of the present embodiment include:
The method comprises the steps of carrying out preliminary ground extraction by utilizing point clouds, obtaining a pavement mask by utilizing an image, fitting the pavement mask, projecting the point clouds to the image to carry out point cloud pavement extraction, carrying out road width calculation by utilizing an edge point tangent method, and finally processing a branch road according to the point cloud pavement extraction result and the calculated road width. In the embodiment, the M1-plus laser radar created by RoboSense speed-up aggregation is used as a point cloud acquisition device, the sea-health industrial camera is used as an image acquisition device, and the road width in the campus is calculated. According to the flowchart shown in fig. 1, the following steps are required in this embodiment:
S1, acquiring point cloud data of a road scene by using a laser radar, estimating the ground by using a patch work++ algorithm, extracting point cloud data of an interested region (namely, a ground region), and filtering non-ground point cloud.
S2, acquiring road scene image data synchronous with the point cloud data by using an industrial camera, performing road semantic segmentation on the acquired image data through BiseNet V, and preliminarily obtaining a road mask according to the semantic segmentation.
Then, in order to eliminate noise and improve the accuracy of the road area, the present embodiment performs morphological operations (such as expansion and corrosion) on the generated mask M (u, v), wherein the number of operations is set to 1 for the expansion and corrosion convolution kernel= (9, 9). Removing small noise region, smoothing mask edge to obtain processed mask M' (u, v)
Finally, edge detection is carried out on the processed mask M' (u, v) to generate a road edge curve Cedge, the road edge curve is fitted, and the mask is filled to obtain a fitted mask M "(u, v);
Cedge={(u,v)∣M'(u,v)=1}
S4, projecting the mask to the point cloud to perform point cloud filtering, and obtaining point cloud pavement data.
Firstly, the processed mask M "(u, v) is projected onto the ground point cloud data P, and the point cloud is filtered through the projection mask to obtain the road point cloud Proad. The ground point cloud data P are further divided, and road and non-road parts are distinguished:
And then, projecting the pixel points of the image to a point cloud coordinate system by using an external parameter matrix T of the laser radar. Point Pi＝(xi,yi,zi) under the lidar coordinate system) and the corresponding pixel point Mi"(ui,vi) are obtained by projective transformation:
Wherein K is an internal reference matrix of the camera, and T is an external reference matrix of the laser radar reaching the camera;
Finally, filtering the point cloud data P to obtain a point cloud corresponding to the mask M "(u, v), and obtaining a road point cloud Proad＝{Pi∈P∣Mi"(ui,vi) =1 }, wherein i=1, 2, N;
S5, calculating road width by using the segmented point cloud road surface data Proad.
Firstly, dividing the road point cloud Proad obtained by dividing the S4 according to the x-axis direction, wherein the dividing step length is set to be 0.1 meter, the dividing range is set to be 3< x <15, and the point cloud data Pxroad with one section every 0.1 meter is obtained.
And setting a judgment standard, and screening out points positioned on the left side and the right side of the road according to the spatial position of the point cloud under the XOY plane coordinate system. For each point Pi＝(xi,yi,zi)∈Proad, it is determined whether it is to the left or right of the road center line based on the magnitude of its transverse coordinate yi value, if yi>ycenter (assuming ycenter is the transverse position of the road center line), then point Pi is classified as left edge point set Pleft, and if yi<ycenter, point Pi is classified as left edge point set Pright. The edge point sets Pleft and Pright on two sides of the road point cloud can be identified from the road point cloud Proad, and the edge point sets correspond to the point cloud data of the left edge and the right edge of the road respectively;
Then calculating the center point of the road through the two edge point clouds Pleft and Pright of the identified roadAnd performing third-order polynomial curve fitting on the left and right edges and the central point of the road to obtain a left and right edge curve Cleft,Cright,Ccenter of the road.
Wherein, n is the order of the fitting polynomial.
Then, at a fixed x-axis position x=7 meters at the far end of the road, a centerline tangent T (x) is calculated and the normal N (x) to this tangent is calculated as y-y0＝mperp(x-x0), where mperp is the normal slope. And calculating the Euclidean distance W (x) = ||P 'left(x)-P'right (x) |between the intersection points by using the intersection points of the normal N (x) and the two side edge fitting curves as P 'left and P 'right, namely the road width.
In the embodiment, the road width is calculated by dividing the road point cloud, the patchwork ++ is utilized to extract the road point cloud, then the BiseNetV2 is utilized to divide the image road mask, the road point cloud is divided based on projection transformation, and the road width is calculated through the road point cloud, so that the problem of calculating the road width in the whole day can be effectively solved.
A road width measurement system based on a point cloud and an image, comprising:
The data acquisition module is used for acquiring road images and corresponding ground area point cloud data;
the image segmentation and road mask fitting module is used for carrying out road segmentation on the road image and carrying out road mask fitting on the basis of a road segmentation result to obtain a fitted road mask;
The projection conversion module is used for projecting the fitted road mask into the point cloud data of the ground area and filtering the point cloud data to obtain the point cloud pavement data corresponding to the road mask;
the road width measurement module is used for screening points positioned on the left side and the right side of the road in the point cloud road surface data to construct a left edge point set and a right edge point set, and acquiring a branch road identification result, road width information and a width change curve corresponding to the advancing direction based on the left edge point set and the right edge point set.
An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the one point cloud and image based road width measurement method.
A computer readable storage medium storing a computer program which when executed by a processor implements the method of road width measurement based on a point cloud and an image.
The present application is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present application are intended to be included in the scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.