技术领域Technical field
本发明涉及自动驾驶中的车道线检测技术领域，具体是一种基于单目图像的2D车道线到3D车道线的重构方法。The present invention relates to the technical field of lane line detection in autonomous driving, specifically a reconstruction method from 2D lane lines to 3D lane lines based on monocular images.
背景技术Background technique
基于计算机视觉的车道线检测技术是自动驾驶领域的关键任务，尤其在缺乏高精地图的道路，基于纯视觉的车道线检测结果可以为车辆沿着车道线保持居中行驶提供依据。2D车道线检测技术旨在图像上准确输出二维车道线坐标点集，该技术已相对较为成熟，在很多论文和开源项目中都已实现。但是在真实驾驶场景中，路面难免存在上下坡，这使得2D车道线与真实世界的3D车道线无法对齐，车道线检测结果存在误差，危及自动驾驶行车安全。因此，正确做法是恢复2D车道线上每个点的深度信息，即完成3D车道线检测。Lane line detection technology based on computer vision is a key task in the field of autonomous driving, especially on roads lacking high-precision maps. Lane line detection results based on pure vision can provide a basis for vehicles to stay centered along the lane line. 2D lane line detection technology aims to accurately output a two-dimensional lane line coordinate point set on the image. This technology is relatively mature and has been implemented in many papers and open source projects. However, in real driving scenarios, the road surface inevitably has uphill and downhill slopes, which makes the 2D lane lines unable to align with the real-world 3D lane lines. There are errors in the lane line detection results, endangering the safety of autonomous driving. Therefore, the correct approach is to restore the depth information of each point on the 2D lane line, that is, to complete the 3D lane line detection.
激光雷达可以提供物体的深度信息，使用激光雷达辅助视觉，将车道线从2D提升到3D的技术路径相对较为简单。但是激光雷达的使用成本较高，目前3D车道线检测领域的主流研究方向仍然倾向于使用单目摄像头，由于单目图像缺乏深度信息，因此从基于纯视觉的单目图像中重构3D车道线十分具有挑战性。Lidar can provide depth information of objects. Using lidar to assist vision, the technical path to upgrade lane lines from 2D to 3D is relatively simple. However, the cost of using lidar is relatively high. Currently, the mainstream research direction in the field of 3D lane line detection still tends to use monocular cameras. Since monocular images lack depth information, 3D lane lines must be reconstructed from pure vision-based monocular images. Very challenging.
目前常用的方法是使用逆透视映射(IPM)将2D车道线重新映射到3D空间中，但是此方法的前提条件是严格基于平坦地面的，在面对真实驾驶场景中路面不平坦、存在上下坡的情形中鲁棒性不强。The currently commonly used method is to use inverse perspective mapping (IPM) to remap 2D lane lines into 3D space. However, the prerequisite of this method is strictly based on flat ground. In real driving scenarios, the road surface is uneven and there are ups and downs. The robustness is not strong in situations.
发明内容Contents of the invention
本发明的目的是克服上述背景技术中的不足，提供一种基于单目图像的2D车道线到3D车道线的重构方法，该方法应能提高二维到三维坐标映射的准确性，降低3D车道线检测的使用成本。The purpose of the present invention is to overcome the deficiencies in the above background technology and provide a reconstruction method from 2D lane lines to 3D lane lines based on monocular images. This method should be able to improve the accuracy of two-dimensional to three-dimensional coordinate mapping and reduce the cost of 3D lane line reconstruction. Cost of use of lane line detection.
本发明的技术方案是：The technical solution of the present invention is:
一种基于单目图像的2D车道线到3D车道线的重构方法，包括以下步骤：A reconstruction method from 2D lane lines to 3D lane lines based on monocular images, including the following steps:
1)建立世界坐标系和相机坐标系；1) Establish the world coordinate system and camera coordinate system;
2)求解水平地面上的三维映射点；2) Solve the three-dimensional mapping points on the horizontal ground;
3)求解考虑坡度信息的三维映射点；3) Solve the three-dimensional mapping points considering slope information;
4)遍历法求解坡度角；4) Use the ergodic method to solve the slope angle;
5)计算3D车道线坐标。5) Calculate 3D lane line coordinates.
所述步骤1)中，世界坐标系的原点为车辆中心，相机坐标系的原点为相机的光心。In step 1), the origin of the world coordinate system is the vehicle center, and the origin of the camera coordinate system is the optical center of the camera.
所述步骤2)包括：The step 2) includes:
选取为世界坐标系下z＝0的平面，在相机坐标系下，相机的光心坐标为(0,0,0)T；2D车道线上的某一像素点P0，在像素坐标系下的坐标为(u,v)，在图像坐标系下的坐标为(x,y)，在相机坐标系下的坐标为(x,y,f)T；在相机坐标系下，P0点对应的光线经过相机的光心，且与相机成像平面的交点为(x,y,f)T；Select the plane with z=0 in the world coordinate system. In the camera coordinate system, the optical center coordinate of the camera is (0,0,0)T ; a certain pixel point P0 on the 2D lane line, in the pixel coordinate system The coordinates of are (u, v), the coordinates in the image coordinate system are (x, y), and the coordinates in the camera coordinate system are (x, y, f)T ; in the camera coordinate system, point P0 corresponds The light ray passes through the optical center of the camera, and the intersection point with the camera imaging plane is (x, y, f)T ;
光线在相机坐标系下的向量方程为：The vector equation of light in the camera coordinate system is:
世界坐标系的点到相机坐标系的映射关系为：The mapping relationship between points in the world coordinate system and the camera coordinate system is:
Pc＝RPw+T (2)Pc =RPw +T (2)
式中，Pw为世界坐标系的点，Pc为Pw在相机坐标系的映射点，R为旋转矩阵，T为偏移矩阵；In the formula, Pw is the point of the world coordinate system, Pc is the mapping point of Pw in the camera coordinate system, R is the rotation matrix, and T is the offset matrix;
改写式(2)得到：Rewrite equation (2) to get:
Pw＝R-1(Pc-T) (3)Pw =R-1 (Pc -T) (3)
将相机坐标系下的点(0,0,0)T与(x,y,f)T代入式(3)得到：Substitute the points (0,0,0)T and (x, y, f)T in the camera coordinate system into equation (3) to get:
光线在世界坐标系下的向量方程为：The vector equation of light in the world coordinate system is:
Lw＝Ow+(Iw-Ow)*t (5)Lw =Ow +(Iw -Ow )*t (5)
将光线方程与已知平面方程联立：Combine the ray equation with the known plane equation:
得到：get:
Lwz＝Oz+dz*t＝0 (7)Lwz ＝Oz +dz *t＝0 (7)
求解xw和yw：Solve for xw and yw :
得到2D车道线上的某一像素点P0在世界坐标系z＝0下的三维映射点Pw(xw,yw,0)。Obtain the three-dimensional mapping point Pw (xw ,yw ,0) of a certain pixel point P0 on the 2D lane line in the world coordinate system z=0.
所述步骤3)包括：The step 3) includes:
设光线的方向向量α为：Let the direction vector α of the light be:
根据C、Pw的坐标，得到光线的方向向量：According to the coordinates of C and Pw , the direction vector of the light is obtained:
列出光线CPw的方程，将其写成参数方程：List the equation of ray CPw and write it as a parametric equation:
设斜坡平面经过点n(n1,n2,n3)，且斜坡平面由地平线绕x轴旋转θ角得到，斜坡平面的法向量为：Assume that the slope plane passes through point n (n1 , n2 , n3 ), and the slope plane is obtained by rotating the horizon around the x-axis by an angle θ. The normal vector of the slope plane is:
斜坡平面的点法式方程为：The point French equation of the slope plane is:
vp1*(x-n1)+vp2*(y-n2)+vp3*(z-n3)＝0 (13)vp1 *(xn1 )+vp2 *(yn2 )+vp3 *(zn3 )＝0 (13)
将式(11)与式(13)联立，得到：Combining equation (11) with equation (13), we get:
将已知量代入式(14)，得到：Substituting the known quantities into equation (14), we get:
将t代入式(11)得到：Substituting t into equation (11) we get:
得到三维映射点P(x,y,z)。Obtain the three-dimensional mapping point P(x,y,z).
所述步骤4)包括：在2D车道线图像的同一水平线上选取两个点Pi、Pj，假设Pi、Pj在坡度角为θ的倾斜道路上对应的点分别为wi、wj，在一定范围内遍历所有的θ角，当|wiwj|-k最小时，得到倾斜坡度θ；k为道路宽度。The step 4) includes: selecting two points Pi and Pj on the same horizontal line of the 2D lane line image. It is assumed that the corresponding points of Pi and Pj on the inclined road with the slope angle θ are wi and w respectively.j , traverse all θ angles within a certain range, and when |wi wj |-k is the smallest, the slope θ is obtained; k is the road width.
所述步骤5)包括：将θ代入式(16)，得到P(x,y,z)。The step 5) includes: substituting θ into equation (16) to obtain P(x, y, z).
本发明的有益效果是：The beneficial effects of the present invention are:
现有的涉及单目图像的2D车道线到3D车道线重构的研究，一般都是基于理想平坦路面进行建模，所谓理想平坦路面就是不考虑道路的坡度等因素，然而在真实驾驶场景中，道路的坡度信息是不可以忽略的；本发明提出了一种考虑真实道路坡度信息的2D车道线到3D车道线的重构方法，当单目图像上获取了车道线的二维点集后，为了进一步获取到车道线的三维信息，将提取到的二维车道线像素坐标转化为真实世界的三维坐标，并将道路的坡度信息加入到坐标转换中，更加符合真实驾驶场景的需求，提高了3D车道线检测的准确性，降低了使用成本。Existing research involving the reconstruction of 2D lane lines to 3D lane lines from monocular images is generally modeled based on an ideal flat road surface. The so-called ideal flat road surface does not consider factors such as the slope of the road. However, in real driving scenarios, , the slope information of the road cannot be ignored; the present invention proposes a reconstruction method from 2D lane lines to 3D lane lines that takes into account the real road slope information. When the two-dimensional point set of the lane line is obtained on the monocular image, , in order to further obtain the three-dimensional information of the lane line, the extracted two-dimensional lane line pixel coordinates are converted into real-world three-dimensional coordinates, and the road slope information is added to the coordinate conversion, which is more in line with the needs of real driving scenarios and improves It improves the accuracy of 3D lane line detection and reduces the cost of use.
附图说明Description of drawings
图1是为本发明的流程图。Figure 1 is a flow chart of the present invention.
图2是世界坐标系下的车辆俯视图。Figure 2 is a top view of the vehicle in the world coordinate system.
图3是世界坐标系下的车辆主视图。Figure 3 is the front view of the vehicle in the world coordinate system.
图4是世界坐标系下的车辆右视图。Figure 4 is a right side view of the vehicle in the world coordinate system.
图5是P点在真实道路斜坡上的示意图。Figure 5 is a schematic diagram of point P on a real road slope.
图6是P0点在2D车道线上的示意图。Figure 6 is a schematic diagram of point P0 on the 2D lane line.
图7是Pi点、Pj点在2D车道线上的示意图。Figure 7 is a schematic diagram of point Pi and point Pj on the 2D lane line.
图8是图7的实施例示意图。FIG. 8 is a schematic diagram of the embodiment of FIG. 7 .
具体实施方式Detailed ways
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用于解释本发明，并不用于限定本发明。In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention.
如图1所示，一种基于单目图像的2D车道线到3D车道线的重构方法，包括以下步骤As shown in Figure 1, a reconstruction method from 2D lane lines to 3D lane lines based on monocular images includes the following steps
1)建立世界坐标系和相机坐标系。1) Establish the world coordinate system and camera coordinate system.
参考图2所示，世界坐标系的定义为：点O为世界坐标系的原点，位于车辆的中心；X轴垂直于车辆前进方向并且位于水平地面；Y轴指向车辆前进方向，Y轴位于水平地面并且与车辆的中心线位于同一竖直平面；Z轴正向朝上；点C为相机，相机放置在车辆挡风玻璃正中央位置。Referring to Figure 2, the definition of the world coordinate system is: point O is the origin of the world coordinate system, located at the center of the vehicle; the The ground is on the same vertical plane as the centerline of the vehicle; the Z-axis is pointing upward; point C is the camera, and the camera is placed in the center of the vehicle windshield.
相机坐标系的原点为相机的光心。The origin of the camera coordinate system is the optical center of the camera.
2)求解水平地面上的三维映射点：求解单目图像上2D车道线像素坐标在世界坐标系z＝0(即理想平坦道路)下的三维映射点Pw(xw,yw,0)。2) Solve the three-dimensional mapping point on the horizontal ground: Solve the three-dimensional mapping point Pw (xw ,yw ,0) of the 2D lane line pixel coordinates on the monocular image in the world coordinate system z=0 (i.e., the ideal flat road) .
单目图像上2D车道线像素坐标在基于理想平坦道路条件下的三维映射点Pw(xw,yw,0)，是利用光线与已知平面的交点来获取三维信息，此交点就是图像中的像素点所对应的世界坐标。The three-dimensional mapping point Pw (xw ,yw ,0) of the 2D lane line pixel coordinates on the monocular image based on ideal flat road conditions uses the intersection of light and a known plane to obtain three-dimensional information. This intersection is the image The world coordinate corresponding to the pixel in .
测量平面选取世界坐标系下z＝0的平面，即为水平地面。The measurement plane selects the plane with z=0 in the world coordinate system, which is the horizontal ground.
2D车道线上的某一像素点P0，在像素坐标系下的坐标为(u,v)，在图像坐标系下的坐标为(x,y)，在相机坐标系下的坐标为(x,y,f)T，其中f为相机的焦距。在相机坐标系下，P0点对应的光线经过相机的光心，且与相机成像平面的交点为(x,y,f)T。A certain pixel point P0 on the 2D lane line, the coordinates in the pixel coordinate system are (u, v), the coordinates in the image coordinate system are (x, y), and the coordinates in the camera coordinate system are (x ,y,f)T , where f is the focal length of the camera. In the camera coordinate system, the light corresponding to point P0 passes through the optical center of the camera, and the intersection point with the camera imaging plane is (x, y, f)T .
光线方程可由两个点来确定，其中一个点选取相机的光心，在相机坐标系下的坐标为(0,0,0)T，另一个点选取光线与成像平面的交点(x,y)，考虑到成像平面在光心前(距离光心f处)，则光线上第二个点的坐标为(x,y,f)T。由于两点确定一条直线，光线穿过点(0,0,0)T，且光线的方向向量为：The light equation can be determined by two points. One point selects the optical center of the camera, whose coordinates in the camera coordinate system are (0,0,0)T , and the other point selects the intersection point (x,y) of the light ray and the imaging plane. , considering that the imaging plane is in front of the optical center (at a distance f from the optical center), the coordinates of the second point on the ray are (x, y, f)T . Since two points determine a straight line, the light passes through the point (0,0,0)T , and the direction vector of the light is:
因此，光线在相机坐标系下的向量方程为：Therefore, the vector equation of light in the camera coordinate system is:
其中，t为参数，t∈R。Among them, t is a parameter, t∈R.
由于最终求解的是世界坐标系下的交点，所以需将光线方程建立在世界坐标系中。世界坐标系的点到相机坐标系的映射关系为：Since the final solution is the intersection point in the world coordinate system, the light equation needs to be established in the world coordinate system. The mapping relationship between points in the world coordinate system and the camera coordinate system is:
Pc＝RPw+T (2)Pc =RPw +T (2)
式中，Pw是世界坐标系的点，Pc为Pw在相机坐标系的映射点，R为旋转矩阵，T为偏移矩阵；In the formula, Pw is a point in the world coordinate system, Pc is the mapping point of Pw in the camera coordinate system, R is the rotation matrix, and T is the offset matrix;
改写式(2)得到：Rewrite equation (2) to get:
Pw＝R-1(Pc-T) (3)Pw =R-1 (Pc -T) (3)
将相机坐标系下的点(0,0,0)T与点(x,y,f)T代入上式中的PC，将其转换到世界坐标系，得到：Substituting point (0,0,0)T and point (x, y, f)T in the camera coordinate system into PC in the above formula and converting it to the world coordinate system, we get:
式中：Ow、Iw分别为相机的光心、交点在世界坐标系的坐标；In the formula: Ow and Iw are the coordinates of the camera's optical center and intersection point in the world coordinate system respectively;
同理，在世界坐标系下，两点确定一条直线，光线穿过点Ow，且光线的方向向量为：In the same way, in the world coordinate system, two points determine a straight line, the light passes through the point Ow , and the direction vector of the light is:
光线在世界坐标系下的向量方程为：The vector equation of light in the world coordinate system is:
Lw＝Ow+(Iw-Ow)*t (5)Lw =Ow +(Iw -Ow )*t (5)
将光线方程与已知平面方程联立：Combine the ray equation with the known plane equation:
得到：get:
Lwz＝Oz+dz*t＝0 (7)Lwz ＝Oz +dz *t＝0 (7)
其中，上式为光线方程与已知平面方程联立后的中间表达式，Lwz表示空间直线Lw在z轴已知(取0)条件下的关于x、y的表达式。Among them, the above formula is the intermediate expression after combining the light equation and the known plane equation, Lwz represents the expression of the space straight line Lw with respect to x and y under the condition that the z axis is known (taken as 0).
求2D车道线上的某一像素点P0在世界坐标系z＝0下，即在基于理想平坦道路条件下的三维映射点Pw(xw,yw,0)：Find a certain pixel point P0 on the 2D lane line in the world coordinate system z=0, that is, the three-dimensional mapping point Pw (xw ,yw ,0) based on ideal flat road conditions:
得到2D车道线上的某一像素点P0在世界坐标系z＝0下的三维映射点Pw(xw,yw,0)。Obtain the three-dimensional mapping point Pw (xw ,yw ,0) of a certain pixel point P0 on the 2D lane line in the world coordinate system z=0.
综上所述，得到2D车道线上的某一像素点在世界坐标系z＝0下的映射点。因为z＝0为水平地面，所以2D车道线图像上的某一点，在不考虑道路坡度的情况时，即θ＝0时，在对应的水平地面上的三维映射点为Pw(xw,yw,0)。In summary, the mapping point of a certain pixel on the 2D lane line in the world coordinate system z=0 is obtained. Because z=0 is a horizontal ground, so for a certain point on the 2D lane line image, without considering the road slope, that is, when θ=0, the corresponding three-dimensional mapping point on the horizontal ground is Pw (xw , yw ,0).
3)求解考虑坡度信息的三维映射点P(x,y,z)。3) Solve the three-dimensional mapping point P(x, y, z) considering the slope information.
求解考虑坡度信息的三维映射点P(x,y,z)，参考图5和图6，图6中2D车道线图像上的P0点对应图5中道路斜坡上的P(x,y,z)，其投影到水平地面上的点为Pw(xw,yw,0)，而步骤2)已经得到Pw(xw,yw,0)。Solve the three-dimensional mapping point P(x,y,z) that takes the slope information into account. Refer to Figures 5 and 6. Point P0 on the 2D lane line image in Figure 6 corresponds to P(x,y, z), the point projected onto the horizontal ground is Pw (xw ,yw ,0), and step 2) has obtained Pw (xw ,yw ,0).
由于透射原理，点C、P、Pw在一条直线上，所以求解斜坡上P点三维坐标的问题就转化为求直线CPw与斜坡平面的交点的问题。Due to the transmission principle, points C, P, and Pw are on a straight line, so the problem of finding the three-dimensional coordinates of point P on the slope is transformed into the problem of finding the intersection of the straight line CPw and the slope plane.
设光线的方向向量α为：Let the direction vector α of the light be:
根据C、Pw的坐标可求得光线的方向向量为：According to the coordinates of C and Pw , the direction vector of the light can be obtained as:
已知光线过点C(xc,yc,zc)，且已知光线的方向向量为可列出光线CPw的方程，将其写成参数方程：It is known that the light passes through the point C (xc ,yc ,zc ), and the direction vector of the light is known to be The equation of light CPw can be listed and written as a parametric equation:
设斜坡平面经过点n(n1,n2,n3)，且斜坡平面由地平线绕x轴旋转θ角得到，斜坡平面的法向量为：Assume that the slope plane passes through point n (n1 , n2 , n3 ), and the slope plane is obtained by rotating the horizon around the x-axis by an angle θ. The normal vector of the slope plane is:
其中，n点位于世界坐标系中的水平地面上，可以任意选取，也可以直接取世界坐标系的原点，即车辆的中心。Among them, point n is located on the horizontal ground in the world coordinate system and can be selected arbitrarily, or it can directly take the origin of the world coordinate system, that is, the center of the vehicle.
将斜坡平面的点法式方程为：The point French equation of the slope plane is:
vp1*(x-n1)+vp2*(y-n2)+vp3*(z-n3)＝0 (13)vp1 *(xn1 )+vp2 *(yn2 )+vp3 *(zn3 )＝0 (13)
将式(11)与式(13)联立，得到：Combining equation (11) with equation (13), we get:
将已知量代入式(14)，可得t是一个关于θ的函数：Substituting the known quantities into equation (14), we can get that t is a function of θ:
其中，已知量包括n点的世界坐标，相机C的世界坐标、v1、v2、v3。Among them, the known quantities include the world coordinates of point n, the world coordinates of camera C, v1 , v2 , and v3 .
将t代入式(11)得到P(x,y,z)，此时的(x,y,z)仍然是一个有关θ的函数，Substitute t into equation (11) to get P(x, y, z). At this time, (x, y, z) is still a function about θ,
P(x,y,z)即三维映射点。P(x,y,z) is the three-dimensional mapping point.
4)遍历法求解坡度角θ。4) Use the ergodic method to solve the slope angle θ.
参考图7，在2D车道线图像的同一水平线上选取两个点Pi、Pj，假设Pi、Pj在坡度角为θ的倾斜道路上对应的点分别为wi、wj，由步骤3)可知，wi、wj的坐标是一个与θ相关的函数，wi与wj之间的距离|wiwj|也是一个与θ相关的函数。所以在一定范围内(例如0到45度)遍历所有的θ角，都可以得到相应的|wiwj|的值。已知道路的真实宽度为k，当使得|wiwj|-k的值取到最小时，认为此时的θ角为道路的倾斜坡度。Referring to Figure 7, select two points Pi and Pj on the same horizontal line of the 2D lane line image. Assume that the corresponding points of Pi and Pj on the inclined road with a slope angle of θ are wi and wj respectively. According to Step 3) It can be seen that the coordinates of wi and wj are a function related to θ, and the distance between wi and wj |wi wj | is also a function related to θ. Therefore, by traversing all θ angles within a certain range (such as 0 to 45 degrees), the corresponding values of |wi wj | can be obtained. It is known that the true width of the road is k. When the value of |wi wj |-k is minimized, the θ angle at this time is considered to be the slope of the road.
5)计算3D车道线坐标。5) Calculate 3D lane line coordinates.
代入θ计算出P(x,y,z)的具体值，将遍历法求得的θ代入式(16)，即可得到P(x,y,z)的具体数值(考虑道路真实坡度信息的3D车道线坐标)，即实现了基于单目图像的2D车道线到3D车道线的重构。Substituting θ to calculate the specific value of P(x,y,z), and substituting θ obtained by the ergodic method into equation (16), the specific value of P(x,y,z) can be obtained (considering the real slope information of the road) 3D lane line coordinates), which realizes the reconstruction of 2D lane lines to 3D lane lines based on monocular images.
以下对世界坐标系到相机坐标系的转换过程进行说明(与求解真实平坦道路上的三维映射点的原理相关)。The following describes the conversion process from the world coordinate system to the camera coordinate system (related to the principle of solving three-dimensional mapping points on a real flat road).
参考图5和图6，图6中2D车道线上的P0点映射到真实平坦道路上对应的点为图5中的Pw点。Referring to Figures 5 and 6, the P0 point on the 2D lane line in Figure 6 is mapped to the corresponding point on the real flat road as the Pw point in Figure 5.
设世界坐标系为(Xw,Yw,Zw)，相机坐标系为(Xc,Yc,Zc)，图像坐标系为(x,y)，像素坐标系为(u,v)。首先是世界坐标系到相机坐标系的转换，此步骤属于刚体变换，即物体不会发生形变，只需要进行旋转和平移。Let the world coordinate system be (Xw , Yw , Zw ), the camera coordinate system be (Xc , Yc , Zc ), the image coordinate system be (x, y), and the pixel coordinate system be (u, v) . The first is the conversion from the world coordinate system to the camera coordinate system. This step is a rigid body transformation, that is, the object will not deform and only needs to be rotated and translated.
世界坐标系到相机坐标系的变换可以表示为：The transformation from the world coordinate system to the camera coordinate system can be expressed as:
式中：T为平移矩阵，Rx，Ry，Rz为世界坐标系分别绕x，y，z轴转动α，β，γ角得到的旋转矩阵；式(101)可以进一步简写成为：In the formula: T is the translation matrix, Rx , Ry , and Rz are the rotation matrices obtained by rotating the world coordinate system by α, β, and γ angles around the x, y, and z axes respectively; Equation (101) can be further abbreviated as:
式中：R为3*3的旋转矩阵，T为3*1的平移矩阵；In the formula: R is a 3*3 rotation matrix, T is a 3*1 translation matrix;
相机坐标系到图像坐标系转换利用针孔成像原理，转换表达式为：The conversion from camera coordinate system to image coordinate system uses the pinhole imaging principle, and the conversion expression is:
式中：f为相机的焦距；In the formula: f is the focal length of the camera;
图像坐标系与像素坐标系都在同一成像平面上，只是各自的原点与度量单位不同，图像坐标系到像素坐标系的转换涉及尺度缩放与平移；The image coordinate system and the pixel coordinate system are both on the same imaging plane, but their respective origins and measurement units are different. The conversion of the image coordinate system to the pixel coordinate system involves scale scaling and translation;
设像素坐标系的原点为(u0,v0)，则二者的转换关系可以表示为：Assuming that the origin of the pixel coordinate system is (u0 , v0 ), the conversion relationship between the two can be expressed as:
式中：dx与dy表示每列或每行的单位像素所代表的毫米数，单位为毫米/像素；In the formula: dx and dy represent the number of millimeters represented by the unit pixel of each column or row, and the unit is mm/pixel;
上式写成矩阵的形式为：The above formula is written in the form of a matrix:
通过以上四个坐标系的转换可以得到世界坐标系到像素坐标系的转换关系：Through the transformation of the above four coordinate systems, the transformation relationship from the world coordinate system to the pixel coordinate system can be obtained:
式中：为相机的内参矩阵，/>为相机的外参矩阵；In the formula: is the internal parameter matrix of the camera,/> is the external parameter matrix of the camera;
内参矩阵与外参矩阵均可以通过相机的标定求得。Both the intrinsic parameter matrix and the extrinsic parameter matrix can be obtained through camera calibration.
所以通过以上步骤，可以计算出三维世界坐标系中的某一点在二维图像上与之对应的像素点的坐标。Therefore, through the above steps, the coordinates of the corresponding pixel point on the two-dimensional image for a certain point in the three-dimensional world coordinate system can be calculated.
本发明所述的像素坐标到世界坐标的映射，是以上步骤的逆过程。但由于二维到三维的映射中，无法获知式(107)中的深度信息Zc，所以二维到三维的映射并不是一个简单的求逆矩阵的过程。The mapping of pixel coordinates to world coordinates described in the present invention is the reverse process of the above steps. However, since the depth information Zc in equation (107) cannot be known in the two-dimensional to three-dimensional mapping, the two-dimensional to three-dimensional mapping is not a simple process of inverting the matrix.
像素坐标到世界坐标的映射通常有两种方法：第一种方法需要使用多个摄像机在不同空间上同时拍摄同一物体的两幅或者多幅图像，才可以进行测量；第二种方法只需要单个摄像机拍摄被测物体，但必须将被测物体放在一个已知的平面上。There are usually two methods for mapping pixel coordinates to world coordinates: the first method requires using multiple cameras to capture two or more images of the same object in different spaces at the same time before measurement can be performed; the second method only requires a single The camera captures the object being measured, but the object must be placed on a known plane.
本发明涉及的3D车道线重构是基于单目图像，所以选用第二种方法。The 3D lane line reconstruction involved in this invention is based on monocular images, so the second method is selected.
附图中给出了本发明的较佳实施例。但是，本发明可以以许多不同的形式来实现，并不限于本说明书所描述的实施例。相反地，提供这些实施例的目的是使对本发明的公开内容的理解更加透彻全面。Preferred embodiments of the invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. Rather, these embodiments are provided so that a thorough understanding of the present disclosure will be provided.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311658373.3ACN117611438B (en) | 2023-12-06 | 2023-12-06 | A method for reconstructing 2D lane lines to 3D lane lines based on monocular images |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311658373.3ACN117611438B (en) | 2023-12-06 | 2023-12-06 | A method for reconstructing 2D lane lines to 3D lane lines based on monocular images |
| Publication Number | Publication Date |
|---|---|
| CN117611438Atrue CN117611438A (en) | 2024-02-27 |
| CN117611438B CN117611438B (en) | 2024-10-11 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311658373.3AActiveCN117611438B (en) | 2023-12-06 | 2023-12-06 | A method for reconstructing 2D lane lines to 3D lane lines based on monocular images |
| Country | Link |
|---|---|
| CN (1) | CN117611438B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115995161A (en)* | 2023-02-01 | 2023-04-21 | 华人运通(上海)自动驾驶科技有限公司 | Method and electronic device for determining parking position based on projection |
| CN119559348A (en)* | 2024-11-27 | 2025-03-04 | 广汽埃安新能源汽车股份有限公司 | A method and device for generating 3D coordinates of lane lines |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018086764A1 (en)* | 2016-11-10 | 2018-05-17 | Lacos Computerservice Gmbh | Method for predictively generating data for controlling a travel path and an operating sequence for agricultural vehicles and machines |
| CN110148169A (en)* | 2019-03-19 | 2019-08-20 | 长安大学 | A kind of vehicle target 3 D information obtaining method based on PTZ holder camera |
| CN110176022A (en)* | 2019-05-23 | 2019-08-27 | 广西交通科学研究院有限公司 | A kind of tunnel overall view monitoring system and method based on video detection |
| CN112102413A (en)* | 2020-07-22 | 2020-12-18 | 西安交通大学 | Virtual lane line-based automatic calibration method for vehicle-mounted camera |
| CN115655205A (en)* | 2022-11-16 | 2023-01-31 | 清智汽车科技(苏州)有限公司 | Method and device for assisting distance measurement by using lane |
| CN116665166A (en)* | 2023-05-18 | 2023-08-29 | 南京航空航天大学 | Intelligent vehicle 3D target detection method suitable for uneven road surface scene |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018086764A1 (en)* | 2016-11-10 | 2018-05-17 | Lacos Computerservice Gmbh | Method for predictively generating data for controlling a travel path and an operating sequence for agricultural vehicles and machines |
| CN110148169A (en)* | 2019-03-19 | 2019-08-20 | 长安大学 | A kind of vehicle target 3 D information obtaining method based on PTZ holder camera |
| CN110176022A (en)* | 2019-05-23 | 2019-08-27 | 广西交通科学研究院有限公司 | A kind of tunnel overall view monitoring system and method based on video detection |
| CN112102413A (en)* | 2020-07-22 | 2020-12-18 | 西安交通大学 | Virtual lane line-based automatic calibration method for vehicle-mounted camera |
| CN115655205A (en)* | 2022-11-16 | 2023-01-31 | 清智汽车科技(苏州)有限公司 | Method and device for assisting distance measurement by using lane |
| CN116665166A (en)* | 2023-05-18 | 2023-08-29 | 南京航空航天大学 | Intelligent vehicle 3D target detection method suitable for uneven road surface scene |
| Title |
|---|
| 王源隆 等: "智能车辆车道线识别方法研究", 重庆理工大学学报, 31 December 2022 (2022-12-31)* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115995161A (en)* | 2023-02-01 | 2023-04-21 | 华人运通(上海)自动驾驶科技有限公司 | Method and electronic device for determining parking position based on projection |
| CN119559348A (en)* | 2024-11-27 | 2025-03-04 | 广汽埃安新能源汽车股份有限公司 | A method and device for generating 3D coordinates of lane lines |
| Publication number | Publication date |
|---|---|
| CN117611438B (en) | 2024-10-11 |
| Publication | Publication Date | Title |
|---|---|---|
| CN112102413B (en) | An automatic calibration method for vehicle-mounted cameras based on virtual lane lines | |
| CN103744086B (en) | A kind of high registration accuracy method of ground laser radar and close-range photogrammetry data | |
| US20220207776A1 (en) | Disparity image fusion method for multiband stereo cameras | |
| CN117611438B (en) | A method for reconstructing 2D lane lines to 3D lane lines based on monocular images | |
| US20230215187A1 (en) | Target detection method based on monocular image | |
| WO2021004312A1 (en) | Intelligent vehicle trajectory measurement method based on binocular stereo vision system | |
| JP7502440B2 (en) | Method for measuring the topography of an environment - Patents.com | |
| CN102938142B (en) | Based on the indoor LiDAR missing data complementing method of Kinect | |
| CN108802785A (en) | Vehicle method for self-locating based on High-precision Vector map and monocular vision sensor | |
| CN115239922A (en) | AR-HUD three-dimensional coordinate reconstruction method based on binocular camera | |
| CN107133985A (en) | A kind of vehicle-mounted vidicon automatic calibration method for the point that disappeared based on lane line | |
| CN107792179A (en) | A kind of parking guidance method based on vehicle-mounted viewing system | |
| CN100567886C (en) | A three-point calibration measurement method | |
| CN102829763B (en) | Pavement image collecting method and system based on monocular vision location | |
| CN108519102B (en) | A binocular visual odometry calculation method based on secondary projection | |
| CN111709985A (en) | A method of underwater target ranging based on binocular vision | |
| CN112329678B (en) | Monocular pedestrian 3D positioning method based on information fusion | |
| CN109685855A (en) | A kind of camera calibration optimization method under road cloud monitor supervision platform | |
| CN113781562A (en) | Lane line virtual and real registration and self-vehicle positioning method based on road model | |
| CN116309813A (en) | Solid-state laser radar-camera tight coupling pose estimation method | |
| CN112489106A (en) | Video-based vehicle size measuring method and device, terminal and storage medium | |
| CN116403186B (en) | FPN Swin Transformer and Pointnet ++ based automatic driving three-dimensional target detection method | |
| CN115239822A (en) | Real-time visual identification and positioning method and system for multi-module space of split type flying vehicle | |
| CN111476798A (en) | Vehicle space form recognition method and system based on contour constraint | |
| CN111145263A (en) | Vehicle-mounted-based automatic camera calibration method |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | Effective date of registration:20250826 Address after:311200 Zhejiang Province, Hangzhou City, Xiaoshan District, Ningwei Street, Xiaoshan Economic and Technological Development Zone, Qiaonan Block, Hongxing Road No. 477, Building 2, 3rd Floor, Room 301 Patentee after:Zhejiang Provincial Transportation Group Technology Research Institute Co.,Ltd. Country or region after:China Address before:310020 Zhejiang Province, Hangzhou City, Jianggan District, Mingzhu International Business Center Building 3, Room 509 Patentee before:Intelligent transportation research branch of Zhejiang Transportation Investment Group Co.,Ltd. Country or region before:China | |
| TR01 | Transfer of patent right |