CN102175221B

Movatterモバイル変換

Info

Publication number: CN102175221B
Application number: CN201110023561.XA
Authority: CN
Inventors: 张海威; 杨欣; 刘玉亭
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2011-01-20
Filing date: 2011-01-20
Publication date: 2014-04-09
Anticipated expiration: 2031-01-20
Also published as: CN102175221A

Abstract

本发明公开了基于鱼眼镜头的车载移动摄影测量系统，达到测量范围大、影像色彩信息丰富完整的特点。其技术方案为：系统包括数据采集装置、数据存储装置和数据处理装置，其中数据采集装置采集影像和数据，其中的鱼眼相机立体视觉模块包括鱼眼镜头、相机以及标定装置。数据存储装置存储采集到的影像数据。数据处理装置处理所采集到的数据，实现对鱼眼影像的测量。

The invention discloses a vehicle-mounted mobile photogrammetry system based on a fisheye lens, which has the characteristics of large measurement range and rich and complete image color information. The technical solution is: the system includes a data acquisition device, a data storage device and a data processing device, wherein the data acquisition device collects images and data, and the fisheye camera stereo vision module includes a fisheye lens, a camera and a calibration device. The data storage device stores the collected image data. The data processing device processes the collected data to realize the measurement of the fisheye image.

Description

Translated fromChinese

基于鱼眼镜头的车载移动摄影测量系统Vehicle-mounted mobile photogrammetry system based on fisheye lens

技术领域technical field

本发明涉及一种车载移动摄影测量系统，尤其涉及利用鱼眼镜头快速采集影像数据，通过立体视觉校正，对所获取的影像数据进行实时测量和三维建模的系统。The invention relates to a vehicle-mounted mobile photogrammetry system, in particular to a system for rapidly collecting image data with a fisheye lens, and performing real-time measurement and three-dimensional modeling on the acquired image data through stereo vision correction.

背景技术Background technique

随着城市化进程的加快，城市规模越来越大，变化日新月异，城市规划和管理变得越来越复杂，对于交通、能源、市政管理、基础设施建设等各个领域，信息的采集和更新显得尤为重要，特别是具有带有GIS信息的测量，成为这些领域信息获取不可或缺的一部分。比如，城市管理部门采集和统计城市各类资产数据，如区域内建筑的信息，广告牌位置、数量、长宽高等数据；道路交通部门统计道路资产数据；能源部门统计变电站的位置、数量等信息。这些数据通常都有数据量大，统计复杂的特点。如果使用人工实地测量统计的方式，工作量大，效率低，危险性高，并且容易出错。With the acceleration of the urbanization process, the scale of the city is getting bigger and bigger, the changes are changing with each passing day, and the urban planning and management are becoming more and more complicated. For various fields such as transportation, energy, municipal management, and infrastructure construction, the collection and update of information is becoming more and more important. It is especially important, especially to have measurements with GIS information, to become an integral part of information acquisition in these fields. For example, the city management department collects and counts various asset data in the city, such as building information in the area, billboard location, quantity, length, width and height data; the road traffic department collects and counts road asset data; the energy department counts information such as the location and quantity of substations . These data usually have the characteristics of large amount of data and complex statistics. If the method of manual field measurement and statistics is used, the workload is heavy, the efficiency is low, the risk is high, and it is easy to make mistakes.

目前市场存在的测量系统包括激光测量和立体视觉测量。激光测量，通常没有图像的色彩信息，分辨率低，数据量大，采集范围有限，后处理复杂耗时。而现有的立体视觉测量设备，都基于普通的相机，视野范围小，城市环境中高楼大厦鳞次栉比，比如拍摄城市中的高楼大厦，当距离很近时，只能拍到楼的一部分，如果要拍摄到整个大楼，则需要距离目标很远，这样影响目标物体的成像精度，如果要做基于图像的测量，则会影响到测量的精度，造成很大的误差。因此传统相机立体视觉测量系统受到极大挑战。Measurement systems that currently exist in the market include laser measurement and stereo vision measurement. Laser measurement usually has no image color information, low resolution, large amount of data, limited acquisition range, complex and time-consuming post-processing. However, the existing stereoscopic vision measurement equipment is based on ordinary cameras with a small field of view. There are many high-rise buildings in the urban environment. To capture the entire building, you need to be far away from the target, which affects the imaging accuracy of the target object. If you want to do image-based measurement, it will affect the measurement accuracy and cause a large error. Therefore, the traditional camera stereo vision measurement system is greatly challenged.

随着摄影器材的发展，出现一种具有超大视野范围的超广角镜头，鱼眼镜头，能在单张的鱼眼图像中实现视野范围达到甚至超过180度。同样是拍摄城市中的高楼大厦，鱼眼镜头能够同时保证目标物体的视野范围和精度，可以很好地解决大范围大视角的拍摄和测量的问题。With the development of photographic equipment, a super wide-angle lens with a large field of view, fisheye lens, can achieve a field of view of 180 degrees or even exceed 180 degrees in a single fisheye image. The same is to shoot high-rise buildings in the city. The fisheye lens can ensure the field of view and accuracy of the target object at the same time, and can well solve the problem of shooting and measuring in a large range and a large angle of view.

不过鱼眼镜头在获得很大的视野范围的同时，也给图像带来了很大的畸变，一般来说，视野范围越大，畸变也越大。传统的立体视觉测量是基于微小畸变的普通相机进行测量的，无法用大畸变的鱼眼相机进行测量。对配备普通镜头的相机来说，其成像原理类似于通常所说的针孔成像。在针孔成像中，光线是沿直线(线性)传播的。其优点是成像畸变很小甚至可以忽略，但其缺点也很明显，即视野范围(FOV)很小。对于大场景尤其是180度的无限场景来说，要通过基于光线直线传播的普通镜头在感光器件上成像的话，必须要求感光器件无限大，这在实际中显然是不可能实现的。因此只有改变光线的传播方向，让其进入镜头后不再沿直线传播，而是沿曲线(非线性)投射到感光器件上，这样就可以在有限的感光器件上容纳无限的场景，鱼眼镜头就是这样做的。但光线的曲线传播同时也带来几个问题：首先，由于光线是沿曲线传播的，这给确定其传播路径带来很大的困难。其次，虽然都是沿曲线传播，但不同视角范围内的光线其传播路径的曲率是不一样的，视角越大，曲率越大。表现在图像中，就是越靠近镜头中心图像畸变越小，越靠近镜头边缘图像畸变越大。因此给所有视角范围内的光线确定一个统一的传播模型就变得更加困难。再次，由于镜头边缘的畸变太大，图像边缘上的几个像素所包含的内容相当于图像中心处数十个甚至上百个像素所包含的内容。因此在图像边缘上一个像素的偏差相当于图像中心处数十上百个像素的偏差，这给模型的精确性带来了非常大的挑战。综上所述，鱼眼镜头目前只应用于安防监控等单纯需要大视角信息的场合，却很少用于测量或者三维建模等对精度要求高的场合。However, while the fisheye lens obtains a large field of view, it also brings great distortion to the image. Generally speaking, the larger the field of view, the greater the distortion. The traditional stereo vision measurement is based on the ordinary camera with small distortion, which cannot be measured with the fisheye camera with large distortion. For a camera equipped with an ordinary lens, its imaging principle is similar to the so-called pinhole imaging. In pinhole imaging, light travels in straight lines (linear). Its advantage is that the imaging distortion is small or even negligible, but its disadvantage is also obvious, that is, the field of view (FOV) is very small. For large scenes, especially 180-degree infinite scenes, if an ordinary lens based on the straight line propagation of light is used to form an image on a photosensitive device, the photosensitive device must be infinitely large, which is obviously impossible to achieve in practice. Therefore, the only way to change the direction of propagation of light is to let it enter the lens and no longer travel along a straight line, but to project onto the photosensitive device along a curve (non-linear), so that unlimited scenes can be accommodated on a limited photosensitive device. Fisheye lens That's what it does. However, the curved propagation of light also brings several problems: first, since the light propagates along the curve, it brings great difficulties to determine its propagation path. Secondly, although they all propagate along the curve, the curvature of the propagation path of light in different viewing angles is different. The larger the viewing angle, the greater the curvature. It is manifested in the image, that is, the closer to the center of the lens, the smaller the image distortion, and the closer to the edge of the lens, the greater the image distortion. It thus becomes more difficult to determine a uniform propagation model for rays in all viewing angles. Again, due to the large distortion at the edge of the lens, a few pixels on the edge of the image contain as much content as dozens or even hundreds of pixels in the center of the image. Therefore, the deviation of one pixel on the edge of the image is equivalent to the deviation of tens or hundreds of pixels at the center of the image, which poses a great challenge to the accuracy of the model. To sum up, fisheye lenses are currently only used in security monitoring and other occasions that simply require large viewing angle information, but are rarely used in measurement or 3D modeling and other occasions that require high precision.

通过对鱼眼相机进行精确数学建模来获取鱼眼相机的精确成像参数(光线的传播路径)，进一步地获取鱼眼相机立体视觉对的参数(鱼眼相机间的旋转和平移)，就可以运用大畸变的鱼眼相机来实现立体视觉测量。这个获取鱼眼相机的成像参数以及鱼眼相机立体视觉对参数的过程称之为鱼眼相机立体视觉对的标定。普通相机的标定已有非常成熟的解决方案，比如张正友、Tsai的方法。然而因为成像原理不同，这些方法不能用于标定鱼眼镜头，必须找到新的数学模型来描述鱼眼镜头的成像过程。近来出现一种基于多项式拟合的技术较好地解决了这一问题。By performing precise mathematical modeling on the fisheye camera to obtain the precise imaging parameters of the fisheye camera (the propagation path of light), and further obtain the parameters of the stereoscopic vision pair of the fisheye camera (rotation and translation between fisheye cameras), you can A large distortion fisheye camera is used to realize stereo vision measurement. The process of obtaining the imaging parameters of the fisheye camera and the parameters of the stereoscopic vision pair of the fisheye camera is called the calibration of the stereoscopic vision pair of the fisheye camera. There are very mature solutions for the calibration of ordinary cameras, such as the methods of Zhang Zhengyou and Tsai. However, due to the different imaging principles, these methods cannot be used to calibrate fisheye lenses, and a new mathematical model must be found to describe the imaging process of fisheye lenses. Recently, a technology based on polynomial fitting has appeared to solve this problem better.

如果能解决上述问题，提供一个基于鱼眼镜头的立体视觉的测量系统解决方案，即可以解决现有测量系统视野范围小、无法在测量的同时获取全面的彩色影像等问题，将为城市管理、能源、统计、勘测等各部门的数据采集和测量提供一种全新而简便的集成化解决方案，为城市的数字化建设提供新的解决方案。If the above problems can be solved, a fisheye lens-based stereo vision measurement system solution can be provided, which can solve the problems of the existing measurement system such as the small field of view and the inability to obtain comprehensive color images while measuring, which will provide urban management, It provides a new and simple integrated solution for data collection and measurement of various departments such as energy, statistics, surveying, etc., and provides a new solution for the digital construction of the city.

发明内容Contents of the invention

本发明的目的在于解决上述问题，提供了一种基于鱼眼镜头的车载移动摄影测量系统，能克服上述不足，达到测量范围大、影像色彩信息丰富完整的特点。The purpose of the present invention is to solve the above problems and provide a vehicle-mounted mobile photogrammetry system based on a fisheye lens, which can overcome the above shortcomings and achieve the characteristics of large measurement range and rich and complete image color information.

本发明的技术方案为：本发明揭示了一种基于鱼眼镜头的车载移动摄影测量系统，包括数据采集装置、数据存储装置和数据处理装置，其中：The technical solution of the present invention is: the present invention discloses a vehicle-mounted mobile photogrammetry system based on a fisheye lens, including a data acquisition device, a data storage device and a data processing device, wherein:

数据采集装置，用于采集影像和数据，其包括鱼眼相机立体视觉模块，用于采集鱼眼图像信息，鱼眼相机立体视觉模块包括鱼眼镜头、相机以及标定装置，其中：The data acquisition device is used to collect images and data, and it includes a fisheye camera stereo vision module for collecting fisheye image information. The fisheye camera stereo vision module includes a fisheye lens, a camera and a calibration device, wherein:

鱼眼镜头，与相机连接，获取鱼眼影像；Fisheye lens, connected with the camera, to obtain fisheye images;

相机，与鱼眼镜头连接，接收鱼眼镜头获取的鱼眼影像并进行影像采集；The camera is connected with the fisheye lens, receives the fisheye image acquired by the fisheye lens and collects the image;

标定装置，获取鱼眼镜头的成像参数以及鱼眼镜头立体视觉对参数；A calibration device for obtaining imaging parameters of the fisheye lens and stereoscopic vision pair parameters of the fisheye lens;

数据存储装置，耦接数据采集装置，用于存储采集到的影像数据；A data storage device, coupled to the data acquisition device, for storing the collected image data;

数据处理装置，耦接数据存储装置，用于处理所采集到的数据，实现对鱼眼影像的测量。The data processing device, coupled with the data storage device, is used to process the collected data and realize the measurement of the fisheye image.

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，数据采集装置还包括：According to an embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention, the data acquisition device also includes:

车辆位姿获取模块，用于获取车辆的位置、姿态、运行距离的信息；The vehicle pose acquisition module is used to acquire information about the position, attitude, and running distance of the vehicle;

数据存储装置还存储车辆位姿获取模块得到的车辆的位置、姿态、运行距离的信息。The data storage device also stores information on the vehicle's position, attitude, and running distance obtained by the vehicle pose acquisition module.

全景采集模块，用于采集连续的道路全景影像。The panorama acquisition module is used to acquire continuous road panorama images.

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，鱼眼相机立体视觉模块至少包括两台鱼眼相机。According to an embodiment of the vehicle-mounted mobile photogrammetry system based on a fisheye lens of the present invention, the fisheye camera stereo vision module includes at least two fisheye cameras.

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，标定装置包括：According to an embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention, the calibration device includes:

鱼眼成像关系建立模块，建立半单位球面模型，并在单位球面模型上建立鱼眼成像关系；The fisheye imaging relationship building module establishes a semi-unit spherical model and establishes a fisheye imaging relationship on the unit spherical model;

初始化内参模块，耦接鱼眼成像关系建立模块，初始化内参，其中内参是鱼眼相机自身的参数，与外部环境无关；Initialize the internal reference module, couple the fisheye imaging relationship establishment module, and initialize the internal reference, where the internal reference is the parameter of the fisheye camera itself and has nothing to do with the external environment;

单应矩阵计算模块，耦接初始化内参模块，计算单应矩阵；The homography matrix calculation module, coupled with the initialization internal reference module, calculates the homography matrix;

初始化外参模块，耦接单应矩阵计算模块，初始化外参，其中外参是鱼眼相机与外部环境之间的参数；Initialize the external parameter module, couple the homography matrix calculation module, and initialize the external parameter, where the external parameter is the parameter between the fisheye camera and the external environment;

迭代优化模块，耦接初始化外参模块，LM迭代最小化重投影误差，得到优化后的内参与外参。The iterative optimization module is coupled with the initialization extrinsic parameter module, and the LM iteratively minimizes the reprojection error to obtain the optimized internal and external parameters.

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，鱼眼成像关系建立模块的处理如下：According to an embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention, the processing of the fisheye imaging relationship establishment module is as follows:

记空间中某一点x在鱼眼图像上的成像点为(u，v)，空间点x指向单位球球心的入射角度为

其中θ是入射光线与单位球Z轴正方向的夹角，

是入射光线在单位球XY平面上的投影与单位球X轴正方向的夹角，则由入射光线的入射角度到鱼眼图像上的成像点(u，v)的鱼眼成像模型由以下方程描述：Note that the imaging point of a certain point x in the space on the fisheye image is (u, v), and the incident angle of the space point x pointing to the center of the unit sphere is

Where θ is the angle between the incident ray and the positive direction of the Z axis of the unit sphere,

is the angle between the projection of the incident ray on the XY plane of the unit sphere and the positive direction of the X-axis of the unit sphere, then the angle of incidence of the incident ray The fisheye imaging model to the imaging point (u, v) on the fisheye image is described by the following equation:

r(θ)＝k₁θ+k₂θ²+k₃θ³+k₄θ⁴+k₅θ⁵+...k_nθⁿ (1)r(θ)=k₁ θ+k₂ θ² +k₃ θ³ +k₄ θ⁴ +k₅ θ⁵ +...k_n θⁿ (1)

r表示图像上的某个像素点到图像主点的距离，k₁，...k_n是鱼眼镜头的成像参数；r represents the distance from a certain pixel on the image to the main point of the image, k₁ ,... k_n are the imaging parameters of the fisheye lens;

Δ_r表示鱼眼镜头的径向畸变，l₁，...l_n，i₁，...i₄为径向畸变参数；Δ_r represents the radial distortion of the fisheye lens, l₁ ,...l_n , i₁ ,...i₄ are radial distortion parameters;

Δ_t表示鱼眼镜头的切向畸变，m₁，...m_n，j₁，...j₄为切向畸变参数；_Δt represents the tangential distortion of the fisheye lens, m₁ ,...m_n , j₁ ,...j₄ are tangential distortion parameters;

x_d为像素点的位置向量，即(x_d，y_d)，u_r为径向单位向量，

为切向单位向量；x_d is the position vector of the pixel point, namely (x_d , y_d ), u_r is the radial unit vector,

is the tangential unit vector;

$(\begin{matrix} u u \\ v v \end{matrix}) = = (\begin{matrix} {m m}_{u u} & 00 \\ 00 & {m m}_{v v} \end{matrix}) (\begin{matrix} {x x}_{d d} \\ {y the y}_{d d} \end{matrix}) + + (\begin{matrix} {u u}_{00} \\ {v v}_{00} \end{matrix}) - - - - - - ((55))$

(u₀，v₀)为图像的主点坐标，(m_u，m_v)分别为CCD水平与垂直方向上单位距离上的像素数，(k₁，k₂，k₃，k₄，k₅，l₁，l₂，l₃，i₁，i₂，i₃，i₄，m₁，m₂，m₃，j₁，j₂，j₃，j₄，m_u，m_v，u₀，v₀)为待标定的鱼眼镜头的参数。(u₀ , v₀ ) are the principal point coordinates of the image, (m_u , m_v ) are the number of pixels per unit distance in the horizontal and vertical directions of the CCD, respectively, (k₁ , k₂ , k₃ , k₄ , k₅ , l₁ , l₂ , l₃ , i 1 , i₂ , i₃ ,_i₄ , m₁ , m₂ , m₃ , j 1 , j₂ , j₃_, j₄ ,_mu , m_v , u₀ , v₀ ) are the parameters of the fisheye lens to be calibrated.

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，初始化内参模块的处理如下：According to an embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention, the process of initializing the internal reference module is as follows:

读取镜头参数，镜头参数包括焦距f和最大视角θ_max；Read lens parameters, lens parameters include focal length f and maximum viewing angle θ_max ;

令k₁＝f，r_max＝fθ_max；Let k₁ = f, r_max = fθ_max ;

检测鱼眼图像边界，对边界点进行椭圆拟合：

求得u₀，v₀，a，b，则

其中(a，b)为椭圆的长短半轴，(u₀，v₀)为椭圆圆心；Detect the boundary of the fisheye image and perform ellipse fitting on the boundary points:

Find u₀ , v₀ , a, b, then

Where (a, b) is the semi-axis of the ellipse, (u₀ , v₀ ) is the center of the ellipse;

其它的参数设置为0。Other parameters are set to 0.

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，单应矩阵计算模块包括：According to an embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention, the homography matrix calculation module includes:

棋盘格图像读取单元，读取标定板上的棋盘格图像；The checkerboard image reading unit reads the checkerboard image on the calibration board;

交叉点选取单元，耦接棋盘格图像读取单元，在每张棋盘格图像上依次选取棋盘格顶点处的4个交叉点；The intersection selection unit is coupled to the checkerboard image reading unit, and sequentially selects 4 intersection points at the vertices of the checkerboard on each checkerboard image;

反投影单元，耦接交叉点选取单元，利用初始化内参，将交叉点

反投影到单位球上得到单位向量

其中j为第j幅图像，i为第i个棋盘格交叉点；The back-projection unit is coupled to the intersection point selection unit, and the intersection point is set to

Back-project onto the unit sphere to get the unit vector

Where j is the jth image, i is the ith checkerboard intersection point;

单应矩阵估计单元，耦接反投影单元，估计单应矩阵H_j，将单位向量表示为

单位向量与标定板上的空间点xⁱ之间存在单应变换H_j，通过线性算法估计单应变换H_j，得到标定板上的空间点xⁱ在单应变换H_j下的对应点：

其中x_pⁱ是第j幅棋盘格图像上第i个交叉点的空间坐标；The homography matrix estimation unit, coupled with the back-projection unit, estimates the homography matrix H_j , and converts the unit vector Expressed as

There is a homography transformation H_j between the unit vector and the spatial point xⁱ on the calibration board. The homography transformation H_j is estimated by a linear algorithm, and the corresponding point of the spatial point xⁱ on the calibration board under the homography transformation H_j is obtained:

where x_pⁱ is the spatial coordinate of the i-th intersection point on the j-th checkerboard image;

单应矩阵优化单元，耦接单应矩阵估计单元，通过LM迭代最小化误差函数

以优化单应矩阵H_j，其中是向量

和

之间的夹角；The homography matrix optimization unit, coupled with the homography matrix estimation unit, minimizes the error function through LM iterations

To optimize the homography matrix H_j , where is a vector

and

the angle between

交叉点映射单元，耦接单应矩阵优化单元，将标定板上的所有交叉点通过优化后的单应矩阵H_j映射到单位球上得到对应点：

The intersection mapping unit, coupled with the homography matrix optimization unit, maps all the intersection points on the calibration board to the unit sphere through the optimized homography matrix_Hj to obtain the corresponding points:

交叉点图像坐标获取单元，耦接交叉点映射单元，将单位向量变换到图像上：

在投影点

的邻域寻找交叉点的图像坐标

The intersection image coordinate acquisition unit is coupled with the intersection mapping unit to transform the unit vector onto the image:

at the projected point

Image coordinates for finding the intersection point of the neighborhood of

交叉点选取单元，耦接棋盘格图像读取单元，在每张棋盘格图像上依次选取所有的棋盘格交叉点；The intersection selection unit is coupled to the checkerboard image reading unit, and sequentially selects all checkerboard intersections on each checkerboard image;

反投影到单位球上得到单位向量

Back-project onto the unit sphere to get the unit vector

Where j is the jth image, i is the ith checkerboard intersection point;

单应矩阵估计单元，耦接反投影单元，估计单应矩阵H_j，将单位向量

表示为

单位向量与标定板上的空间点xⁱ之间存在单应变换H_j，通过线性算法估计单应变换H_j，得到标定板上的空间点xⁱ在单应变换H_j下的对应点：其中x_pⁱ是第j幅棋盘格图像上第i个交叉点的空间坐标；The homography matrix estimation unit, coupled with the back-projection unit, estimates the homography matrix H_j , and converts the unit vector

Expressed as

There is a homography transformation H_j between the unit vector and the spatial point xⁱ on the calibration board. The homography transformation H_j is estimated by a linear algorithm, and the corresponding point of the spatial point xⁱ on the calibration board under the homography transformation H_j is obtained: where x_pⁱ is the spatial coordinate of the i-th intersection point on the j-th checkerboard image;

以优化单应矩阵H_j，其中

是向量

和

之间的夹角。The homography matrix optimization unit, coupled with the homography matrix estimation unit, minimizes the error function through LM iterations

To optimize the homography matrix H_j , where

is a vector

and

angle between.

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，初始化外参模块的处理如下：According to an embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention, the process of initializing the external parameter module is as follows:

外参

由单应矩阵H_j初始化如下：External reference

Initialized by the homography matrix_Hj as follows:

${r r}_{j j}^{11} = = {λ λ}_{j j} {h h}_{j j}^{11},,$ ${r r}_{j j}^{22} = = {λ λ}_{j j} {h h}_{j j}^{22},,$ ${r r}_{j j}^{33} = = {r r}_{j j}^{11} \times \times {r r}_{j j}^{22},,$ ${t t}_{j j} = = {λ λ}_{j j} {h h}_{j j}^{33}$

其中，

R_j为旋转参数，T_j为位移参数，

为第j个单应矩阵H_j的第i个列向量。in,

R_j is the rotation parameter, T_j is the displacement parameter,

is the i-th column vector of the j-th homography matrix H_j .

根据本发明的基于鱼眼镜头的车载移动摄影测量系统的一实施例，迭代优化模块的处理如下：According to an embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention, the processing of the iterative optimization module is as follows:

LM迭代最小化重投影误差

得到优化后的内参和外参，其中

为图像点

之间的像素距离，M为每幅棋盘格图像上的交叉点数量，N为棋盘格图像数量。LM iteratively minimizes reprojection error

Get the optimized internal and external references, where

is the image point

The pixel distance between, M is the number of intersections on each checkerboard image, and N is the number of checkerboard images.

本发明对比现有技术有如下的有益效果：本发明的主要特点是将鱼眼相机立体视觉模块应用于车载测量系统，利用鱼眼镜头视野范围大的特点，实现大范围的测量。通过车辆位姿获取单元，可以得到所采集图像的位置及距离信息，包括经纬度、姿态、以及车辆运行的距离等，加上全景采集模块，同时获取具有丰富色彩信息的全景影像数据。从而解决了现有测量系统测量范围小，图像色彩少或信息不完全的问题，可让用户快速获取所需要的信息，尤其针对城市环境，大大提升了测量的效率和精准度，为数据统计和更新，特别是城市建设、规划、勘测提供了更加简易和全面的方法。Compared with the prior art, the present invention has the following beneficial effects: the main feature of the present invention is that the fisheye camera stereo vision module is applied to the vehicle-mounted measurement system, and the large-scale measurement is realized by utilizing the characteristic of the large field of view of the fisheye lens. Through the vehicle pose acquisition unit, the position and distance information of the collected images can be obtained, including latitude and longitude, attitude, and the distance of the vehicle, etc., plus the panoramic acquisition module, and panoramic image data with rich color information can be obtained at the same time. In this way, the problems of small measurement range, few image colors or incomplete information of the existing measurement system are solved, allowing users to quickly obtain the required information, especially for urban environments, greatly improving the efficiency and accuracy of measurement, providing data statistics and Updates, especially for urban construction, planning, and surveying, provide a simpler and more comprehensive approach.

附图说明Description of drawings

图1示例性的示出了本发明的基于鱼眼镜头的车载移动摄影测量系统的第一实施例的原理图。Fig. 1 exemplarily shows the schematic diagram of the first embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention.

图2示例性的示出了本发明的基于鱼眼镜头的车载移动摄影测量系统的第二实施例的原理图。Fig. 2 exemplarily shows the schematic diagram of the second embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention.

图3示例性的示出了本发明的基于鱼眼镜头的车载移动摄影测量系统的第三实施例的原理图。Fig. 3 exemplarily shows the schematic diagram of the third embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention.

图4示例性的示出了本发明的标定装置的实施例的原理图。Fig. 4 exemplarily shows a principle diagram of an embodiment of the calibration device of the present invention.

图5示例性的示出了本发明的标定装置中的单应矩阵计算模块的一种示例的细化原理图。Fig. 5 exemplarily shows a refined schematic diagram of an example of the homography matrix calculation module in the calibration device of the present invention.

图6示例性的示出了本发明的标定装置中的单应矩阵计算模块的另一种示例的细化原理图。Fig. 6 exemplarily shows a refined principle diagram of another example of the homography matrix calculation module in the calibration device of the present invention.

图7示例性的示出了本发明的半单位球面模型的示意图。Fig. 7 exemplarily shows a schematic diagram of a semi-unit sphere model of the present invention.

具体实施方式Detailed ways

下面结合附图和实施例对本发明作进一步的描述。The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

在介绍本发明的各个实施例之前，还要对鱼眼图像(影像)作一个说明：鱼眼影像是指通过鱼眼镜头汇聚光线透射到感光成像元件上并存储获得的影像，由于是通过鱼眼镜头获取的，则称这类照片为鱼眼影像。Before introducing the various embodiments of the present invention, an explanation will be made to the fisheye image (image): the fisheye image refers to the image obtained by converging light through the fisheye lens and transmitting it to the photosensitive imaging element and storing it. Such photos are called fisheye images.

介绍本发明的各个实施例之前，还要对鱼眼相机做一个说明：将相机与鱼眼镜头连接，所生成的影像能获得超广角的视野范围，则称这类相机为鱼眼相机。Before introducing the various embodiments of the present invention, an explanation must be made about the fisheye camera: when the camera is connected with a fisheye lens, the generated image can obtain a super wide-angle field of view, and this type of camera is called a fisheye camera.

在介绍本发明的各个实施例之前，首先对全景或者全景图像(影像)作一个说明：若将图像放在一个球空间、或立方体空间、或柱形空间、或锥形空间、或椭球空间中，通过上述空间中固定一点作为观察点，通过该观察点采用单点透视该图像所获得的影像的过程即为全景播放，则该图像被对应的称为球形全景、或立方体全景、或柱形全景、或锥形全景、或椭球全景，并且统称该图像为全景或全景图像(影像)。Before introducing each embodiment of the present invention, at first make an explanation to panorama or panorama image (image): if image is placed on a spherical space, or cube space, or cylindrical space, or cone space, or ellipsoid space In the above-mentioned space, by fixing a point in the above space as the observation point, the process of using the single-point perspective of the image to obtain the image through the observation point is the panorama playback, and the image is called a spherical panorama, or a cube panorama, or a cylindrical panorama. Shaped panorama, or conical panorama, or ellipsoidal panorama, and the image is collectively referred to as panorama or panorama image (image).

基于鱼眼镜头的车载移动摄影测量系统的第一实施例The first embodiment of the vehicle-mounted mobile photogrammetry system based on fisheye lens

图1示出了本发明的基于鱼眼镜头的车载移动摄影测量系统的第一实施例。请参见图1。本实施例的基于鱼眼镜头的车载移动摄影测量系统包括数据采集装置10a、数据存储装置20a、数据处理单元装置30a。Fig. 1 shows the first embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention. See Figure 1. The vehicle-mounted mobile photogrammetry system based on a fisheye lens in this embodiment includes adata acquisition device 10a, adata storage device 20a, and a dataprocessing unit device 30a.

这三个装置之间的连接关系是：数据采集装置10a的输出端连接数据存储装置20a，数据存储装置20a的输出端连接数据处理装置30a。The connection relationship between these three devices is: the output end of thedata collection device 10a is connected to thedata storage device 20a, and the output end of thedata storage device 20a is connected to thedata processing device 30a.

数据采集装置10a包括鱼眼相机立体视觉模块102a，由至少一个立体视觉对组成，即两台鱼眼相机。鱼眼立体相机立体视觉模块102a包括鱼眼镜头1022a，CCD相机1024a和标定装置1026a。鱼眼镜头1022a连接到CCD相机1024a上，组成可以生成超广角视野的鱼眼影像。标定装置1026a用于获取鱼眼相机的成像参数以及鱼眼相机立体视觉对参数。Thedata acquisition device 10a includes a fisheye camera stereo vision module 102a, which is composed of at least one stereo vision pair, that is, two fisheye cameras. The fisheye stereo camera stereo vision module 102a includes afisheye lens 1022a, aCCD camera 1024a and acalibration device 1026a. Thefisheye lens 1022a is connected to theCCD camera 1024a to form a fisheye image that can generate a super wide-angle field of view. Thecalibration device 1026a is used to acquire the imaging parameters of the fisheye camera and the stereo vision pair parameters of the fisheye camera.

标定装置1026a是采用基于多项式拟合的技术来建立数学模型并描述鱼眼镜头的成像过程。Thecalibration device 1026a adopts a technique based on polynomial fitting to establish a mathematical model and describe the imaging process of the fisheye lens.

标定装置1026a的细化原理如图4所示，鱼眼成像关系建立模块40、初始化内参模块50、单应矩阵计算模块60、初始化外参模块70、迭代优化模块80。The refinement principle of thecalibration device 1026a is shown in FIG. 4 , the fisheye imagingrelationship establishment module 40 , the initializationinternal reference module 50 , the homographymatrix calculation module 60 , the initializationexternal parameter module 70 , and theiterative optimization module 80 .

这些模块之间的连接关系是：鱼眼成像关系建立模块40的输出端耦接初始化内参模块50，初始化内参模块50的输出端连接单应矩阵计算模块60，单应矩阵计算模块60的输出端耦接初始化外参模块70，初始化外参模块70的输出端耦接迭代优化模块80。The connection relationship between these modules is: the output end of the fisheye imagingrelationship establishment module 40 is coupled to the initializationinternal reference module 50, the output end of the initializationinternal reference module 50 is connected to the homographymatrix calculation module 60, and the output end of the homographymatrix calculation module 60 It is coupled to the initializationexternal parameter module 70 , and the output terminal of the initializationexternal parameter module 70 is coupled to theiterative optimization module 80 .

以下是标定装置1026a的其中一种示例的实施情况。The following is one example implementation of thecalibration device 1026a.

鱼眼成像关系建立模块40用于建立半单位球面模型，并在单位球面模型上建立鱼眼成像关系。The fisheye imagingrelationship establishing module 40 is used to establish a semi-unit spherical model, and establish a fisheye imaging relationship on the unit spherical model.

半单位球面模型的示意如图7所示，记空间中一点x，其在鱼眼图像上的成像点为(u，v)，其指向单位球球心的入射光线的入射角度为

其中θ是入射光线与单位球Z轴正方向的夹角，

是入射光线在单位球XY平面上的投影与单位球X轴正方向的夹角，则由入射光线的入射角度

到鱼眼图像上的成像点(u，v)的鱼眼成像模型由以下方程描述：The schematic diagram of the semi-unit sphere model is shown in Figure 7, record a point x in the space, its imaging point on the fisheye image is (u, v), and the incident angle of the incident light pointing to the center of the unit sphere is

is the angle between the projection of the incident ray on the XY plane of the unit sphere and the positive direction of the X-axis of the unit sphere, then the angle of incidence of the incident ray

The fisheye imaging model to the imaging point (u, v) on the fisheye image is described by the following equation:

r表示图像上的某个像素点到图像主点的距离，k₁，...k_n是鱼眼镜头的成像参数。r represents the distance from a certain pixel on the image to the main point of the image, k₁ ,...k_n are the imaging parameters of the fisheye lens.

Δ_r表示鱼眼镜头的径向畸变，l₁，...l_n，i₁，...i₄为径向畸变参数。Δ_r represents the radial distortion of the fisheye lens, and l₁ ,...l_n , i₁ ,...i₄ are radial distortion parameters.

Δ_t表示鱼眼镜头的切向畸变，m₁，...m_n，j₁，...j₄为切向畸变参数。_Δt represents the tangential distortion of the fisheye lens, and m₁ ,...m_n , j₁ ,...j₄ are tangential distortion parameters.

x_d为像素点的位置向量，即(x_d，y_d)，u_r为径向单位向量，

为切向单位向量。x_d is the position vector of the pixel point, namely (x_d , y_d ), u_r is the radial unit vector,

is a tangential unit vector.

(u₀，v₀)为图像的主点坐标，(m_u，m_v)分别为CCD水平与垂直方向上单位距离上的像素数。(u₀ , v₀ ) are the principal point coordinates of the image, (m_u , m_v ) are the number of pixels per unit distance in the horizontal and vertical directions of the CCD, respectively.

其中已知信息为空间点x的入射角度

以及空间点x在图像上的对应点(u，v)，其余参数(k₁，k₂，k₃，k₄，k₅，l₁，l₂，l₃，i₁，i₂，i₃，i₄，m₁，m₂，m₃，j₁，j₂，j₃，j₄，m_u，m_v，u₀，v₀)为待标定的鱼眼镜头参数。需要指出的是，公式(1)、(2)、(3)中的多项式的阶次可以到无穷次，实验中发现，公式(1)中多项式阶次取到5次，公式(2)、公式(3)中多项式阶次取到3次已经能够精确建模鱼眼镜头成像过程。The known information is the incident angle of the spatial point x

And the corresponding point (u, v) of the spatial point x on the image, the remaining parameters (k₁ , k₂ , k₃ , k₄ , k₅ , l₁ , l₂ , l₃ , i₁ , i₂ , i₃ , i₄ , m₁ , m₂ , m₃ , j₁ , j₂ , j₃ , j₄ , mu_u , m_v , u₀ , v₀ ) are the fisheye lens parameters to be calibrated. It should be pointed out that the order of polynomials in formulas (1), (2), and (3) can reach infinite times. It is found in experiments that the order of polynomials in formula (1) is taken to 5 times, and the order of polynomials in formulas (2), Taking the order of the polynomial in formula (3) to three times can already accurately model the imaging process of the fisheye lens.

初始化内参模块50用于初始化内参。The initializationinternal reference module 50 is used for initializing internal references.

初始化内参的具体处理如下。读取厂商提供的镜头参数：焦距f以及最大视角θ_max令k₁＝f，r_max＝fθ_max。检测鱼眼图像边界，对边界点进行椭圆拟合：The specific process of initializing internal parameters is as follows. Read the lens parameters provided by the manufacturer: the focal length f and the maximum viewing angle θ_max set k₁ =f, r_max =fθ_max . Detect the boundary of the fisheye image and perform ellipse fitting on the boundary points:

${((\frac{u u - - {u u}_{00}}{a a}))}^{22} + + {((\frac{v v - - {v v}_{00}}{b b}))}^{22} = = 11 . .$

然后根据椭圆拟合求得u₀，v₀，a，b，则

其中(a，b)为椭圆的长短半轴，(u₀，v₀)为椭圆圆心。并将其它参数设置为0。Then get u₀ , v₀ , a, b according to ellipse fitting, then

Where (a, b) is the semi-major and minor axes of the ellipse, and (u₀ , v₀ ) is the center of the ellipse. And set other parameters to 0.

内参是指鱼眼相机自身的参数，与外部环境无关，本实施例中所指的内参为：(k₁，k₂，k₃，k₄，k₅，l₁，l₂，l₃，i₁，i₂，i₃，i₄，m₁，m₂，m₃，j₁，j₂，j₃，j₄，m_u，m_v，u₀，v₀)需要指出的是，公式(1)、(2)、(3)中的多项式的阶次可以到无穷次，实验中发现，公式(1)中多项式阶次取到5次，公式(2)、公式(3)中多项式阶次取到3次已经能够精确建模鱼眼镜头成像过程。The internal reference refers to the parameters of the fisheye camera itself, which has nothing to do with the external environment. The internal reference referred to in this embodiment is:₍ k₁ , k₂ , k₃ , k 4 , k₅ , l₁ , l₂ , l₃ , i₁ , i₂ , i₃ , i₄ , m 1 , m₂ , m₃ , j₁ , j₂ , j₃ , j₄ , m_u , m_v ,_{u 0}_, v₀ ) It should be noted that, The order of polynomials in formulas (1), (2), and (3) can reach infinite times. It is found in experiments that the order of polynomials in formula (1) is taken to 5 times, and the order of polynomials in formulas (2) and (3) Taking the polynomial order to 3 times has been able to accurately model the imaging process of the fisheye lens.

单应矩阵计算模块60用于计算单应矩阵。The homographymatrix calculation module 60 is used to calculate the homography matrix.

计算单应矩阵的原理如图5所示，包括棋盘格图像读取单元600、交叉点选取单元601、反投影单元602、单应矩阵估计单元603、单应矩阵优化单元604、交叉点映射单元605以及交叉点图像坐标获取单元606。The principle of calculating the homography matrix is shown in Figure 5, including a checkerboardimage reading unit 600, anintersection selection unit 601, aback projection unit 602, a homographymatrix estimation unit 603, a homographymatrix optimization unit 604, and anintersection mapping unit 605 and an intersection image coordinateacquisition unit 606.

这些单元之间的连接关系是：棋盘格图像读取单元600的输出端耦接交叉点选取单元601，交叉点选取单元601的输出端耦接反投影单元602，反投影单元602的输出端耦接单应矩阵估计单元603，单应矩阵估计单元603的输出端耦接单应矩阵优化单元604，单应矩阵优化单元604的输出端耦接交叉点映射单元605，交叉点映射单元605的输出端耦接交叉点图像坐标获取单元606。The connection relationship between these units is: the output end of the checkerboardimage reading unit 600 is coupled to theintersection selection unit 601, the output end of theintersection selection unit 601 is coupled to theback projection unit 602, and the output end of theback projection unit 602 is coupled to Connect the homographymatrix estimation unit 603, the output terminal of the homographymatrix estimation unit 603 is coupled to the homographymatrix optimization unit 604, the output terminal of the homographymatrix optimization unit 604 is coupled to theintersection mapping unit 605, the output of theintersection mapping unit 605 The end is coupled to the intersection point image coordinateacquisition unit 606 .

棋盘格图像读取单元600读取棋盘格图像。The checkerboardimage reading unit 600 reads a checkerboard image.

交叉点选取单元601在每张棋盘格图像上依次选取棋盘格顶点处的4个交叉点。The intersectionpoint selection unit 601 sequentially selects four intersection points at vertices of the checkerboard on each checkerboard image.

反投影单元602利用初始化内参，将交叉点

反投影到单位球上得到单位向量

过程如下：The back-projection unit 602 uses the initialization internal parameters to convert the intersection point

Back-project onto the unit sphere to get the unit vector

The process is as follows:

$(\begin{matrix} {x x}_{j j}^{i i} \\ {y the y}_{j j}^{i i} \end{matrix}) = = (\begin{matrix} \frac{11}{{m m}_{u u}} & 00 \\ 00 & \frac{11}{{m m}_{v v}} \end{matrix}) (\begin{matrix} {u u}_{j j}^{i i} - - {u u}_{00} \\ {v v}_{j j}^{i i} - - {v v}_{00} \end{matrix})$

${r r}_{j j}^{i i} = = \sqrt{{(({x x}_{j j}^{i i}))}^{22} + + {(({y the y}_{j j}^{i i}))}^{22}}$

${k k}_{11} {θ θ}_{j j}^{i i} + + {k k}_{22} {(({θ θ}_{j j}^{i i}))}^{22} + + {k k}_{33} {(({θ θ}_{j j}^{i i}))}^{33} + + {k k}_{44} {(({θ θ}_{j j}^{i i}))}^{44} + + {k k}_{55} {(({θ θ}_{j j}^{i i}))}^{55} - - {r r}_{j j}^{i i} = = 00$

其中j为第j幅图像，i为第i个棋盘格交叉点。where j is the j-th image, and i is the i-th checkerboard intersection.

单应矩阵估计单元603估计单应矩阵。单位向量

可以表示为：其与标定板上的空间点xⁱ之间存在单应变换H_j，通过线性算法估计该单应变换H_j，得到空间点xⁱ在单应变换H_j下的对应点：The homographymatrix estimating unit 603 estimates a homography matrix. unit vector

It can be expressed as: There is a homography transformation H_j between it and the spatial point xⁱ on the calibration board, and the homography transformation H_j is estimated by a linear algorithm to obtain the corresponding point of the spatial point xⁱ under the homography transformation H_j :

${\overset{^^}{x x}}_{j j}^{i i} = = {H h}_{j j} {x x}_{p p}^{i i} / / | | | | {H h}_{j j} {x x}_{p p}^{i i} | | | | . .$

单应矩阵优化单元604通过LM迭代最小化误差函数以优化单应矩阵H_j，其中

是向量

和

之间的夹角。Thehomography optimization unit 604 minimizes the error function by LM iteration To optimize the homography matrix H_j , where

is a vector

and

angle between.

交叉点映射单元605将标定板上的所有交叉点通过优化后的单应矩阵H_j映射到单位球上得到对应点：

其中x_pⁱ是第j幅棋盘格图像上第i个交叉点的空间坐标。Theintersection mapping unit 605 maps all intersections on the calibration plate to the unit sphere through the optimized homography matrix_Hj to obtain corresponding points:

where x_pⁱ is the spatial coordinate of the i-th intersection point on the j-th checkerboard image.

交叉点图像坐标获取单元606将单位向量变换到图像上：在投影点的邻域寻找交叉点的图像坐标

Intersection image coordinateacquisition unit 606 transforms the unit vector onto the image: at the projected point Image coordinates for finding the intersection point of the neighborhood of

重复反投影单元602、单应矩阵估计单元603以及单应矩阵优化单元604的运作就能得到基于所有交叉点估计的更加准确的单应矩阵H_j。By repeating the operations of the back-projection unit 602 , the homographymatrix estimation unit 603 and the homographymatrix optimization unit 604 , a more accurate homography matrix H_j estimated based on all intersection points can be obtained.

初始化外参模块70初始化外参。The initializationexternal parameter module 70 initializes the external parameter.

外参可由单应矩阵H_j初始化如下：External reference It can be initialized by the homography matrix H_j as follows:

其中，

R_j为旋转参数，T_j为位移参数，

为第j个单应矩阵H_j的第i个列向量。in,

R_j is the rotation parameter, T_j is the displacement parameter,

is the i-th column vector of the j-th homography matrix H_j .

外参是指鱼眼相机与外部环境之间的参数，本实施例中是指鱼眼相机与标定板之间的参数R_j和T_j。The external parameters refer to the parameters between the fisheye camera and the external environment, and in this embodiment refer to the parameters R_j and T_j between the fisheye camera and the calibration board.

迭代优化模块80用于LM迭代最小化重投影误差

得到优化后的内参与外参。其中

为图像点

之间的像素距离，M为每幅棋盘格图像上的交叉点数量，N为棋盘格图像数量。Iterative optimization module 80 is used for LM iterative minimization of reprojection error

Get the optimized internal and external parameters. in

is the image point

以下是标定装置1026a的其中另一种示例的实施情况。The following is another exemplary implementation of thecalibration device 1026a.

这一种示例的标定装置的鱼眼成像关系建立模块40用于建立半单位球面模型，并在单位球面模型上建立鱼眼成像关系。The fisheye imagingrelationship establishment module 40 of this exemplary calibration device is used to establish a semi-unit spherical model, and establish a fisheye imaging relationship on the unit spherical model.

其中θ是入射光线与单位球Z轴正方向的夹角，

x_d为像素点的位置向量，即(x_d，y_d)，ur为径向单位向量，为切向单位向量。x_d is the position vector of the pixel point, namely (x_d , y_d ), ur is the radial unit vector, is a tangential unit vector.

其中已知信息为空间点x的入射角度

然后根据椭圆拟合求得u₀，v₀，a，b，则

本实施例的计算单应矩阵的原理如图6所示，包括棋盘格图像读取单元610、交叉点选取单元611、反投影单元612、单应矩阵估计单元613、单应矩阵优化单元614。The principle of calculating the homography matrix in this embodiment is shown in FIG. 6 , including a checkerboardimage reading unit 610 , an intersectionpoint selection unit 611 , a back-projection unit 612 , a homographymatrix estimation unit 613 , and a homographymatrix optimization unit 614 .

这些单元之间的连接关系是：棋盘格图像读取单元610的输出端耦接交叉点选取单元611，交叉点选取单元611的输出端耦接反投影单元612，反投影单元612的输出端耦接单应矩阵估计单元613，单应矩阵估计单元613的输出端耦接单应矩阵优化单元614。The connection relationship between these units is: the output end of the checkerboardimage reading unit 610 is coupled to theintersection selection unit 611, the output end of theintersection selection unit 611 is coupled to theback projection unit 612, and the output end of theback projection unit 612 is coupled to Thehomography estimation unit 613 is connected to the output end of thehomography estimation unit 613 coupled to thehomography optimization unit 614 .

棋盘格图像读取单元610读取棋盘格图像。The checkerboardimage reading unit 610 reads a checkerboard image.

交叉点选取单元611在每张棋盘格图像上依次选取所有的棋盘格交叉点。The intersectionpoint selection unit 611 sequentially selects all checkerboard intersection points on each checkerboard image.

反投影单元612利用初始化内参，将交叉点

反投影到单位球上得到单位向量

过程如下：The back-projection unit 612 uses the initialization internal parameters to convert the intersection point

Back-project onto the unit sphere to get the unit vector

The process is as follows:

单应矩阵估计单元613估计单应矩阵。单位向量

可以表示为：其与标定板上的空间点xⁱ之间存在单应变换H_j，通过线性算法估计该单应变换H_j，得到空间点xⁱ在单应变换H_j下的对应点：

其中x_pⁱ是第j幅棋盘格图像上第i个交叉点的空间坐标。The homographymatrix estimating unit 613 estimates a homography matrix. unit vector

单应矩阵优化单元614通过LM迭代最小化误差函数以优化单应矩阵H_j，其中

是向量

和

之间的夹角。Thehomography optimization unit 614 minimizes the error function by LM iteration To optimize the homography matrix H_j , where

is a vector

and

angle between.

外参

可由单应矩阵H_j初始化如下：External reference

It can be initialized by the homography matrix H_j as follows:

其中，R_j为旋转参数，T_j为位移参数，

为第j个单应矩阵H_j的第i个列向量。in, R_j is the rotation parameter, T_j is the displacement parameter,

is the i-th column vector of the j-th homography matrix H_j .

迭代优化模块80用于LM迭代最小化重投影误差得到优化后的内参与外参。其中

为图像点

之间的像素距离，其中M为每幅棋盘格图像上的交叉点数量，N为棋盘格图像数量。Iterative optimization module 80 is used for LM iterative minimization of reprojection error Get the optimized internal and external parameters. in

is the image point

The pixel distance between, where M is the number of intersections on each checkerboard image, and N is the number of checkerboard images.

数据采集装置10a中的鱼眼立体视觉模块102a采集连续的鱼眼图像，并将这些影像数据存储到数据存储装置20a上，利用鱼眼立体视觉对的参数，数据处理装置30a对所采集的数据进行处理，实现对所采集的鱼眼影像的测量，比如道路宽度、转弯半径，建筑的宽、高，广告牌的面积等。The fish-eye stereo vision module 102a in thedata acquisition device 10a collects continuous fish-eye images, and stores these image data on thedata storage device 20a. Using the parameters of the fish-eye stereo vision pair, thedata processing device 30a processes the collected data Perform processing to realize the measurement of the collected fisheye images, such as road width, turning radius, building width and height, billboard area, etc.

值得注意的是，其中的CCD相机1024a也可以是其他形式的例如CMOS相机。It should be noted that theCCD camera 1024a can also be of other forms such as a CMOS camera.

基于鱼眼镜头的车载移动摄影测量系统的第二实施例The second embodiment of the vehicle-mounted mobile photogrammetry system based on fisheye lens

图2示出了本发明的基于鱼眼镜头的车载移动摄影测量系统的第二实施例。请参见图2。本实施例的基于鱼眼镜头的车载移动摄影测量系统包括数据采集装置10b，数据存储装置20b，数据处理装置30b。Fig. 2 shows the second embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention. See Figure 2. The vehicle-mounted mobile photogrammetry system based on a fisheye lens in this embodiment includes adata acquisition device 10b, a data storage device 20b, and a data processing device 30b.

这三个装置之间的连接关系是：数据采集装置10b的输出端连接数据存储装置20b，数据存储装置20b的输出端连接数据处理装置30b。The connection relationship between these three devices is: the output end of thedata collection device 10b is connected to the data storage device 20b, and the output end of the data storage device 20b is connected to the data processing device 30b.

数据采集装置10b包括鱼眼相机立体视觉模块102b和车辆位姿获取模块104b。鱼眼相机立体视觉模块102b，由至少一个立体视觉对组成，即两台鱼眼相机。鱼眼立体相机立体视觉模块102b包括鱼眼镜头1022b，CCD相机1024b和标定装置1026b。鱼眼镜头1022b连接到CCD相机1024b上，组成可以生成超广角视野的鱼眼影像。标定装置1026b用于获取鱼眼相机的成像参数以及鱼眼相机立体视觉对参数。标定装置1026b的具体实施方式(有两种)均在第一实施例中详细描述，在此不再赘述。Thedata acquisition device 10b includes a fisheye camera stereo vision module 102b and a vehicle pose acquisition module 104b. The fisheye camera stereo vision module 102b is composed of at least one stereo vision pair, that is, two fisheye cameras. The fisheye stereo camera stereo vision module 102b includes a fisheye lens 1022b, a CCD camera 1024b and a calibration device 1026b. The fisheye lens 1022b is connected to the CCD camera 1024b to form a fisheye image that can generate a super wide-angle field of view. The calibration device 1026b is used to acquire the imaging parameters of the fisheye camera and the stereo vision pair parameters of the fisheye camera. The specific implementation manners (there are two types) of the calibration device 1026b are described in detail in the first embodiment, and will not be repeated here.

车辆位姿获取模块104b用于获取车辆行驶过程中的经纬度、姿态信息以及车辆行驶的准确距离。车辆位姿获取模块104b可以包括卫星定位系统，如GPS天线、GPS接收器、GPS基站，车辆姿态获取装置，如陀螺仪、加速度计等，车辆运行距离获取装置，如汽车总线或车轮编码器等。The vehicle pose acquisition module 104b is used to acquire the latitude and longitude, attitude information and the accurate distance traveled by the vehicle during the driving process. The vehicle pose acquisition module 104b may include a satellite positioning system, such as a GPS antenna, a GPS receiver, a GPS base station, a vehicle attitude acquisition device, such as a gyroscope, an accelerometer, etc., and a vehicle running distance acquisition device, such as an automobile bus or a wheel encoder, etc. .

数据采集装置10b中的鱼眼立体视觉模块102b采集连续的鱼眼图像，车辆位姿获取模块104b获取车辆行驶过程中的经纬度、姿态及车辆运行的距离信息，这些影像数据和位置、姿态、距离信息存储到数据存储装置20b上。利用鱼眼立体视觉对的参数，数据处理装置30b对所采集的数据进行处理，实现对所采集的鱼眼影像的测量，比如道路宽度，建筑的宽、高等。结合车辆位姿获取模块104b所采集到的信息，数据处理装置30b可以计算出鱼眼影像中物体的精确大地坐标和方向，即影像中感兴趣物体的绝对位置信息，从而对感兴趣物体进行标注，以实现城市管理、普查和规划的目的。The fish-eye stereo vision module 102b in thedata acquisition device 10b collects continuous fish-eye images, and the vehicle pose acquisition module 104b acquires the latitude and longitude, attitude and distance information of the vehicle during driving. These image data and position, attitude, distance The information is stored on the data storage device 20b. Using the parameters of the fish-eye stereo vision pair, the data processing device 30b processes the collected data to realize the measurement of the collected fish-eye image, such as the width of a road, the width and height of a building, and the like. Combined with the information collected by the vehicle pose acquisition module 104b, the data processing device 30b can calculate the precise geodetic coordinates and direction of the object in the fisheye image, that is, the absolute position information of the object of interest in the image, so as to mark the object of interest , in order to achieve the purpose of urban management, census and planning.

基于鱼眼镜头的车载移动摄影测量系统的第三实施例The third embodiment of the vehicle-mounted mobile photogrammetry system based on fisheye lens

图3示出了本发明的基于鱼眼镜头的车载移动摄影测量系统的第三实施例。请参见图3。本实施例的基于鱼眼镜头的车载移动摄影测量系统包括数据采集装置10c、数据存储装置20c、数据处理装置30c。Fig. 3 shows the third embodiment of the vehicle-mounted mobile photogrammetry system based on the fisheye lens of the present invention. See Figure 3. The vehicle-mounted mobile photogrammetry system based on a fisheye lens in this embodiment includes adata collection device 10c, adata storage device 20c, and adata processing device 30c.

这三个装置之间的连接关系是：数据采集装置10c的输出端连接数据存储装置20c，数据存储装置20c的输出端连接数据处理装置30c。The connection relationship between these three devices is: the output end of thedata collection device 10c is connected to thedata storage device 20c, and the output end of thedata storage device 20c is connected to thedata processing device 30c.

数据采集装置10c包括鱼眼相机立体视觉模块102c、车辆位姿获取模块104c和全景采集模块106c。Thedata acquisition device 10c includes a fisheye camerastereo vision module 102c, a vehiclepose acquisition module 104c and apanorama acquisition module 106c.

鱼眼相机立体视觉模块102c由至少一个立体视觉对组成，即两台鱼眼相机。鱼眼立体相机立体视觉模块102c包括鱼眼镜头1022c，CCD相机1024c和标定装置1026c。鱼眼镜头1022c连接到CCD相机1024c上，组成可以生成超广角视野的鱼眼影像。标定装置1026c用于获取鱼眼相机的成像参数以及鱼眼相机立体视觉对参数。标定装置1026c的具体实施方式(有两种)均在第一实施例中详细描述，在此不再赘述。The fisheye camerastereo vision module 102c is composed of at least one stereo vision pair, that is, two fisheye cameras. The fisheye stereo camerastereo vision module 102c includes afisheye lens 1022c, aCCD camera 1024c and acalibration device 1026c. Thefisheye lens 1022c is connected to theCCD camera 1024c to form a fisheye image that can generate a super wide-angle field of view. Thecalibration device 1026c is used to acquire the imaging parameters of the fisheye camera and the stereo vision pair parameters of the fisheye camera. The specific implementation manners of thecalibration device 1026c (there are two types) are described in detail in the first embodiment, and will not be repeated here.

车辆位姿获取模块104c用于获取车辆行驶过程中的经纬度、姿态信息，以及车辆行驶的准确距离。车辆位姿获取模块104c可以包括卫星定位系统，如GPS天线、GPS接收器、GPS基站，车辆姿态获取装置，如陀螺仪、加速度计等，车辆运行距离获取装置，如汽车总线或车轮编码器等。The vehicle poseacquisition module 104c is used to acquire the latitude and longitude, attitude information, and the accurate distance traveled by the vehicle during driving. The vehicle poseacquisition module 104c may include a satellite positioning system, such as a GPS antenna, a GPS receiver, a GPS base station, a vehicle attitude acquisition device, such as a gyroscope, an accelerometer, etc., and a vehicle running distance acquisition device, such as an automobile bus or a wheel encoder, etc. .

全景采集模块106c用于采集高质量连续的全景影像数据。Thepanoramic collection module 106c is used for collecting high-quality continuous panoramic image data.

数据采集装置10c中的鱼眼立体视觉模块102c采集连续的鱼眼图像，车辆位姿获取模块104c获取车辆行驶过程中的经纬度、姿态及车辆运行的距离信息，全景采集模块106c采集连续的道路全景影像，这些影像数据和位置、姿态、距离信息存储到数据存储装置20c上。利用鱼眼立体视觉对的参数，数据处理装置30c对所采集的数据进行处理，实现对所采集的鱼眼影像的测量，比如道路宽度，建筑的宽、高等。结合车辆位姿获取模块104c所采集到的信息，数据处理装置30c可以计算出鱼眼影像的精确大地坐标和方向，即所测量物体的绝对位置信息，同时可以得到具有GIS信息的全景影像数据。The fisheyestereo vision module 102c in thedata acquisition device 10c collects continuous fisheye images, the vehicle poseacquisition module 104c acquires the latitude and longitude, attitude and distance information of the vehicle during driving, and thepanorama acquisition module 106c acquires continuous road panoramas Images, these image data and position, posture, and distance information are stored on thedata storage device 20c. Using the parameters of the fish-eye stereo vision pair, thedata processing device 30c processes the collected data to realize the measurement of the collected fish-eye image, such as the width of a road, the width and height of a building, and the like. Combined with the information collected by the vehicle poseacquisition module 104c, thedata processing device 30c can calculate the precise geodetic coordinates and direction of the fisheye image, that is, the absolute position information of the measured object, and at the same time can obtain panoramic image data with GIS information.

上述实施例是提供给本领域普通技术人员来实现和使用本发明的，本领域普通技术人员可在不脱离本发明的发明思想的情况下，对上述实施例做出种种修改或变化，因而本发明的发明范围并不被上述实施例所限，而应该是符合权利要求书所提到的创新性特征的最大范围。The above-mentioned embodiments are provided for those of ordinary skill in the art to implement and use the present invention. Those of ordinary skill in the art can make various modifications or changes to the above-mentioned embodiments without departing from the inventive idea of the present invention. Therefore, the present invention The inventive scope of the invention is not limited by the above examples, but should be the widest scope consistent with the innovative features mentioned in the claims.

Claims

Translated fromChinese

1.一种基于鱼眼镜头的车载移动摄影测量系统，包括数据采集装置、数据存储装置和数据处理装置，其中：1. A vehicle-mounted mobile photogrammetry system based on a fisheye lens, comprising a data acquisition device, a data storage device and a data processing device, wherein:

数据处理装置，耦接数据存储装置，用于处理所采集到的数据，实现对鱼眼影像的测量；A data processing device, coupled to the data storage device, is used to process the collected data and realize the measurement of the fisheye image;

其中标定装置包括：The calibration devices include:

2.根据权利要求1所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，数据采集装置还包括：2. the vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 1, is characterized in that, data collection device also comprises:

3.根据权利要求1所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，数据采集装置还包括：3. the vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 1, is characterized in that, data collection device also comprises:

4.根据权利要求1所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，鱼眼相机立体视觉模块至少包括两台鱼眼相机。4. The vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 1, wherein the fisheye camera stereo vision module comprises at least two fisheye cameras.

5.根据权利要求1所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，鱼眼成像关系建立模块的处理如下：5. the vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 1, is characterized in that, the processing of fisheye imaging relation establishment module is as follows:

记空间中某一点x在鱼眼图像上的成像点为(u,v)，空间点x指向单位球球心的入射角度为

，其中θ是入射光线与单位球Z轴正方向的夹角，是入射光线在单位球XY平面上的投影与单位球X轴正方向的夹角，则由入射光线的入射角度

到鱼眼图像上的成像点(u,v)的鱼眼成像模型由以下方程描述：Note that the imaging point of a certain point x in the space on the fisheye image is (u, v), and the incident angle of the space point x pointing to the center of the unit sphere is

, where θ is the angle between the incident ray and the positive direction of the Z-axis of the unit sphere, is the angle between the projection of the incident ray on the XY plane of the unit sphere and the positive direction of the X-axis of the unit sphere, then the angle of incidence of the incident ray

The fisheye imaging model to the imaging point (u, v) on the fisheye image is described by the following equation:r(θ)=k₁θ+k₂θ²+k₃θ³+k₄θ⁴+k₅θ⁵+...k_nθⁿ （1）r(θ)=k₁ θ+k₂ θ² +k₃ θ³ +k₄ θ⁴ +k₅ θ⁵ +...k_n θⁿ (1)

r表示图像上的某个像素点到图像主点的距离，k₁,...k_n是鱼眼镜头的成像参数；r represents the distance from a certain pixel on the image to the principal point of the image, k₁ ,...k_n are the imaging parameters of the fisheye lens;

Δ_r表示鱼眼镜头的径向畸变，l₁,...l_n,i₁,...i₄为径向畸变参数；Δ_r represents the radial distortion of the fisheye lens, l₁ ,...l_n ,i₁ ,...i₄ are the radial distortion parameters;

Δ_t表示鱼眼镜头的切向畸变，m₁,...m_n,j₁,...j₄为切向畸变参数；_Δt represents the tangential distortion of the fisheye lens, m₁ ,...m_n ,j₁ ,...j₄ are tangential distortion parameters;

x_d为像素点的位置向量，即(x_d,y_d)，u_r为径向单位向量，

is the tangential unit vector;

(\begin{matrix} u u \\ v v \end{matrix}) = = (\begin{matrix} {m m}_{u u} & 00 \\ 00 & {m m}_{v v} \end{matrix}) (\begin{matrix} {x x}_{d d} \\ {y the y}_{d d} \end{matrix}) + + (\begin{matrix} {u u}_{00} \\ {v v}_{00} \end{matrix}) - - - - - - ((55))

(u₀,v₀)为图像的主点坐标，(m_u,m_v)分别为CCD水平与垂直方向上单位距离上的像素数，(k₁,k₂,k₃,k₄,k₅,l₁,l₂,l₃,i₁,i₂,i₃,i₄,m₁,m₂,m₃,j₁,j₂,j₃,j₄,m_u,m_v,u₀,v₀)为待标定的鱼眼镜头的参数。(u₀ , v₀ ) are the principal point coordinates of the image, (m_u , m_v ) are the number of pixels per unit distance in the horizontal and vertical directions of the CCD, respectively, (k₁ ,k₂ ,k₃ ,k₄ ,k₅ ,l₁ ,l₂ ,l₃ ,i 1 ,i₂ ,i₃ ,_i₄ ,m₁ ,m₂ ,m 3 ,j₁ ,j₂ ,j₃ ,_j₄ ,m_u ,m_v , u₀ , v₀ ) are the parameters of the fisheye lens to be calibrated.

6.根据权利要求5所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，单应矩阵计算模块包括：6. the vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 5, is characterized in that, homography matrix calculation module comprises:

反投影到单位球上得到单位向量其中j为第j幅图像，i为第i个棋盘格交叉点；The back-projection unit is coupled to the intersection point selection unit, and the intersection point is set to

Back-project onto the unit sphere to get the unit vector Where j is the jth image, i is the ith checkerboard intersection point;

表示为

其中x_pⁱ是第j幅棋盘格图像上第i个交叉点的空间坐标；The homography matrix estimation unit, coupled with the back-projection unit, estimates the homography matrix H_j , and converts the unit vector

Expressed as

以优化单应矩阵H_j，其中

是向量

和

To optimize the homography matrix H_j , where

is a vector

and

the angle between

(\begin{matrix} {\hat{u}}_{j}^{i} \\ {\hat{v}}_{j}^{i} \end{matrix}) = (\begin{matrix} m_{u} & 0 \\ 0 & m_{v} \end{matrix}) (\begin{matrix} {\hat{x}}_{j}^{i} \\ {\hat{y}}_{j}^{i} \end{matrix}) + (\begin{matrix} u_{0} \\ v_{0} \end{matrix}),

在投影点

的邻域寻找交叉点的图像坐标

(m_u,m_v)分别为CCD水平与垂直方向上单位距离上的像素数。The intersection image coordinate acquisition unit is coupled with the intersection mapping unit to transform the unit vector onto the image:

(\begin{matrix} {\hat{u}}_{j}^{i} \\ {\hat{v}}_{j}^{i} \end{matrix}) = (\begin{matrix} m_{u} & 0 \\ 0 & m_{v} \end{matrix}) (\begin{matrix} {\hat{x}}_{j}^{i} \\ {\hat{the y}}_{j}^{i} \end{matrix}) + (\begin{matrix} u_{0} \\ v_{0} \end{matrix}),

at the projected point

Image coordinates for finding the intersection point of the neighborhood of

(m_u , m_v ) are the number of pixels per unit distance in the horizontal and vertical directions of the CCD, respectively.

7.根据权利要求5所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，单应矩阵计算模块包括：7. The vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 5, wherein the homography matrix calculation module comprises:

反投影到单位球上得到单位向量

Back-project onto the unit sphere to get the unit vector

Where j is the jth image, i is the ith checkerboard intersection point;

表示为

Expressed as

There is a homography transformation H_j between the unit vector and the spatial point xⁱ on the calibration board, the homography transformation H_j is estimated by a linear algorithm, and the corresponding point of the spatial point xⁱ on the calibration board under the homography transformation H_j is obtained:

以优化单应矩阵H_j，其中是向量

和

To optimize the homography matrix H_j , where is a vector

and

angle between.

8.根据权利要求6或7所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，初始化外参模块的处理如下：8. the vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 6 or 7, is characterized in that, the processing of initializing the external reference module is as follows:

外参

由单应矩阵H_j初始化如下：External reference

Initialized by the homography matrix_Hj as follows:

{r r}_{j j}^{11} = = {λ λ}_{j j} {h h}_{j j}^{11},, {r r}_{j j}^{22} = = {λ λ}_{j j} {h h}_{j j}^{22},, {r r}_{j j}^{33} = = {r r}_{j j}^{11} \times \times {r r}_{j j}^{22},, {t t}_{j j} = = {λ λ}_{j j} {h h}_{j j}^{33}

其中，

R_j为旋转参数，T_j为位移参数，

为第j个单应矩阵H_j的第i个列向量。in,

R_j is the rotation parameter, T_j is the displacement parameter,

is the i-th column vector of the j-th homography matrix H_j .

9.根据权利要求8所述的基于鱼眼镜头的车载移动摄影测量系统，其特征在于，迭代优化模块的处理如下：9. the vehicle-mounted mobile photogrammetry system based on fisheye lens according to claim 8, is characterized in that, the processing of iterative optimization module is as follows:

LM迭代最小化重投影误差

得到优化后的内参和外参，其中

为图像点

Get the optimized internal and external references, where

is the image point