CN113888652A - Automatic calibration technology of internal and external parameters of 4D millimeter wave radar and thermal camera - Google Patents

Abstract

The invention belongs to the technical field of sensor calibration, and discloses an automatic internal and external parameter calibration technology for a 4D millimeter wave radar and a thermal camera. The invention has the beneficial effects that: the calibration object is easy to manufacture, low in cost and low in use complexity, only one calibration object is needed, common feature points in the millimeter wave radar and the thermal camera can be extracted simultaneously, external reference calibration is achieved, and the method is simple, robust and high in precision.

Description

Internal and external parameter automatic calibration technology for 4D millimeter wave radar and thermal sensor camera

[ technical field ] A method for producing a semiconductor device

The invention relates to the technical field of sensor calibration, in particular to an internal and external parameter automatic calibration technology of a 4D millimeter wave radar and a thermal sensor camera.

[ background of the invention ]

With the advent of the intelligent era, particularly the development of mobile robots, the perception capability of the robots in severe environments is gaining more and more attention. The detection robustness of the millimeter wave radar and the thermal camera in the environment of rain, snow, smoke, fog and the like is mainly benefited, but the calibration of the millimeter wave radar and the thermal camera still has larger challenges. After 4D millimeter wave radar comes into the market, a related algorithm is still to be developed, and although a long distance can be detected, three-dimensional point cloud is very sparse and lacks texture information. The thermal camera is insensitive to illumination, can work robustly under different illumination conditions, but has lower resolution and lacks three-dimensional information for acquiring image data. Although both the sensors have robustness under special environments, common characteristic points are difficult to find between the sensors.

In order to fully utilize complementary information of the millimeter wave radar and the thermal camera, accurate external parameter calibration is indispensable. However, the current practice is to realize calibration by introducing other sensors, and manually selecting feature points is required. The calibration process is complicated, and various different calibration plates are required. Due to the use of various calibration plates, calibration errors are large along with the transmission of errors. In addition, the existing calibration method is not user-friendly, and requires the user to have relevant professional knowledge, which limits the use scene of the sensor suite.

Therefore, it is necessary to provide an internal and external parameter automatic calibration technology for a 4D millimeter wave radar and a thermal camera, which can detect common feature points under the millimeter wave radar and the thermal camera, improve the problems of tedious calibration process, large calibration error and user unfriendliness of the millimeter wave radar and the thermal camera, further expand the applicable scenarios of the sensor suite, and enhance the reliability of detection under the existing scenarios

[ summary of the invention ]

The invention discloses an internal and external parameter automatic calibration technology of a 4D millimeter wave radar and a thermal camera, which can effectively solve the technical problems related to the background technology.

In order to achieve the purpose, the technical scheme of the invention is as follows:

the internal and external parameter automatic calibration technology of the 4D millimeter wave radar and the thermal sensor camera comprises the following steps:

s1, preparing a calibration object, wherein the calibration object is a sphere with a notch, the notch is 1/8 sphere in size, the notch comprises 3 induction surfaces which are vertical pairwise, and the 3 induction surfaces form an angular reflector;

s2, heating or cooling the calibration object, and enabling the corner reflector to face a sensor suite, wherein the sensor suite comprises a 4D millimeter wave radar and a thermal camera;

s3, collecting data, moving the sensor suite to different positions and heights, and storing corresponding three-dimensional point cloud and thermal sensing images;

s4, calculating internal parameters of the thermal camera, calculating initial internal parameters through a homography matrix according to the detected circle center, and calculating accurate internal parameters through a nonlinear optimization method;

s5, detecting the center point of the calibration object under the radar point cloud, determining the space interval of the center point of the calibration object in the radar point cloud by utilizing point cloud segmentation, and obtaining the 3D coordinate of the center point based on the characteristic that the echo intensity of the center point of the calibration object is highest;

s6, detecting the center point of the calibration object in the thermal image, firstly carrying out binarization processing on the thermal image, then extracting the outline of the calibration object, calculating the minimum circumscribed rectangle of the outline, and extracting the 2D coordinate of the center point;

s7, calculating external parameters of the millimeter wave radar and the thermal camera, solving the external parameters through nonlinear optimization according to the 3D coordinates of the center point of the calibration object detected by the radar point cloud and the 2D coordinates of the center point of the calibration object detected in the thermal image, and minimizing the feature matching error.

As a preferred improvement of the present invention: and tinfoil is arranged on the sensing surface.

As a preferred improvement of the present invention: the sphere radius of the calibration object is 10 cm.

The invention has the following beneficial effects:

1. a new calibration object is designed, and a convenient, robust and accurate automatic external reference calibration technology based on the calibration object is provided, so that external reference calibration of a millimeter wave radar and a thermal camera is realized;

2. the calibration object is easy to manufacture, low in cost and low in use complexity, and only one calibration object is needed, common feature points in the millimeter wave radar and the thermal camera can be simultaneously extracted, so that external reference calibration is realized;

3. the precision is high, and calibration parameters are continuously optimized through calibration at different distances;

4. the method is user-friendly, does not need excessive professional knowledge, and a common user only needs to hold the sensor suite to collect calibration object data, so that the algorithm can automatically select reliable data and complete calibration.

[ description of the drawings ]

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without inventive efforts, wherein:

FIG. 1 is a flow chart of the internal and external parameter automatic calibration technique of the 4D millimeter wave radar and thermal camera according to the present invention;

FIG. 2 is a schematic diagram of the structure of the calibration object of the present invention.

In the figure: 1-calibration object, 11-induction surface.

[ detailed description ] embodiments

The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the following embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that all the directional indicators (such as up, down, left, right, front, and rear … …) in the embodiment of the present invention are only used to explain the relative position relationship between the components, the movement situation, etc. in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indicator is changed accordingly.

In addition, the descriptions related to "first", "second", etc. in the present invention are only for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.

In the present invention, unless otherwise expressly stated or limited, the terms "connected," "secured," and the like are to be construed broadly, and for example, "secured" may be a fixed connection, a removable connection, or an integral part; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.

In addition, the technical solutions in the embodiments of the present invention may be combined with each other, but it must be based on the realization of those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination of technical solutions should not be considered to exist, and is not within the protection scope of the present invention.

Referring to fig. 1, the present invention provides an internal and external parameter automatic calibration technique for a 4D millimeter wave radar and a thermal sensor camera, which includes the following steps:

s1, preparing acalibration object 1, wherein thecalibration object 1 is a sphere with a notch, the notch is 1/8 sphere in size, the notch comprises 3induction surfaces 11 which are vertical to each other in pairs, and the 3induction surfaces 11 form a corner reflector. Specifically, a new calibration object is provided, so that two sensors (4D millimeter wave radar and thermal camera) can detect a common characteristic, and the center of sphere is taken as the common characteristic for calibration. Since the reflection intensity of the trihedral (corner reflector) for the millimeter-wave radar is high, it is easy to find this calibration object, i.e., the center of sphere, from the point cloud of the millimeter-wave radar. The advantage of combining a corner reflector with a ball is that the ball (center of the ball) is view-invariant to the thermal camera; meanwhile, the millimeter wave radar has high response to the corner reflector; therefore, the two sensors are combined to prepare the novel spherical-corner reflector, so that the common characteristic extraction which is a great difficulty in calibrating the two heterogeneous sensors is solved. The two sensors can detect the center of the sphere, and the center of the sphere is a point characteristic, so that the calibration precision is guaranteed. In addition, the same feature is directly extracted by detecting the center of the sphere, and the calibration precision is higher than that of the existing indirect common feature extraction.

S2, heating or cooling thecalibration object 1, and enabling the corner reflector to face to a sensor suite, wherein the sensor suite comprises a 4D millimeter wave radar and a thermal camera. Specifically, thecalibration object 1 is subjected to temperature processing to facilitate sensing of a thermal camera, and then the position of thecalibration object 1 is adjusted to enable the corner reflector to face the sensor suite.

And S3, acquiring data, moving the sensor suite to different positions and heights, and storing corresponding three-dimensional point cloud and thermal sensing images.

S4, calculating internal parameters of the thermal camera, calculating initial internal parameters through a homography matrix according to the detected circle center, and calculating accurate internal parameters through a nonlinear optimization method, wherein a 4 multiplied by 11 round hole calibration plate is adopted.

S5, detecting the center point of thecalibration object 1 under the radar point cloud, determining the space interval of the center point of thecalibration object 1 in the radar point cloud by utilizing point cloud segmentation, and obtaining the 3D coordinate of the center point based on the characteristic that the echo intensity of the center point of thecalibration object 1 is highest.

S6, detecting the central point of thecalibration object 1 in the thermal image, firstly carrying out binarization processing on the thermal image, then extracting the outer contour of thecalibration object 1, calculating the minimum circumscribed rectangle of the outer contour, and extracting the 2D coordinate of the central point.

S7, calculating external parameters of the millimeter wave radar and the thermal camera, solving the external parameters through nonlinear optimization according to the 3D coordinates of the center point of thecalibration object 1 detected by the radar point cloud and the 2D coordinates of the center point of thecalibration object 1 detected in the thermal image, and minimizing the feature matching error.

In this embodiment, thesensing surface 11 is provided with tinfoil, the millimeter wave radar has high reflectivity to surface metals and stronger response, and the sphere radius of thecalibration object 1 is 10 cm.

The working principle is as follows: thecalibration object 1 is placed with the corner reflector facing the 4D millimeter wave radar and thermal camera. And moving the 4D millimeter wave radar and the thermal camera to acquire data. And calculating the result, and completing calibration.

While embodiments of the invention have been disclosed above, it is not limited to the applications set forth in the specification and the embodiments, which are fully applicable to various fields of endeavor for which the invention pertains, and further modifications may readily be made by those skilled in the art, it being understood that the invention is not limited to the details shown and described herein without departing from the general concept defined by the appended claims and their equivalents.

Claims (3)

Translated fromChinese

1.4D毫米波雷达与热感相机的内外参自动标定技术，其特征在于，包括如下步骤：1.4D millimeter wave radar and thermal camera automatic calibration technology for internal and external parameters, characterized in that it includes the following steps:S1、制备标定物(1)，所述标定物(1)为带缺口的球体，缺口为1/8球体大小，缺口处包括3个两两垂直的感应面(11)，3个所述感应面(11)形成角反射器；S1. Prepare a calibration object (1), the calibration object (1) is a sphere with a gap, the gap is 1/8 the size of a sphere, and the gap includes three sensing surfaces (11) that are perpendicular to each other. The surface (11) forms a corner reflector;S2、加热或者降温所述标定物(1)，并将角反射器朝向传感器套件，所述传感器套件包括4D毫米波雷达与热感相机；S2, heating or cooling the calibration object (1), and orienting the corner reflector toward the sensor kit, where the sensor kit includes a 4D millimeter-wave radar and a thermal camera;S3、采集数据，将传感器套件移动到不同的位置及高度，存储对应的三维点云和热感图像；S3. Collect data, move the sensor kit to different positions and heights, and store the corresponding 3D point cloud and thermal image;S4、计算热感相机内参，根据检测到的圆心，通过单应性矩阵计算初始内参，再通过非线性优化方法计算精确的内参；S4. Calculate the internal parameters of the thermal camera, calculate the initial internal parameters through the homography matrix according to the detected center of the circle, and then calculate the accurate internal parameters through the nonlinear optimization method;S5、雷达点云下检测所述标定物(1)中心点，利用点云分割，确定所述标定物(1)中心点在雷达点云中的空间区间，基于所述标定物(1)中心点回波强度最高的特点，得到中心点的3D坐标；S5. Detect the center point of the calibration object (1) under the radar point cloud, and use point cloud segmentation to determine the space interval of the center point of the calibration object (1) in the radar point cloud, based on the center of the calibration object (1) The point with the highest echo intensity can obtain the 3D coordinates of the center point;S6、热感图像中检测所述标定物(1)中心点，先对热感图像进行二值化处理，再提取所述标定物(1)外轮廓，计算外轮廓的最小外接矩形，提取中心点2D坐标；S6. Detect the center point of the calibration object (1) in the thermal image, first perform binarization processing on the thermal image, then extract the outer contour of the calibration object (1), calculate the minimum circumscribed rectangle of the outer contour, and extract the center point 2D coordinates;S7、计算毫米波雷达与热感相机的外参，根据雷达点云检测到的所述标定物(1)中心点的3D坐标和热感图像中检测到的所述标定物(1)中心点的2D坐标，通过非线性优化求解外参，最小化特征匹配误差。S7. Calculate the external parameters of the millimeter-wave radar and the thermal camera, according to the 3D coordinates of the center point of the calibration object (1) detected by the radar point cloud and the center point of the calibration object (1) detected in the thermal image The 2D coordinates of , and the external parameters are solved by nonlinear optimization to minimize the feature matching error.2.根据权利要求1所述的4D毫米波雷达与热感相机的内外参自动标定技术，其特征在于：所述感应面(11)处设有锡纸。2 . The automatic calibration technology for internal and external parameters of a 4D millimeter-wave radar and a thermal camera according to claim 1 , wherein the sensing surface ( 11 ) is provided with tin foil. 3 .3.根据权利要求1所述的4D毫米波雷达与热感相机的内外参自动标定技术，其特征在于：所述标定物(1)的球体半径为10cm。3 . The automatic calibration technology for internal and external parameters of a 4D millimeter wave radar and a thermal camera according to claim 1 , wherein the spherical radius of the calibration object ( 1 ) is 10 cm. 4 .

Images