CN114419302A - Multichannel image fusion system and method based on inertial space alignment - Google Patents
Multichannel image fusion system and method based on inertial space alignmentDownload PDFInfo
- Publication number
- CN114419302A CN114419302ACN202111502699.8ACN202111502699ACN114419302ACN 114419302 ACN114419302 ACN 114419302ACN 202111502699 ACN202111502699 ACN 202111502699ACN 114419302 ACN114419302 ACN 114419302A
- Authority
- CN
- China
- Prior art keywords
- image
- photoelectric imaging
- imaging device
- acquisition
- inertial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Processing (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明涉及光电探测技术领域，尤其涉及一种基于惯性空间对准的多通道图像融合系统及方法。The invention relates to the technical field of photoelectric detection, in particular to a multi-channel image fusion system and method based on inertial space alignment.
背景技术Background technique
在国防与工业领域，红外、可见光等光电探测设备被广泛使用。其在电磁静默环境下对来袭的飞机、无人机、导弹以及制导炸弹等的警戒侦察功能十分重要。尤其对于高速来袭的低空目标，发现距离越远，留给防御武器的机动时间越多，越能提高防御概率。In the field of national defense and industry, photoelectric detection equipment such as infrared and visible light are widely used. It is very important for the alert and reconnaissance function of incoming aircraft, drones, missiles and guided bombs in an electromagnetically silent environment. Especially for low-altitude targets attacking at high speed, the farther the detection distance is, the more maneuvering time is left for the defense weapon, and the more the defense probability can be improved.
在探测器性能一定的情况下对目标探测距离与探测概率的提升有多种方式，一种为硬件路线：通过增大光学系统的口径来提高光线收集能力，但口径越大制造难度越高。且经验表明，目前光学系统的制造成本约与口径的三次方成正比。另一种为软件路线：通过对多个传感器的探测图像进行融合处理，提高探测效果。但目前的专利及研究多专注于不同波段之间的图像融合。Under the condition of certain detector performance, there are many ways to improve the target detection distance and detection probability. One is the hardware route: increasing the aperture of the optical system to improve the light collection capability, but the larger the aperture, the more difficult it is to manufacture. And experience has shown that the manufacturing cost of the current optical system is approximately proportional to the cube of the aperture. The other is the software route: by fusing the detection images of multiple sensors, the detection effect is improved. However, most of the current patents and research focus on image fusion between different bands.
硬件路线中，以授权号“CN 201903705 U”的专利为例，其公开了一种大孔径全反射式光学合成孔径成像系统，其使用多个小孔径的光学系统收集光线后合成到同一探测器上成像。该类方案只使用一个探测器，需要对不同小孔径收集的光波相位进行精确调制，否则会在探测器靶面上产生干涉条纹，对设备的装调精度和稳定性要求极高，且硬件较复杂，制造成本高。In the hardware route, taking the patent with the authorization number "CN 201903705 U" as an example, it discloses a large-aperture total reflection optical synthetic aperture imaging system, which uses a plurality of small-aperture optical systems to collect light and then combine it into the same detector image on. This type of scheme only uses one detector, which requires precise modulation of the phase of the light waves collected by different small apertures, otherwise interference fringes will be generated on the detector target surface, which requires extremely high adjustment accuracy and stability of the equipment, and the hardware is relatively high. Complex and expensive to manufacture.
软件路线中，以公开号“CN 1545064A、CN 101714251 A、CN 101799915 A和CN101853492 A”为例，其公开了多种红外波段之间，红外与可见光波段之间的图像融合方法。可以发现这些方法均专注于研究不同波段图像的融合计算，且图像处理中均采用了各类滤波与尺度匹配方式，需要在原始图像可以提取到目标的时才可用于匹配融合。In the software route, taking publication numbers "CN 1545064A, CN 101714251 A, CN 101799915 A and CN101853492 A" as examples, it discloses image fusion methods between various infrared bands and between infrared and visible light bands. It can be found that these methods focus on the fusion calculation of images of different bands, and various filtering and scale matching methods are used in image processing, which can only be used for matching and fusion when the original image can be extracted to the target.
发明内容SUMMARY OF THE INVENTION
本发明针对现有技术中存在的技术问题，提供一种基于惯性空间对准的多通道图像融合系统及方法，无需使用复杂的光学系统或对光电探测设备进行改造，针对分开安装的多个同类图像探测装置，在不改变硬件的前提下，首先对图像进行像素级畸变标定，实现多个通道间的像素对齐；再通过协同扫描与外触发进行同步成像，利用探测装置自身携带的惯性器件在惯性空间下对准图像；后台同步获取多个探测器图像后进行图像融合，提升信噪比；可有效解决多个设备之间动态基准不一致，时间基准不一致的问题；最终实现弱小目标的信息增强，提升探测距离与探测概率，用于光电警戒侦察设备或远距离目标跟踪设备的探测能力增强。尤其适用于在动平台上分散安装的多个同类型光电探测设备的协同使用。Aiming at the technical problems existing in the prior art, the present invention provides a multi-channel image fusion system and method based on inertial space alignment, which does not require the use of complex optical systems or modification of photoelectric detection equipment. The image detection device, on the premise of not changing the hardware, firstly performs pixel-level distortion calibration on the image to achieve pixel alignment among multiple channels; Align images in inertial space; synchronously acquire multiple detector images in the background and then perform image fusion to improve the signal-to-noise ratio; it can effectively solve the problem of inconsistent dynamic benchmarks and inconsistent time benchmarks between multiple devices; finally achieve information enhancement of weak and small targets , improve the detection distance and detection probability, and enhance the detection ability of photoelectric warning reconnaissance equipment or long-distance target tracking equipment. It is especially suitable for the coordinated use of multiple photoelectric detection devices of the same type that are dispersedly installed on the moving platform.
根据本发明的第一方面，提供了一种基于惯性空间对准的多通道图像融合系统，包括：安装于不同位置的同类型的至少两个的光电成像设备、与所述光电成像设备一一对应设置的惯性测量单元和图像同步采集与处理设备；According to a first aspect of the present invention, a multi-channel image fusion system based on inertial spatial alignment is provided, comprising: at least two photoelectric imaging devices of the same type installed in different positions, and one of the photoelectric imaging devices. Correspondingly set inertial measurement unit and image synchronization acquisition and processing equipment;
所述惯性测量单元用于测量对应的所述光电成像设备的光轴指向的动态变化量；The inertial measurement unit is used to measure the dynamic change amount of the optical axis pointing of the corresponding photoelectric imaging device;
所述图像同步采集与处理设备向各个所述光电成像设备发送同步的外触发信号；The image synchronous acquisition and processing device sends a synchronous external trigger signal to each of the photoelectric imaging devices;
各个所述光电成像设备接收到所述外触发信号后进行图像的同步采集，并将采集的图像及对应的所述惯性测量单元采集的信息回传给所述图像同步采集与处理设备；After receiving the external trigger signal, each photoelectric imaging device performs synchronous image acquisition, and sends the collected image and the corresponding information collected by the inertial measurement unit back to the image synchronous acquisition and processing device;
所述图像同步采集与处理设备根据所述惯性测量单元采集的信息对各个图像进行惯性空间对准后，对各个图像进行融合。The image synchronous acquisition and processing device performs inertial space alignment on each image according to the information collected by the inertial measurement unit, and then fuses each image.
在上述技术方案的基础上，本发明还可以作出如下改进。On the basis of the above technical solutions, the present invention can also make the following improvements.
可选的，安装所述光电成像设备时，通过图像标定方法获得每个光电成像通道的内参矩阵与外参矩阵，求解得到畸变参数；根据该畸变参数对对应的光电成像设备安装后的扫描轴系与光轴进行标定，获得其运动学参数模型；根据所述运动学参数模型，对该光电成像设备的指向和图像进行修正。Optionally, when the photoelectric imaging device is installed, the internal parameter matrix and the external parameter matrix of each photoelectric imaging channel are obtained by an image calibration method, and the distortion parameter is obtained by solving; The system is calibrated with the optical axis to obtain its kinematic parameter model; according to the kinematic parameter model, the orientation and image of the photoelectric imaging device are corrected.
可选的，各个所述光电成像设备进行同步采集之前，使各个所述光电成像设备指向同一方向或同一目标进行同步扫描。Optionally, before each photoelectric imaging device performs synchronous acquisition, each photoelectric imaging device is directed to the same direction or the same target to perform synchronous scanning.
可选的，各个所述光电成像设备将修正畸变后的图像、伺服信息和惯性测量单元测量的信息传递给所述图像同步采集与处理设备后，所述图像同步采集与处理设备对各个所述光电成像设备的伺服信息及惯性信息进行处理，计算得到各图像对应的惯性空间坐标，再以同一时刻各图像光轴的平均指向为基准剪裁和/或旋转各图像。Optionally, after each of the photoelectric imaging devices transmits the distorted image, the servo information, and the information measured by the inertial measurement unit to the synchronous image acquisition and processing device, the synchronous image acquisition and processing device can perform the synchronous image acquisition and processing on each of the described images. The servo information and inertial information of the photoelectric imaging device are processed to calculate the inertial space coordinates corresponding to each image, and then each image is cropped and/or rotated based on the average orientation of the optical axis of each image at the same time.
可选的，所述图像同步采集与处理设备对各个图像进行像素叠加实现图像融合。Optionally, the image synchronous acquisition and processing device performs pixel stacking on each image to implement image fusion.
可选的，所述图像同步采集与处理设备对图像进行像素叠加后还包括：获得目标的指向信息或进行类型识别。Optionally, after the image synchronous acquisition and processing device superimposes the pixels on the image, the method further includes: obtaining orientation information of the target or performing type identification.
可选的，所述多通道图像融合系统还包括图像显示单元，用于对融合后图像进行显示。Optionally, the multi-channel image fusion system further includes an image display unit for displaying the fused image.
根据本发明的第二方面，提供一种基于惯性空间对准的多通道图像融合方法，包括：According to a second aspect of the present invention, a multi-channel image fusion method based on inertial space alignment is provided, comprising:
步骤1，在不同位置安装同类型的至少两个的光电成像设备，对图像畸变和安装基准进行标定；Step 1, install at least two photoelectric imaging devices of the same type in different positions, and calibrate the image distortion and installation reference;
步骤2，通过各个所述光电成像设备的跟踪伺服系统进行同步扫描，使各个所述光电成像设备指向同一方向或同一目标；Step 2, performing synchronous scanning through the tracking servo system of each photoelectric imaging device, so that each photoelectric imaging device points to the same direction or the same target;
步骤3，当各个所述光电成像设备扫描或跟踪进入稳定状态后，各个所述光电成像设备根据所述图像同步采集与处理设备发送来的外触发脉冲进行图像采集；采集完成后，各个所述光电成像设备将修正畸变后的图像、伺服信息和所述惯性测量单元采集的信息传递给所述图像同步采集与处理设备；Step 3: After each photoelectric imaging device scans or tracks into a stable state, each photoelectric imaging device performs image acquisition according to the external trigger pulse sent by the image synchronization acquisition and processing device; after the acquisition is completed, each photoelectric imaging device performs image acquisition. The photoelectric imaging device transmits the distorted image, the servo information and the information collected by the inertial measurement unit to the image synchronization acquisition and processing device;
步骤4，所述图像同步采集与处理设备对各个所述光电成像设备的伺服信息及惯性信息进行处理，计算得到各图像对应的惯性空间坐标，再以同一时刻各图像光轴的平均指向为基准剪裁和/或旋转各图像；Step 4: The image synchronous acquisition and processing device processes the servo information and inertial information of each of the photoelectric imaging devices, calculates the inertial space coordinates corresponding to each image, and then takes the average orientation of the optical axes of each image at the same time as the benchmark. crop and/or rotate each image;
步骤5，对消除畸变和惯性系坐标误差的图像进行像素叠加。Step 5: Perform pixel stacking on the image whose distortion and inertial system coordinate errors are eliminated.
在上述技术方案的基础上，本发明还可以作出如下改进。On the basis of the above technical solutions, the present invention can also make the following improvements.
可选的，所述步骤1中对图像畸变和安装基准进行标定的过程包括：Optionally, the process of calibrating image distortion and installation datum in step 1 includes:
通过图像标定方法获得每个光电成像通道的内参矩阵与外参矩阵，求解得到畸变参数；根据该畸变参数对对应的光电成像设备安装后的扫描轴系与光轴进行标定，获得其运动学参数模型；根据所述运动学参数模型，对该光电成像设备的指向和图像进行修正。The internal parameter matrix and external parameter matrix of each photoelectric imaging channel are obtained by the image calibration method, and the distortion parameters are obtained by solving; according to the distortion parameters, the scanning axis and optical axis of the corresponding photoelectric imaging device after installation are calibrated to obtain its kinematic parameters model; according to the kinematic parameter model, the orientation and image of the photoelectric imaging device are corrected.
可选的，所述步骤5之后还包括：Optionally, after
步骤6，获得目标的指向信息或进行类型识别。
本发明实施例提供的一种基于惯性空间对准的多通道图像融合系统及方法，针对分开安装的多个同类图像探测装置，在不改变硬件的前提下，首先对图像进行像素级畸变标定，实现多个通道间的像素对齐；再通过协同扫描与外触发进行同步成像，利用探测装置自身携带的惯性器件在惯性空间下对准图像；后台同步获取多个探测器图像后进行图像融合，提升信噪比；可有效解决多个设备之间动态基准不一致，时间基准不一致的问题；对图像的融合也直接作用于像素匹配，无需多尺度滤波，无需在探测到目标后进行目标和背景的图像分割；最终实现在不降低探测图像频率的情况下，有效提升目标信噪比(理论上可提升n1/2倍，n为通道数量)，从而提升目标探测距离和探测概率，尤其适用于在动平台上分散安装的多个同类型光电探测设备的协同使用。The embodiment of the present invention provides a multi-channel image fusion system and method based on inertial space alignment. For a plurality of image detection devices of the same type installed separately, without changing the hardware, first perform pixel-level distortion calibration on the image, Realize pixel alignment between multiple channels; then perform synchronous imaging through coordinated scanning and external triggering, and use the inertial device carried by the detection device to align the image in the inertial space; the background synchronously acquires multiple detector images and then performs image fusion to improve the Signal-to-noise ratio; can effectively solve the problem of inconsistent dynamic benchmarks and inconsistent time benchmarks among multiple devices; image fusion also directly affects pixel matching, no need for multi-scale filtering, and no need to perform target and background images after the target is detected Segmentation; ultimately, without reducing the frequency of detection images, the target signal-to-noise ratio can be effectively improved (in theory, it can be increased by n1/2 times, n is the number of channels), thereby improving the target detection distance and detection probability, especially suitable for Synergistic use of multiple photoelectric detection devices of the same type that are dispersedly installed on the moving platform.
附图说明Description of drawings
图1为本发明实施例提供的一种基于惯性空间对准的多通道图像融合系统的结构示意图；1 is a schematic structural diagram of a multi-channel image fusion system based on inertial space alignment provided by an embodiment of the present invention;
图2为本发明实施例提供的一种基于惯性空间对准的多通道图像融合方法的流程图；2 is a flowchart of a multi-channel image fusion method based on inertial space alignment provided by an embodiment of the present invention;
附图中，各标号所代表的部件列表如下：In the accompanying drawings, the list of components represented by each number is as follows:
1、光电成像设备，2、惯性测量单元，3、图像同步采集与处理设备，4、图像显示单元。1. Photoelectric imaging equipment, 2. Inertial measurement unit, 3. Image synchronization acquisition and processing equipment, 4. Image display unit.
具体实施方式Detailed ways
以下结合附图对本发明的原理和特征进行描述，所举实例只用于解释本发明，并非用于限定本发明的范围。The principles and features of the present invention will be described below with reference to the accompanying drawings. The examples are only used to explain the present invention, but not to limit the scope of the present invention.
图1为本发明提供的一种基于惯性空间对准的多通道图像融合系统的结构示意图，如图1所示，该多通道图像融合系统包括：安装于不同位置的同类型的至少两个的光电成像设备1、与光电成像设备1一一对应设置的惯性测量单元2和图像同步采集与处理设备3。1 is a schematic structural diagram of a multi-channel image fusion system based on inertial space alignment provided by the present invention. As shown in FIG. 1 , the multi-channel image fusion system includes: at least two of the same type installed in different positions The photoelectric imaging device 1 , the inertial measurement unit 2 and the image synchronous acquisition and processing device 3 are provided in a one-to-one correspondence with the photoelectric imaging device 1 .
具体实施中，该光电成像设备1可以为同类型的光电警戒、侦察或跟踪设备。In specific implementation, the photoelectric imaging device 1 can be the same type of photoelectric warning, reconnaissance or tracking device.
惯性测量单元2可以为直接安装在光电成像设备1内的惯性元器件。通常安装在动平台(如舰船、车辆、飞机)上的光电设备由于需要与外界交换目标坐标信息，为了消除自身运动所带来的误差，一般都会安装惯性测量单元2(如陀螺仪、惯导等)。The inertial measurement unit 2 may be an inertial component directly installed in the photoelectric imaging device 1 . The optoelectronic devices usually installed on moving platforms (such as ships, vehicles, and aircraft) need to exchange target coordinate information with the outside world. In order to eliminate the errors caused by their own motion, inertial measurement units 2 (such as gyroscopes, inertial guide, etc.).
惯性测量单元2用于测量对应的光电成像设备1的光轴指向的动态变化量。The inertial measurement unit 2 is used to measure the dynamic variation of the optical axis pointing of the corresponding photoelectric imaging device 1 .
图像同步采集与处理设备3向各个光电成像设备1发送同步的外触发信号。The image synchronous acquisition and processing device 3 sends a synchronous external trigger signal to each photoelectric imaging device 1 .
各个光电成像设备1接收到外触发信号后进行图像的同步采集，收到信号后进行采集可以确保各设备图像采集时间节点同步，此时采集的各个图像为时间对准的图像，可以保证对应图像采集的目标处于空间中同一位置。并将采集的图像及对应的惯性测量单元2采集的信息回传给图像同步采集与处理设备3。Each photoelectric imaging device 1 performs synchronous image acquisition after receiving the external trigger signal. After receiving the signal, the acquisition can ensure that the image acquisition time nodes of each device are synchronized. The images collected at this time are time-aligned images, which can ensure that the corresponding images are The collected objects are in the same position in space. The collected images and the corresponding information collected by the inertial measurement unit 2 are sent back to the image synchronous collection and processing device 3 .
图像同步采集与处理设备3根据惯性测量单元采集的信息对各个图像进行惯性空间对准后，对各个图像进行融合。The image synchronous acquisition and processing device 3 performs inertial space alignment on each image according to the information collected by the inertial measurement unit, and then fuses each image.
本发明提供一种基于惯性空间对准的多通道图像融合系统，无需使用复杂的光学系统或对光电探测设备进行改造，针对分开安装的多个同类图像探测装置，在不改变硬件的前提下，首先对图像进行像素级畸变标定，实现多个通道间的像素对齐；再通过协同扫描与外触发进行同步成像，利用探测装置自身携带的惯性器件在惯性空间下对准图像；后台同步获取多个探测器图像后进行图像融合，提升信噪比；可有效解决多个设备之间动态基准不一致，时间基准不一致的问题；最终实现弱小目标的信息增强，提升探测距离与探测概率，用于光电警戒侦察设备或远距离目标跟踪设备的探测能力增强。尤其适用于在动平台上分散安装的多个同类型光电探测设备的协同使用。The present invention provides a multi-channel image fusion system based on inertial space alignment, which does not require the use of complex optical systems or the modification of photoelectric detection equipment. First, the pixel-level distortion calibration is performed on the image to achieve pixel alignment among multiple channels; then synchronous imaging is performed through coordinated scanning and external triggering, and the inertial device carried by the detection device is used to align the image in the inertial space; the background synchronously acquires multiple Image fusion is performed after the detector image to improve the signal-to-noise ratio; it can effectively solve the problem of inconsistent dynamic benchmarks and inconsistent time benchmarks among multiple devices; finally, the information enhancement of weak and small targets can be realized, and the detection distance and detection probability can be improved for photoelectric warning. Enhanced detection capabilities of reconnaissance equipment or long-range target tracking equipment. It is especially suitable for the coordinated use of multiple photoelectric detection devices of the same type that are dispersedly installed on the moving platform.
实施例1Example 1
本发明提供的实施例1为本发明提供的一种基于惯性空间对准的多通道图像融合系统的实施例，结合图1可知，该多通道图像融合系统的实施例包括：安装于不同位置的同类型的至少两个的光电成像设备1、与光电成像设备1一一对应设置的惯性测量单元2和图像同步采集与处理设备3。Embodiment 1 provided by the present invention is an embodiment of a multi-channel image fusion system based on inertial space alignment provided by the present invention. It can be seen from FIG. 1 that the embodiment of the multi-channel image fusion system includes: At least two photoelectric imaging devices 1 of the same type, inertial measurement units 2 and image synchronization acquisition and processing devices 3 that are provided in a one-to-one correspondence with the photoelectric imaging devices 1 .
在一种可能的实施例方式中，安装光电成像设备1时，通过图像标定方法(如张正友标定法等)获得每个光电成像通道的内参矩阵与外参矩阵，求解得到畸变参数；根据该畸变参数对对应的光电成像设备1安装后的扫描轴系与光轴进行标定，获得其运动学参数模型；根据运动学参数模型，对该光电成像设备1的指向和图像进行修正。In a possible embodiment, when the photoelectric imaging device 1 is installed, the internal parameter matrix and the external parameter matrix of each photoelectric imaging channel are obtained by an image calibration method (such as Zhang Zhengyou's calibration method, etc.), and the distortion parameters are obtained by solving; The parameters calibrate the scanning axis and optical axis of the corresponding photoelectric imaging device 1 after installation, and obtain its kinematic parameter model; according to the kinematic parameter model, the orientation and image of the photoelectric imaging device 1 are corrected.
惯性测量单元2用于测量对应的光电成像设备1的光轴指向的动态变化量。The inertial measurement unit 2 is used to measure the dynamic variation of the optical axis pointing of the corresponding photoelectric imaging device 1 .
图像同步采集与处理设备3向各个光电成像设备1发送同步的外触发信号。The image synchronous acquisition and processing device 3 sends a synchronous external trigger signal to each photoelectric imaging device 1 .
在一种可能的实施例方式中，使各光电成像设备1在同一时刻尽量指向同一方向。或当存在明确目标时，对同一目标进行跟踪。In a possible implementation manner, each photoelectric imaging device 1 is made to point in the same direction as much as possible at the same time. Or track the same target when there is a clear target.
各个光电成像设备1接收到外触发信号后进行图像的同步采集，并将采集的图像及对应的惯性测量单元2采集的信息回传给图像同步采集与处理设备3。After receiving the external trigger signal, each photoelectric imaging device 1 performs synchronous image acquisition, and transmits the collected image and the information collected by the corresponding inertial measurement unit 2 back to the image synchronous acquisition and processing device 3 .
图像同步采集与处理设备3根据惯性测量单元采集的信息对各个图像进行惯性空间对准后，对各个图像进行融合。The image synchronous acquisition and processing device 3 performs inertial space alignment on each image according to the information collected by the inertial measurement unit, and then fuses each image.
在一种可能的实施例方式中，各个光电成像设备1将修正畸变后的图像、伺服信息和惯性测量单元2测量的信息传递给图像同步采集与处理设备3后，图像同步采集与处理设备3对各个光电成像设备1的伺服信息及惯性信息进行处理，计算得到各图像对应的惯性空间坐标，再以同一时刻各图像光轴的平均指向为基准剪裁和/或旋转各图像。In a possible embodiment, each photoelectric imaging device 1 transmits the distorted image, the servo information and the information measured by the inertial measurement unit 2 to the image synchronous acquisition and processing device 3, and the image synchronous acquisition and processing device 3 The servo information and inertial information of each photoelectric imaging device 1 are processed, the inertial space coordinates corresponding to each image are obtained by calculation, and then each image is cropped and/or rotated based on the average orientation of the optical axis of each image at the same moment.
在一种可能的实施例方式中，图像同步采集与处理设备3对各个图像进行像素叠加实现图像融合。In a possible embodiment, the image synchronous acquisition and processing device 3 superimposes pixels on each image to implement image fusion.
以消除畸变和惯性系坐标误差的图像进行像素叠加。理论上目标信号的强度将增加n倍，噪声信号的强度增加n1/2倍，故理论上可得到信噪比增强n1/2倍的图像。Pixel overlay is performed on images with distortion and inertial coordinate errors removed. Theoretically, the intensity of the target signal will increase by n times, and the intensity of the noise signal will increase by n1/2 times, so theoretically, an image with an enhanced signal-to-noise ratio of n1/2 times can be obtained.
在一种可能的实施例方式中，图像同步采集与处理设备3对图像进行像素叠加后还包括：对增强后的图像进行进一步处理，获得目标的指向信息或进行类型识别等操作。In a possible embodiment, after the image synchronous acquisition and processing device 3 superimposes the pixels of the image, it further includes: further processing the enhanced image, obtaining the pointing information of the target, or performing operations such as type identification.
在一种可能的实施例方式中，多通道图像融合系统还包括图像显示单元4，用于对融合后图像进行显示。In a possible embodiment, the multi-channel image fusion system further includes an image display unit 4 for displaying the fused image.
实施例2Example 2
本发明提供的实施例2为本发明提供的一种基于惯性空间对准的多通道图像融合方法的实施例，如图2所示为本发明实施例提供的一种基于惯性空间对准的多通道图像融合方法的流程图，结合图2可知，该多通道图像融合方法的实施例包括：Embodiment 2 provided by the present invention is an embodiment of a multi-channel image fusion method based on inertial space alignment provided by the present invention. The flow chart of the channel image fusion method can be seen in conjunction with FIG. 2 , the embodiment of the multi-channel image fusion method includes:
步骤1，在不同位置安装同类型的至少两个的光电成像设备，对图像畸变和安装基准进行标定。Step 1, at least two photoelectric imaging devices of the same type are installed in different positions, and the image distortion and the installation reference are calibrated.
在一种可能的实施例方式中，步骤1中对图像畸变和安装基准进行标定的过程包括：In a possible embodiment, the process of calibrating the image distortion and the installation datum in step 1 includes:
通过图像标定方法(如张正友标定法等)获得每个光电成像通道的内参矩阵与外参矩阵，求解得到畸变参数；根据该畸变参数对对应的光电成像设备安装后的扫描轴系与光轴进行标定，获得其运动学参数模型；根据运动学参数模型，对该光电成像设备的指向和图像进行修正。The internal parameter matrix and external parameter matrix of each photoelectric imaging channel are obtained by the image calibration method (such as Zhang Zhengyou calibration method), and the distortion parameters are obtained by solving; Calibration is performed to obtain its kinematic parameter model; according to the kinematic parameter model, the orientation and image of the photoelectric imaging device are corrected.
步骤2，通过各个光电成像设备的跟踪伺服系统进行同步扫描，使各个光电成像设备指向同一方向或同一目标。Step 2, performing synchronous scanning through the tracking servo system of each photoelectric imaging device, so that each photoelectric imaging device points to the same direction or the same target.
步骤3，当各个光电成像设备扫描或跟踪进入稳定状态后，各个光电成像设备根据图像同步采集与处理设备发送来的外触发脉冲进行图像采集；采集完成后，各个光电成像设备将修正畸变后的图像、伺服信息和惯性测量单元采集的信息传递给图像同步采集与处理设备。Step 3: After each photoelectric imaging device scans or tracks into a stable state, each photoelectric imaging device performs image acquisition according to the external trigger pulse sent by the image synchronous acquisition and processing device; after the acquisition is completed, each photoelectric imaging device will correct the distorted image. The image, servo information and information collected by the inertial measurement unit are transmitted to the image synchronization acquisition and processing equipment.
步骤4，图像同步采集与处理设备对各个光电成像设备的伺服信息及惯性信息进行处理，计算得到各图像对应的惯性空间坐标，再以同一时刻各图像光轴的平均指向为基准剪裁和/或旋转各图像。Step 4: The image synchronization acquisition and processing device processes the servo information and inertial information of each photoelectric imaging device, calculates and obtains the inertial space coordinates corresponding to each image, and then uses the average orientation of the optical axis of each image at the same time as the benchmark to cut and/or Rotate each image.
步骤5，对消除畸变和惯性系坐标误差的图像进行像素叠加。Step 5: Perform pixel stacking on the image whose distortion and inertial system coordinate errors are eliminated.
在一种可能的实施例方式中，步骤5之后还包括：步骤6，获得目标的指向信息或进行类型识别。In a possible implementation manner, after
可以理解的是，本发明提供的一种基于惯性空间对准的多通道图像融合方法与前述各实施例提供的基于惯性空间对准的多通道图像融合系统相对应，基于惯性空间对准的多通道图像融合方法的相关技术特征可参考基于惯性空间对准的多通道图像融合系统的相关技术特征，在此不再赘述。It can be understood that the multi-channel image fusion method based on inertial space alignment provided by the present invention corresponds to the multi-channel image fusion system based on inertial space alignment provided by the foregoing embodiments, and the multi-channel image fusion method based on inertial space alignment For the related technical features of the channel image fusion method, reference may be made to the related technical features of the multi-channel image fusion system based on inertial space alignment, which will not be repeated here.
本发明实施例提供的一种基于惯性空间对准的多通道图像融合系统及方法，针对分开安装的多个同类图像探测装置，在不改变硬件的前提下，首先对图像进行像素级畸变标定，实现多个通道间的像素对齐；再通过协同扫描与外触发进行同步成像，利用探测装置自身携带的惯性器件在惯性空间下对准图像；后台同步获取多个探测器图像后进行图像融合，提升信噪比；可有效解决多个设备之间动态基准不一致，时间基准不一致的问题；对图像的融合也直接作用于像素匹配，无需多尺度滤波，无需在探测到目标后进行目标和背景的图像分割；最终实现在不降低探测图像频率的情况下，有效提升目标信噪比(理论上可提升n1/2倍，n为通道数量)，从而提升目标探测距离和探测概率，尤其适用于在动平台上分散安装的多个同类型光电探测设备的协同使用。The embodiment of the present invention provides a multi-channel image fusion system and method based on inertial space alignment. For a plurality of image detection devices of the same type installed separately, without changing the hardware, first perform pixel-level distortion calibration on the image, Realize pixel alignment between multiple channels; then perform synchronous imaging through coordinated scanning and external triggering, and use the inertial device carried by the detection device to align the image in the inertial space; the background synchronously acquires multiple detector images and then performs image fusion to improve the Signal-to-noise ratio; can effectively solve the problem of inconsistent dynamic benchmarks and inconsistent time benchmarks among multiple devices; image fusion also directly affects pixel matching, no need for multi-scale filtering, and no need to perform target and background images after the target is detected Segmentation; ultimately, without reducing the frequency of detection images, the target signal-to-noise ratio can be effectively improved (in theory, it can be increased by n1/2 times, n is the number of channels), thereby improving the target detection distance and detection probability, especially suitable for Synergistic use of multiple photoelectric detection devices of the same type that are dispersedly installed on the moving platform.
需要说明的是，在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详细描述的部分，可以参见其它实施例的相关描述。It should be noted that, in the foregoing embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式计算机或者其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded computer or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means implementing the functions specified in one or more of the flowcharts and/or one or more blocks of the block diagrams.
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.
尽管已描述了本发明的优选实施例，但本领域内的技术人员一旦得知了基本创造概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本发明范围的所有变更和修改。Although preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of the present invention.
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包括这些改动和变型在内。It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111502699.8ACN114419302B (en) | 2021-12-09 | 2021-12-09 | A multi-channel image fusion system and method based on inertial space alignment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111502699.8ACN114419302B (en) | 2021-12-09 | 2021-12-09 | A multi-channel image fusion system and method based on inertial space alignment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114419302Atrue CN114419302A (en) | 2022-04-29 |
| CN114419302B CN114419302B (en) | 2025-09-02 |
Family
ID=81265542
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111502699.8AActiveCN114419302B (en) | 2021-12-09 | 2021-12-09 | A multi-channel image fusion system and method based on inertial space alignment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114419302B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006295553A (en)* | 2005-04-11 | 2006-10-26 | Tamron Co Ltd | Imaging device having image-shaking preventive function |
| US7496241B1 (en)* | 2005-09-08 | 2009-02-24 | Goodrich Corporation | Precision optical systems with performance characterization and uses thereof |
| US20130064427A1 (en)* | 2010-04-13 | 2013-03-14 | Frederic Picard | Methods and systems for object tracking |
| CN103345751A (en)* | 2013-07-02 | 2013-10-09 | 北京邮电大学 | Visual positioning method based on robust feature tracking |
| CN106197429A (en)* | 2016-07-21 | 2016-12-07 | 触景无限科技（北京）有限公司 | A kind of Multi-information acquisition location equipment and system |
| CN108645402A (en)* | 2018-03-30 | 2018-10-12 | 深圳清创新科技有限公司 | Camera shooting and inertia measurement sensing device, scene cut and pose computing system |
| CN110132060A (en)* | 2019-04-26 | 2019-08-16 | 南京理工大学 | A method of intercepting drones based on visual navigation |
| CN110223223A (en)* | 2019-04-28 | 2019-09-10 | 北京清城同衡智慧园高新技术研究院有限公司 | Street scan method, device and scanner |
| CN110517216A (en)* | 2019-08-30 | 2019-11-29 | 的卢技术有限公司 | A kind of SLAM fusion method and its system based on polymorphic type camera |
| CN111292277A (en)* | 2018-12-10 | 2020-06-16 | 深圳迈瑞生物医疗电子股份有限公司 | Ultrasonic fusion imaging method and ultrasonic fusion imaging navigation system |
| CN112070820A (en)* | 2020-08-29 | 2020-12-11 | 南京翱翔信息物理融合创新研究院有限公司 | Distributed augmented reality positioning terminal, positioning server and positioning system |
| CN112129317A (en)* | 2019-06-24 | 2020-12-25 | 南京地平线机器人技术有限公司 | Information acquisition time difference determining method and device, electronic equipment and storage medium |
| CN112305576A (en)* | 2020-10-31 | 2021-02-02 | 中环曼普科技(南京)有限公司 | Multi-sensor fusion SLAM algorithm and system thereof |
| CN112348775A (en)* | 2020-10-13 | 2021-02-09 | 上海城建城市运营(集团)有限公司 | Vehicle-mounted all-round-looking-based pavement pool detection system and method |
| CN112689084A (en)* | 2020-12-08 | 2021-04-20 | 武汉华中天经通视科技有限公司 | Airborne photoelectric reconnaissance imaging system and electronic image stabilization method |
| CN113160221A (en)* | 2021-05-14 | 2021-07-23 | 深圳市奥昇医疗科技有限责任公司 | Image processing method, image processing device, computer equipment and storage medium |
- 2021
- 2021-12-09CNCN202111502699.8Apatent/CN114419302B/enactiveActive
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006295553A (en)* | 2005-04-11 | 2006-10-26 | Tamron Co Ltd | Imaging device having image-shaking preventive function |
| US7496241B1 (en)* | 2005-09-08 | 2009-02-24 | Goodrich Corporation | Precision optical systems with performance characterization and uses thereof |
| US20130064427A1 (en)* | 2010-04-13 | 2013-03-14 | Frederic Picard | Methods and systems for object tracking |
| CN103345751A (en)* | 2013-07-02 | 2013-10-09 | 北京邮电大学 | Visual positioning method based on robust feature tracking |
| CN106197429A (en)* | 2016-07-21 | 2016-12-07 | 触景无限科技（北京）有限公司 | A kind of Multi-information acquisition location equipment and system |
| CN108645402A (en)* | 2018-03-30 | 2018-10-12 | 深圳清创新科技有限公司 | Camera shooting and inertia measurement sensing device, scene cut and pose computing system |
| CN111292277A (en)* | 2018-12-10 | 2020-06-16 | 深圳迈瑞生物医疗电子股份有限公司 | Ultrasonic fusion imaging method and ultrasonic fusion imaging navigation system |
| CN110132060A (en)* | 2019-04-26 | 2019-08-16 | 南京理工大学 | A method of intercepting drones based on visual navigation |
| CN110223223A (en)* | 2019-04-28 | 2019-09-10 | 北京清城同衡智慧园高新技术研究院有限公司 | Street scan method, device and scanner |
| CN112129317A (en)* | 2019-06-24 | 2020-12-25 | 南京地平线机器人技术有限公司 | Information acquisition time difference determining method and device, electronic equipment and storage medium |
| CN110517216A (en)* | 2019-08-30 | 2019-11-29 | 的卢技术有限公司 | A kind of SLAM fusion method and its system based on polymorphic type camera |
| CN112070820A (en)* | 2020-08-29 | 2020-12-11 | 南京翱翔信息物理融合创新研究院有限公司 | Distributed augmented reality positioning terminal, positioning server and positioning system |
| CN112348775A (en)* | 2020-10-13 | 2021-02-09 | 上海城建城市运营(集团)有限公司 | Vehicle-mounted all-round-looking-based pavement pool detection system and method |
| CN112305576A (en)* | 2020-10-31 | 2021-02-02 | 中环曼普科技(南京)有限公司 | Multi-sensor fusion SLAM algorithm and system thereof |
| CN112689084A (en)* | 2020-12-08 | 2021-04-20 | 武汉华中天经通视科技有限公司 | Airborne photoelectric reconnaissance imaging system and electronic image stabilization method |
| CN113160221A (en)* | 2021-05-14 | 2021-07-23 | 深圳市奥昇医疗科技有限责任公司 | Image processing method, image processing device, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| JUAN JURADO 等: "Inertial and imaging sensor fusion for image-aided navigation with affine distortion prediction", 《PROCEEDINGS OF THE 2012 IEEE/ION POSITION, LOCATION AND NAVIGATION SYMPOSIUM》, 12 July 2012 (2012-07-12), pages 518 - 526* |
| 杨观赐 等: "视觉与惯性传感器融合的SLAM技术综述", 《贵州大学学报（自然科学版）》, vol. 37, no. 6, 30 November 2020 (2020-11-30), pages 1 - 12* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114419302B (en) | 2025-09-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2020101932A4 (en) | Binocular vision–based method and system for pose measurement of cantilever tunneling equipment | |
| JP6484729B2 (en) | Unmanned aircraft depth image acquisition method, acquisition device, and unmanned aircraft | |
| CN111354042B (en) | Feature extraction method and device of robot visual image, robot and medium | |
| CN106780601B (en) | Spatial position tracking method and device and intelligent equipment | |
| CN108020825B (en) | Fusion calibration system and method for laser radar, laser camera and video camera | |
| EP3792660A1 (en) | Method, apparatus and system for measuring distance | |
| CN106370884B (en) | Vehicle speed measuring method based on binocular camera computer vision technology | |
| GB2593960A (en) | 3-D imaging apparatus and method for dynamically and finely detecting small underwater objects | |
| CN105115445A (en) | Three-dimensional imaging system and imaging method based on combination of depth camera and binocular vision | |
| CN109263637B (en) | Collision prediction method and device | |
| CN101371116B (en) | Wavefront Phase Difference and Range Measurement Phase Camera | |
| JP2008140370A (en) | Stereo camera intrusion detection system | |
| JP2012189445A (en) | Object detection device and object detection method | |
| CN113895482B (en) | Train speed measuring method and device based on trackside equipment | |
| CN107560592A (en) | A kind of precision ranging method for optronic tracker linkage target | |
| CN110619617B (en) | Three-dimensional imaging method, device, equipment and computer readable storage medium | |
| JP2016075615A (en) | Target detection device | |
| KR101238748B1 (en) | System for measuring distance of target using step-staring infrared sensor unit | |
| CN107680065A (en) | Radiation image bearing calibration and means for correcting and correction system | |
| CN105258673A (en) | Target range finding method and apparatus based on binocular synthetic aperture focus image | |
| JP2019219180A (en) | Object detection device for vehicle | |
| CN115690623A (en) | Remote target damage assessment method based on three-dimensional reconstruction | |
| CN105403886B (en) | A kind of carried SAR scaler picture position extraction method | |
| CN108305290B (en) | A Precise Spectrum Measurement Method for Moving Targets | |
| CN114419302A (en) | Multichannel image fusion system and method based on inertial space alignment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |