Disclosure of Invention
The invention provides a method for registering and fusing an infrared image and an SAR image in unmanned aerial vehicle dynamic flight to solve the problems.
The invention aims to provide a method for registering and fusing an infrared image and an SAR image in unmanned aerial vehicle dynamic flight, which specifically comprises the following steps:
S1, ground calibration, namely measuring and calibrating the relation between the focal length of an infrared camera and the focal length of an infrared Y-direction, recording the relation between the focal length and the focal length as a file, and storing the file on a camera control board;
S2, roughly matching the resolution of the ground pixels, wherein a camera control board controls the motion of an infrared camera through a PID algorithm until the field angle of a single pixel of an infrared image is matched with the field angle of a single pixel of an SAR image;
s3, extracting and registering image descriptors, namely extracting heterogeneous image descriptors of the infrared image and the SAR image, matching the descriptors, and storing the projection relation of each pixel coordinate of the infrared image and each pixel coordinate of the SAR image as a matrix C;
S4, image fusion, namely converting an infrared image format from RGB to YUV according to the registered image, superposing corresponding brightness values of the SAR image on a Y channel, calculating local contrast and information entropy of the infrared image and the SAR image, and dynamically adjusting fusion proportion of the infrared image and the SAR image to generate a registered fusion image.
Preferably, the measurement calibration method in the step S1 specifically comprises the step of gradually measuring the Y-direction view angle of the infrared camera by adopting a foldback infrared SAR double-light common-view-field light pipe, wherein the foldback infrared SAR double-light common-view-field light pipe contains cross filaments, and the stepping length is 0.1m.
Preferably, in step S2, the method for adjusting the field angle of the single pixel of the infrared image to match the field angle of the single pixel of the SAR image is specifically as follows:
the onboard computer calculates the focal length of the infrared camera when the resolution of each pixel of the infrared camera is equal to that of the ground pixels of the SAR image according to the flight height and the photoelectric loading azimuth pitch angle of the unmanned aerial vehicleThe camera control board controls the infrared camera to adjust to the focal length through PID algorithmUntil the field angle of the infrared single pixel matches the field angle of the SAR single pixel.
Preferably, the heterogeneous image descriptors of the infrared image and the SAR image in the step S3 comprise phase consistency descriptors and gradient descriptors, and the matching is carried out by adopting a method of weighting correlation distances, wherein the specific method is as follows:
S301, calculating a phase consistency descriptor of the heterologous image, wherein the phase consistency descriptor is used for measuring the phase of a specific frequency component in the heterologous image, and for each pixel, the phase can be calculated by the following formula:
;
Wherein,Is the pixel coordinates, w is a different scale,Is the phase at the scale w, M represents the number of dimensions;
s302, calculating gradient descriptors of the heterologous images;
Gradient direction descriptors of the heterologous image are as follows:
;;
Wherein,AndThe gradients in x-direction and y-direction for the scale m, respectively, I being the brightness of the image, whereby the gradient descriptor of the heterologous image is expressed as:
;
s303, setting a statistical scale as m, and taking (x, y) pixels as the center, wherein a gradient histogram of 10×10 pixels around the center is expressed as follows:
;
S304, taking the image target size as a reference, taking M/2 scales upwards and M/2 scales downwards, wherein the M scales are altogether; phase consistency descriptor at mth scaleGradient descriptorGradient histogramConstitutes a descriptor vector at the mth scale:
;
S305, matching by adopting a weighted correlation distance method, wherein the expression is as follows:
;
wherein, WCD represents the weighted correlation distance, namely the weight sum of the difference of the infrared image and SAR image descriptor vector from the scale 1 to the scale M; AndA multi-scale descriptor vector representing each feature point in the infrared image and the SAR image, respectively; Is the weight of the m-th scale, expressed as:; is the descriptor vector of the mth scaleAndIs a variance of (2);
When the WCD value is greater than 0.5, it is consideredAndAnd storing the projection relation of each pixel coordinate in the infrared image and the SAR image as a matrix C.
Preferably, the step S4 specifically includes the following sub-steps:
s401, calculating local contrast of the infrared image and the SAR image, normalizing the local contrast of the infrared image and the SAR image to obtain weight of the local contrast of the infrared image and the SAR image;
S402, calculating information entropy of the infrared image and information entropy of the SAR image, and carrying out normalization processing on the information entropy of the infrared image and the information entropy of the SAR image to obtain weights of the information entropy of the infrared image and the information entropy of the SAR image;
S403, converting an RGB format file of an infrared image into YUV, superposing a Y-channel brightness value Yir of the infrared image and a Y-channel brightness value YSAR of an SAR image according to a matrix C, fusing pixel by pixel, and generating a registration fusion image Yfused, wherein the expression is as follows:
;
Wherein Yfused represents a registration fusion image, yir represents a Y-channel brightness value of an infrared image, and YSAR represents a Y-channel brightness value of an SAR image; Representing the infrared image weights based on local contrast,Representing SAR image weights based on local contrast; representing the infrared image weights based on the entropy of the information,And represents SAR image weights based on information entropy.
Preferably, the step S401 specifically includes the following sub-steps:
S4011, determining a local area taking a feature as a center, applying a mean filter to the local area to obtain local average brightness, calculating standard deviation of pixel values in the local area and local average brightness deviation, and taking the maximum value of the standard deviation as local contrast;
the calculation formulas of the local standard deviation of the infrared image and the SAR image are respectively as follows:
;
;
In the formula,Is the pixel value of the infrared image at coordinates (x, y),Is the pixel value of the SAR image at coordinates (x, y),Is the average pixel value of the local region qf in the infrared image or the SAR image, |qf| represents the total number of pixels of the local region qf in the infrared image or the SAR image;
the local contrast expression of the infrared image or SAR image is as follows:
;
;
S4012, carrying out normalization processing on local contrast of the infrared image and the SAR image to enable the sum of IRLocalContrast and SARLocalContrast to be 1, and obtaining weight of the local contrast of the infrared image and the SAR image:
;
In the formula,Representing the infrared image weights based on local contrast,The SAR image weights based on local contrast are represented.
Preferably, the step S402 specifically includes the following sub-steps:
S4021, representing the number of times of occurrence of gray value i in an infrared image or SAR image by using a histogram, wherein the gray value of each pixel point in the gray imageOr (b)The following operations are performed:
;
In the formula,AndRepresenting an infrared histogram and a SAR histogram; i represents a gray value; Representing indication function whenAt the time of being equal to i,1, Otherwise 0, whenAt the time of being equal to i,1, Otherwise 0; representing the gray value of a pixel point with coordinates (x, y) in the infrared image,Representing gray values of pixel points with coordinates of (x, y) in the SAR image;
s4022, carrying out normalization processing, converting the histogram into probability distribution, and dividing the frequency of each gray value by the total pixel number of the image:
;
;
In the formula,AndThe probability distribution for each gray value of the infrared image and the SAR image respectively,AndThe width and height of the infrared image respectively,AndThe width and the height of the SAR image are respectively;
s4023, multiplying the probability of each gray value by the logarithm based on 2 by using a shannon information entropy formula, accumulating all 256 possible gray values, and calculating an infrared image information entropy Hir and an SAR image information entropy HSAR, wherein the calculation formula is as follows:
;
;
s4024, carrying out normalization processing on information entropy of the infrared image and the SAR image so that the sum of Hir and HSAR is 1, and obtaining weight of the information entropy of the infrared image and the SAR image:
;
In the formula,Representing the infrared image weights based on the entropy of the information,And represents SAR image weights based on information entropy.
Preferably, the value range of the information entropy is 0-8 bits, when all pixels have the same gray value, hir=0 and hsar=0, and when the probability of each gray value is equal, the information entropy hir=8 and hsar=8.
Compared with the prior art, the invention has the following beneficial effects:
According to the invention, only the field angles of infrared and SAR are simply calibrated on the ground, after zooming, a registration algorithm is operated on an onboard computer, and the efficiency and accuracy of feature matching are improved by combining a multi-scale descriptor and an optimized search strategy. The method is suitable for infrared SAR image registration under the conditions that the infrared SAR radar has different resolutions and is a zoom lens, is particularly suitable for the fields of image registration of an unmanned aerial vehicle-mounted photoelectric radar comprehensive load variable-focus infrared camera and a variable ground resolution SAR radar and the like, has wide application prospect and practical application value, and remarkably reduces the workload of registration calibration.
Detailed Description
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, like modules are denoted by like reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, a detailed description thereof will not be repeated.
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not to be construed as limiting the invention.
The invention aims to solve the limitations of the prior art, and provides a novel image registration method which is suitable for an unmanned aerial vehicle-mounted photoelectric radar comprehensive load system, can reduce calibration workload and improve the efficiency and accuracy of image registration. By means of simple field angle calibration on the ground and combining the multi-scale descriptors and the optimized searching strategy, a registration algorithm can be operated on an onboard computer in real time, and the requirements of unmanned aerial vehicle real-time image processing are met.
Referring to fig. 1, the invention provides a method for registering and fusing an infrared image and an SAR image in unmanned aerial vehicle dynamic flight, which specifically comprises the following steps:
s1, ground calibration, namely, mounting an infrared camera on an unmanned aerial vehicle-mounted photoelectric turret, measuring and calibrating the relation between the focal length of the infrared camera and the angle of view in the infrared Y direction on the ground, recording the relation between the angle of view and the focal length as a file, and storing the file on a camera control board;
the measurement calibration method specifically comprises gradually measuring the Y-direction field angle of an infrared camera by adopting a foldback infrared SAR double-light common-field-of-view light pipe (the light pipe contains cross filaments, and the stepping length is 0.1 m);
The method comprises the steps of setting a focal length of an infrared camera lens to be IR_F, locking a pitch angle of the photoelectric turret to be 0, aligning the upper side edge of the infrared camera with a cross wire of a light pipe, reading an azimuth angle IRup of the photoelectric turret, rotating the photoelectric turret to align the lower side edge of the infrared camera with the cross wire of the light pipe, reading an azimuth angle IRdown of the photoelectric turret, and recording the corresponding relation between the focal length of the infrared camera lens and the angle of view to be an array c when the focal length of the infrared camera is IR_F, wherein the focal length of the infrared camera lens is IR_Y= IRup-IRdown.
S2, roughly matching the resolution of the ground pixels, wherein a camera control board controls the motion of an infrared camera through a PID algorithm until the field angle of a single pixel of an infrared image is matched with the field angle of a single pixel of an SAR image;
The method comprises the following steps that an onboard computer calculates the focal length of the infrared camera when the resolution of each pixel of the infrared camera is equal to that of the ground pixels of the SAR image according to the flight height and the photoelectric loading azimuth pitch angle of the unmanned aerial vehicleThe camera control board controls the infrared camera to adjust to the focal length through PID algorithmUntil the field angle of the infrared single pixel matches the field angle of the SAR single pixel.
If the ground resolution of the SAR image ground pixel selected by the ground station is 0.2m, the ground size represented by each pixel of the SAR is 0.2m multiplied by 0.2m, and an onboard computer calculates the focal length of the infrared camera when each pixel of the infrared camera represents the ground size of 0.2m multiplied by 0.2m according to the flying height of the unmanned aerial vehicle and the azimuth pitch angle of photoelectric load;
under the condition of considering the pitch angle theta, the calculation formula of the focal length of the infrared camera is as follows:
;
wherein f is the focal length of the infrared camera, H is the flying height of the unmanned aerial vehicle, theta is the pitch angle of the photoelectric load, namely the included angle of the camera relative to the horizontal plane, S is the size of the infrared camera pixel, and P is the ground size represented by the pixel, wherein the ground size is 0.2 meter.
Similarly, the focal length of the infrared camera when SAR is calculated to be of other resolutions is calculated, and the camera control board controls infrared adjustment to the corresponding focal length through PID algorithmThe adjustment is repeated until the field angle of the infrared single pixel matches the field angle of the SAR single pixel.
S3, extracting and registering image descriptors, namely extracting heterogeneous image descriptors of the infrared image and the SAR image, matching the descriptors, and storing the projection relation of each pixel coordinate of the infrared image and each pixel coordinate of the SAR image as a matrix C;
The heterogeneous image descriptor of the infrared image and the SAR image comprises a phase consistency descriptor and a gradient descriptor, the method for matching the descriptors is to extract a general feature descriptor of the infrared image and the SAR image by combining the phase consistency descriptor and the gradient descriptor and match the general feature descriptor by adopting a method of weighting a correlation distance (Weighted Correlation Distance, WCD), and the specific method is as follows:
S301, calculating a phase consistency descriptor of the heterologous image, wherein the phase consistency descriptor is used for measuring the phase of a specific frequency component in the heterologous image, and for each pixel, the phase can be calculated by the following formula:
;
Wherein,Is the pixel coordinates, w is a different scale,The phase at the scale w, M represents the number of dimensions (typically a value of 10);
s302, calculating gradient descriptors of the heterologous images;
Gradient direction descriptors of the heterologous image are as follows:
;;
Wherein,AndThe gradients in x-direction and y-direction for the scale m, respectively, I being the brightness of the image, whereby the gradient descriptor of the heterologous image is expressed as:
;
s303, setting a statistical scale as m, and taking (x, y) pixels as the center, wherein a gradient histogram of 10×10 pixels around the center is expressed as follows:
;
s304, taking the image target size as a reference, taking M/2 scales upwards and M/2 scales downwards, wherein the M scales are altogether. The m-th dimensionPhase consistency descriptor belowGradient descriptor (gradient amplitude)Gradient histogramConstitutes a descriptor vector at the mth scale:
;
The infrared image and the SAR image are operated according to the method, and each characteristic point in the obtained infrared image and SAR image respectively generates a rich multi-scale descriptor vectorAndThe descriptor effectively describes local structural information of the infrared image and the SAR image, and provides a solid foundation for feature matching.
S305, matching by adopting a method of weighting correlation distances (Weighted Correlation Distance, WCD), wherein the expression is as follows:
;
Wherein WCD represents a weighted correlation distance, namely a weight sum of differences between the infrared image and the SAR image descriptor vector from scale 1 to scale M, M is the scale of the descriptor (the value range of M is from 1 to M), M represents the number of sizes (the value is 10 in general); AndA multi-scale descriptor vector representing each feature point in the infrared image and the SAR image, respectively; Is the weight of the m-th scale, expressed as:; is the descriptor vector of the mth scaleAndIs a variance of (2);
When the WCD value is greater than 0.5, it is consideredAndAnd storing the projection relation of each pixel coordinate in the infrared image and the SAR image as a matrix C.
The principle is briefly described that after the step S2, the resolution of the infrared ground pixel is basically consistent with the resolution of the SAR ground pixel in theory, but the resolution of the infrared ground pixel and the resolution of the SAR ground pixel have slight differences due to the calibration error of the field angle in the step S1 and the focus control error in the step S2. A registration operation is required. The calculation of the universal feature descriptors of the infrared image and the SAR image is a key step in the matching of the infrared image and the SAR image, and provides a unique vector representation for each feature point of the infrared image and the SAR image.
The Weighted Correlation Distance (WCD) approach takes into account not only the euclidean distance between feature vectors, but also the correlation between feature dimensions. This approach is particularly applicable where there is an inherent relationship between those feature dimensions.
Because the infrared SAR adopts an external triggering mode, the infrared SAR is exposed at the same time, and after exposure, a new frame of infrared image and SAR image reach an airborne computer, and the real-time registration of the infrared SAR can be completed only by calling the matrix C, so that the real-time performance of the registration is effectively ensured.
S4, image fusion, namely converting an infrared image format from RGB to YUV according to the registered image, superposing corresponding brightness values of SAR images on a Y channel, calculating local contrast and information entropy of the infrared image and the SAR images, and dynamically adjusting fusion proportion of the infrared image and the SAR images to generate a registered fusion image, wherein the method specifically comprises the following sub-steps:
S401, calculating local contrast of an infrared image and an SAR image, carrying out normalization processing on the local contrast of the infrared image and the SAR image to obtain weight of the local contrast of the infrared image and the SAR image, and specifically comprising the following sub-steps:
S4011, determining a local area taking the feature as the center, applying an average filter to the local area to obtain local average brightness, calculating standard deviation of pixel values in the local area and local average brightness deviation, and taking the maximum value of the standard deviation as local contrast.
Local standard deviationIs a local areaThe measurement of the fluctuation of the inner pixel value, and the calculation formulas of the local standard deviation of the infrared image and the SAR image are respectively as follows:
;
;
In the formula,Is the pixel value of the infrared image at coordinates (x, y),Is the pixel value of the SAR image at coordinates (x, y),Is the average pixel value of the local region qf in the infrared image or SAR image, |qf| represents the total number of pixels of the local region qf in the infrared image or SAR image.
The local standard deviation is obtained by summing the squares of the deviations of the pixel values from their mean value uf in the local area q f and taking the square root. This value reflects the degree of dispersion of pixel intensities within the feature region and is an indicator of the complexity of the local texture.
The local contrast of an infrared image or SAR image is defined as the maximum of its local standard deviation:
;
;
S4012, carrying out normalization processing on local contrast of the infrared image and the SAR image to enable the sum of IRLocalContrast and SARLocalContrast to be 1, and obtaining weight of the local contrast of the infrared image and the SAR image:
;
In the formula,Representing the infrared image weights based on local contrast,The SAR image weights based on local contrast are represented.
S402, calculating information entropy of an infrared image and information entropy of an SAR image, and carrying out normalization processing on the information entropy of the infrared image and the information entropy of the SAR image to obtain weights of the information entropy of the infrared image and the information entropy of the SAR image, wherein the method specifically comprises the following sub-steps:
S4021, representing the number of times of occurrence of gray value i in an infrared image or SAR image by using a histogram, wherein the gray value of each pixel point in the gray imageOr (b)The following operations are performed:
;
In the formula,AndRepresenting an infrared histogram and a SAR histogram; i represents a gray value; Representing indication function whenAt the time of being equal to i,1, Otherwise 0, whenAt the time of being equal to i,1, Otherwise 0; representing the gray value of a pixel point with coordinates (x, y) in the infrared image,The gray value of the pixel point with coordinates (x, y) in the SAR image is represented.
S4022, carrying out normalization processing, converting the histogram into probability distribution, and dividing the frequency of each gray value by the total pixel number of the image:
;
;
In the formula,AndThe probability distribution for each gray value of the infrared image and the SAR image respectively,AndThe width and height of the infrared image respectively,AndThe width and height of the SAR image, respectively.
S4023, multiplying the probability of each gray value by the logarithm based on 2 by using a shannon information entropy formula, accumulating all 256 possible gray values, and calculating an infrared image information entropy Hir and an SAR image information entropy HSAR;
;
;
The information entropy ranges from 0 to a maximum of 8 bits, which reflects the amount of information of the image from completely unordered (the gray values of each pixel are random and the probability is equal) to completely ordered (all pixels are the same gray values), hir=0, hsar=0 when all pixels have the same gray values (i.e. the image is completely uniform), and information entropy hir=8, hsar=8, reaches a maximum when the probability of each gray value occurrence is equal.
S4024, carrying out normalization processing on information entropy of the infrared image and the SAR image so that the sum of Hir and HSAR is 1, and obtaining weight of the information entropy of the infrared image and the SAR image:
;
In the formula,Representing the infrared image weights based on the entropy of the information,And represents SAR image weights based on information entropy.
S403, converting an RGB format file of an infrared image into YUV, superposing a Y-channel brightness value Yir of the infrared image and a Y-channel brightness value YSAR of an SAR image according to a matrix C, fusing pixel by pixel, and generating a registration fusion image Yfused, wherein the expression is as follows:
;
Wherein Yfused represents a registration fusion image, yir represents a Y-channel brightness value of an infrared image, and YSAR represents a Y-channel brightness value of an SAR image; Representing the infrared image weights based on local contrast,Representing SAR image weights based on local contrast; representing the infrared image weights based on the entropy of the information,And represents SAR image weights based on information entropy.
In summary, through a simplified ground calibration process, only the field angle and imaging parameters of an infrared camera and SAR are required to be calibrated, a registration algorithm is operated on an onboard computer after the zoom or the parameters are changed, a heterogeneous image descriptor is extracted, the descriptors of the infrared image and the SAR image are matched, and the projection relation of each pixel coordinate of the infrared image and the SAR is stored as a matrix C. In the case of a constant focal length or parameter configuration, the matrix C needs to be calculated only once and stored in an on-board computer.
Because the infrared and SAR adopt a synchronous triggering mode, the infrared and SAR acquire data at the same moment, after the data are acquired, a new frame of infrared image and SAR image arrive at the airborne computer at the same time, and the real-time registration of the infrared and SAR can be completed only by calling the matrix C, so that the real-time performance of the registration is effectively ensured. After each frame of image is registered through a matrix C, converting an infrared video format from RGB to YUV, superposing corresponding brightness values of SAR on a Y channel, calculating local contrast and information entropy of infrared and SAR, and dynamically adjusting fusion proportion of the infrared Y and SAR brightness values according to the local contrast and the information entropy to maximize information content of the fused image.
The method is suitable for image registration under the conditions that the infrared cameras and SAR are of different resolutions and are of variable focal length or variable parameter configuration, is particularly suitable for the fields of unmanned aerial vehicle-mounted photoelectric radar comprehensive load variable focal length infrared camera and SAR image registration and the like, has wide application prospect and practical application value, and remarkably reduces the workload of registration calibration.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.