Disclosure of Invention
The present invention is directed to solving the above-mentioned problems, and provides a method for stably tracking a target in combination with ground speed compensation.
The invention aims to provide a method for stably tracking a target by combining ground speed compensation, which specifically comprises the following steps:
S1, measuring the flight speed and the flight direction of an unmanned aerial vehicle through an unmanned aerial vehicle-mounted IMU, and determining the ground speed;
S2, mounting a visible light camera and an infrared camera on an unmanned aerial vehicle-mounted photoelectric turret, capturing a visible light image and an infrared image by using the unmanned aerial vehicle-mounted photoelectric turret, and performing real-time registration fusion by using an image registration algorithm;
s3, extracting local contrast and information entropy according to the registered and fused images, dynamically adjusting the fusion proportion of the visible light image and the infrared image through a fusion tracking method to generate a fusion tracking image, inputting the fusion tracking image into a related tracking algorithm to track a target, and carrying out target search area offset according to the parameters of ground speed compensation to ensure that the target is searched with maximum probability;
s4, the servo system inputs the ground speed compensation parameter into a speed closed-loop control loop, and stably tracks the target by combining the off-target quantity of the related tracking.
Preferably, the angle parameters in step S1 include azimuth angle α and pitch angle β;
the calculation method of the ground speed compensation value specifically comprises the following steps:
s101, defining the east speed of the unmanned aerial vehicle by the azimuth angle alphaAnd north speedThe east speed of the unmanned aerial vehicleAnd north speedConversion into velocity components relative to the path of flight of the unmanned aerial vehicle:
;
Where, alpha represents the azimuth angle on the flight path of the unmanned aerial vehicle,Indicating the speed of the east direction,Indicating the north speed;
s102, adjusting a speed component according to a pitch angle beta on a flight path of the unmanned aerial vehicleThe adjusted horizontal velocity component is obtained, and the calculation formula is as follows:
;
In the formula,According to the east speed of the unmanned planeAnd north speedA horizontal velocity component calculated from the azimuth angle alpha and the pitch angle beta; Beta represents a pitch angle on the flight path of the unmanned aerial vehicle;
s103, calculating a ground speed compensation value, wherein the calculation formula is as follows:
;
In the formula,Representing a ground speed compensation value; Is a proportionality constant for adjusting the compensation strength; representing a horizontal velocity component.
Preferably, the step S2 specifically includes the following sub-steps:
S201, ground calibration, namely measuring and calibrating the visual field angle of the visible light in the Y direction under each focal length of the visible light camera and the visual field angle of the infrared light in the Y direction under each focal length of the infrared camera on the ground, recording the relation between the visual field angle and the focal length as a file, and storing the file on a camera control board;
S202, roughly matching the angle of view, namely controlling the movement of a visible light camera or an infrared camera by a camera control board through a PID algorithm to enable the angle of view of a single pixel of a visible light image to be equal to the angle of view of a single pixel of an infrared image;
s203, extracting and registering image descriptors, namely extracting heterogeneous image descriptors of the visible light image and the infrared image, matching the descriptors, storing the projection relation of each pixel coordinate of the visible light image and each pixel coordinate of the infrared image as a matrix A, and calling the matrix A to finish registration.
Preferably, the heterogeneous image descriptors of the visible light image and the infrared image in the step S203 comprise phase consistency descriptors and gradient descriptors, and the matching is performed by adopting a method of weighting the correlation distance, and the specific method is as follows:
S2031, calculating a phase consistency descriptor of the heterologous image, wherein the phase consistency descriptor is used for measuring the phase of a specific frequency component in the heterologous image, and for each pixel, the phase can be calculated by the following formula:
;
wherein, theIs the pixel coordinates, w is a different scale,Is the phase at the scale w, M represents the number of dimensions;
s2032, calculating gradient descriptors of the heterologous images;
gradient direction descriptors of the heterologous image are as follows:
;;
wherein, theAndThe gradients in x-direction and y-direction for the scale m, respectively, I being the brightness of the image, whereby the gradient descriptor of the heterologous image is expressed as:
;
In the formula,A direction representing the gradient, i.e. a direction representing the brightness variation of the image at the point (x, y);
S2033. setting a statistical scale to m, and taking (x, y) pixels as the center, and a gradient histogram of 10×10 pixels around is expressed as follows:
;
s2034 taking the image target size as a reference, respectively taking M/2 scales upwards and downwards, and taking M scales in total, wherein the phase consistency descriptor under each scaleGradient descriptorGradient histogramForming descriptor vectors under M scales;
S2035, matching by adopting a weighted correlation distance method, wherein the expression is as follows:
;
wherein WCD represents a weighted correlation distance, i.e., a sum of weights of differences of the visible light image and the infrared image descriptor vector from scale 1 to scale M; AndRepresenting multi-scale descriptor vectors at an mth scale in the visible light image and the infrared image, respectively; Is the weight of the m-th scale, expressed as:; Is the descriptor vector of the mth scaleAndIs a variance of (2);
When the WCD value is greater than 0.5, it is consideredAndAnd storing the projection relation of each pixel coordinate in the visible light image and the infrared image as a matrix A.
Preferably, the method of adjusting the angle of view of the single pixel of the visible light image to be equal to the angle of view of the single pixel of the infrared image in step S202 is as follows:
When the unmanned aerial vehicle executes a task, if the unmanned aerial vehicle takes a visible lens as a main lens, tracking infrared is fused, the following steps are executed, namely the visible lens is taken as a main lens, when a ground station sends a large-view-field small-view-field instruction, only the visible lens responds, after the ground station is controlled, the visible lens is in a static state, at the moment, the focal length value gCCD of visible light is read out, the infrared focal length of a single pixel under the condition of the same view field is calculated according to the focal length value gCCD of the visible light, the size of a visible pixel and the size of an infrared pixel, and a camera control board controls the repeated motion of the infrared camera through a PID algorithm until the focal length of the visible light is 1/6 of the infrared focal length;
When the unmanned aerial vehicle executes a task, if the unmanned aerial vehicle takes the infrared lens as a main lens, the infrared lens is used as a main lens, when a ground station sends a large-view-field small-view-field instruction, only the infrared lens responds, after the ground station is controlled, the infrared lens is in a static state, at the moment, the focal length value gIR of the infrared lens is read out, according to the focal length value gIR of the infrared lens, the visible light focal length of a single pixel under the condition of the same view field is calculated according to the size of the visible pixel and the size of the infrared pixel, and a camera control board controls the repeated movement of the visible light camera through a PID algorithm until the visible light focal length is 1/6 of the infrared focal length.
Preferably, the fusion tracking method in step S3 specifically includes the following sub-steps:
s301, calculating local contrast of the visible light image and the infrared image, normalizing the local contrast of the visible light image and the infrared image to obtain weight of the local contrast of the visible light image and the infrared image;
S302, calculating information entropy of the visible light image and information entropy of the infrared image, and carrying out normalization processing on the information entropy of the visible light image and the information entropy of the infrared image to obtain weights of the information entropy of the visible light image and the information entropy of the infrared image;
S303, converting an RGB format file of a visible light image into YUV, superposing a Y-channel brightness value Yccd of the visible light image and a Y-channel brightness value Yir of an infrared image according to a matrix A, carrying out fusion tracking pixel by pixel, and generating a fusion tracking image Yfused, wherein the expression is as follows:
;
Wherein Yfused represents a fusion tracking image, yccd represents a Y-channel brightness value of a visible light image, and Yir represents a Y-channel brightness value of an infrared image; representing visible light image weights based on local contrast,Representing the infrared image weights based on local contrast; represents visible light image weights based on information entropy,Representing the infrared image weight based on the information entropy.
Preferably, the step S301 specifically includes the following sub-steps:
s3011, determining a local area taking a feature as a center, applying a mean filter to the local area to obtain local average brightness, calculating standard deviation of pixel values in the local area and local average brightness deviation, and taking the maximum value of the standard deviation as local contrast;
the calculation formulas of the local standard deviation of the visible light image and the infrared image are respectively as follows:
;
;
In the formula,Is the pixel value of the visible image at coordinates (x, y),Is the pixel value of the infrared image at coordinates (x, y),Is the average pixel value of the local area qf in the visible or infrared image, |Ω f| represents the total number of pixels of the local area qf in the visible or infrared image;
the local contrast expression of the visible or infrared image is as follows:
CCDLocalContrast=max();
IRLocalContrast=max();
S3012, carrying out normalization processing on local contrast of the visible light image and the infrared image to enable the sum of CCDLocalContrast and IRLocalContrast to be 1, and obtaining weight of the local contrast of the visible light image and the infrared image:
;
In the formula,Representing visible light image weights based on local contrast,Representing the infrared image weights based on local contrast;
The step S302 specifically includes the following sub-steps:
s3021, representing the number of times a gray value i appears in a visible light image or an infrared image by using a histogram, wherein the gray value is for each pixel point in the gray imageOr (b)The following operations are performed:
;
In the formula,AndRepresenting a visible histogram and an infrared histogram; i represents a gray value; Representing indication function whenWhen (1);Representing the gray value of a pixel point with coordinates (x, y) in the visible image,Gray values of pixel points with coordinates of (x, y) in the infrared image are represented;
s3022, carrying out normalization processing, converting the histogram into probability distribution, and dividing the frequency of each gray value by the total pixel number of the image:
;
;
In the formula,AndThe probability distribution for each gray value of the visible image and the infrared image respectively,AndThe width and height of the visible light image respectively,AndThe width and the height of the infrared image are respectively;
S3023, multiplying the probability of each gray value by the logarithm based on 2 by using a shannon information entropy formula, accumulating all 256 possible gray values, and calculating a visible light image information entropy Hccd and an infrared image information entropy Hir;
;
;
S3024, carrying out normalization processing on information entropy of the visible light image and the infrared image so that the sum of Hccd and Hir is 1, and obtaining weight of the information entropy of the visible light image and the information entropy of the infrared image:
;
In the formula,Represents visible light image weights based on information entropy,Representing the infrared image weight based on the information entropy.
Preferably, the correlation tracking algorithm in step S3 specifically includes the following steps:
s304, initializing a target area, namely determining an initial area of a target in the fused tracking image through a target detection algorithm, wherein the initial area is used as a starting point of the tracking algorithm;
S305, calculating template matching and correlation, namely searching for a target by calculating the correlation between the template and each possible position in the image by using an initial area of the target as the template, wherein the correlation can be calculated by the following formula:
;
wherein R (x, y) is a correlation score at the fused image location (x, y), T (I, j) is the pixel value of the fused template image, I (x+i, y+j) is the pixel value of the target fused image, and μT and μI are the average of the template and the target region, respectively;
s306, peak detection and target positioning, namely determining a new position of the target by searching a local maximum value in the correlation diagram, and if the peak value is higher than a preset threshold value, considering that the target is successfully tracked at the position, and updating a model of the target according to a tracking result.
Preferably, the method for performing the target search area offset according to the parameters of the ground speed compensation in the step S3 specifically includes the following steps:
s307, calculating the distance delta Poffset of the unmanned aerial vehicle moving at the ground speed C in delta t time according to the ground speed compensation value C, wherein the expression is as follows:
S308, determining the center of an initial search area as a target position Pcurrent observed by the unmanned aerial vehicle at present, and moving the search area according to delta Poffset, wherein the center of a new search area is as follows:
;
S309, performing a related tracking algorithm in the new search area, searching for a target with maximum probability, and calculating the target miss distance delta Perror.
Preferably, the step S4 specifically includes the following sub-steps:
s401, the servo system comprises two stages of closed control loops, namely a speed loop and a tracking loop, wherein the speed loop receives the ground speed compensation value C to form a speed loop with the ground speed compensation value;
;
Wherein Delta Pspeed _error represents the angular rate error of the speed loop, namely the difference between the expected angular rate and the actual angular rate, C is a ground speed compensation value, kp_speed is the proportional gain of the speed loop, ki_speed is the integral gain of the speed loop, delta Pspeed _correction is the speed loop control quantity output for completing the speed loop closed loop;
S402, the error input of a tracking loop of the servo system is the off-target quantity delta Perror, and the direction of the photoelectric turret is adjusted according to the off-target quantity delta Perror so as to reduce deviation and realize stable tracking:
;
Where Δpcorrection is the tracking loop control quantity output for completing tracking loop closed loop to achieve accurate tracking of the target, Δperror represents the off-target quantity, and Kp, ki and Kd are the proportional, integral and differential gains, respectively, of the tracking loop of the servo control system.
Compared with the prior art, the invention has the following beneficial effects:
the ground speed compensation mechanism is introduced, so that the tracking stability and accuracy of the unmanned aerial vehicle to the target in a dynamic flight environment are obviously improved.
The image registration algorithm based on the multi-scale descriptor and the optimized search strategy realizes the real-time registration fusion of images and improves the real-time performance of target tracking.
The fusion tracking method combines local contrast and information entropy, dynamically adjusts the fusion proportion of images, maximizes the information quantity of the fused images, and enhances the accuracy and the robustness of target tracking.
The method is suitable for image registration under different resolution ratios and zoom conditions, reduces the workload of registration calibration, and improves the efficiency of image registration.
In conclusion, the visible infrared images are registered in real time and are dynamically fused, the fused images are rich in characteristics, target characteristics can be effectively extracted, target searching area deviation is carried out according to the ground speed compensation parameters, the reliability of a target tracking algorithm is effectively improved, the ground speed compensation parameters are input into a speed closed-loop control loop by a servo system, the targets can be stably tracked by combining with the off-target quantity, and the tracking performance of the unmanned aerial vehicle on the ground targets in a dynamic flight environment is effectively improved. The method is suitable for the visual infrared image registration under the conditions that the visual infrared cameras with different resolutions are the zoom lenses, is particularly suitable for the fields of the image registration of the unmanned aerial vehicle-mounted photoelectric turret variable focal length visual light camera and the variable focal length infrared camera, and the like, has wide application prospect and practical application value, and remarkably reduces the workload of registration calibration.
Detailed Description
Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, like modules are denoted by like reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, a detailed description thereof will not be repeated.
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not to be construed as limiting the invention.
Referring to fig. 1, the invention provides a method for stably tracking a target in combination with ground speed compensation, which specifically comprises the following steps:
S1, measuring the flight speed and the flight direction of an unmanned aerial vehicle through an unmanned aerial vehicle-mounted IMU, and determining the ground speed;
the ground speed compensation is to compensate the motion of the unmanned plane relative to the ground, namely, the speeds of the sky, the east and the north are converted into projections on the ground plane, and the calculation method of the ground speed compensation value specifically comprises the following steps:
Unmanned aerial vehicle has upward velocityEast speedAnd north speedAlpha is an included angle between the azimuth angle of the photoelectric turret and the flight path of the unmanned aerial vehicle, beta is the pitch angle of the photoelectric turret, and the ground speed compensation value is calculated according to the speed component and the angle;
S101, east speedAnd north speedConversion into a velocity component relative to the unmanned aerial vehicle flight path (defined by azimuth angle alpha):
;
Where, alpha represents the azimuth angle on the flight path of the unmanned aerial vehicle,Indicating the speed of the east direction,Indicating the north speed;
s102, adjusting a speed component according to a pitch angle beta on a flight path of the unmanned aerial vehicleThe adjusted horizontal velocity component is obtained by considering the inclination of the unmanned plane relative to the horizontal plane, and the calculation formula is as follows:
;
In the formula,According to the east speed of the unmanned planeAnd north speedThe horizontal velocity component obtained by calculating the azimuth angle alpha and the pitch angle beta of the photoelectric turret; Beta represents a pitch angle on the flight path of the unmanned aerial vehicle;
s103, calculating a ground speed compensation value, wherein the calculation formula is as follows:
;
In the formula,Representing a ground speed compensation value; Is a proportionality constant for adjusting the compensation strength; representing a horizontal velocity component.
The step S1 also comprises inputting the ground speed compensation value into a related tracking algorithm to predict the dynamic position change of the target.
S2, mounting a visible light camera and an infrared camera on an unmanned aerial vehicle-mounted photoelectric turret, capturing a visible light image and an infrared image by using the unmanned aerial vehicle-mounted photoelectric turret (see fig. 2 and 3), and performing real-time registration fusion by an image registration algorithm, wherein the method specifically comprises the following sub-steps:
S201, ground calibration, namely measuring and calibrating the visual field angle of the visible light in the Y direction under each focal length of the visible light camera and the visual field angle of the infrared light in the Y direction under each focal length of the infrared camera on the ground, recording the relation between the visual field angle and the focal length as a file, and storing the file on a camera control board;
The measurement calibration method specifically comprises gradually measuring the Y-direction angle of view of a visible light camera and the Y-direction angle of view of an infrared camera by adopting a foldback type visible infrared double-light common-view-field light pipe (the light pipe contains cross wires and the stepping length is 0.1 m);
The method comprises the steps of setting a focal length of a visible light camera lens to be CCD_F, locking a pitch angle of the photoelectric turret to be 0, aligning the upper side edge of the visible light camera to a cross wire of a light pipe, reading out the azimuth angle CCDup of the photoelectric turret, rotating the photoelectric turret to align the lower side edge of the visible light camera to the cross wire of the light pipe, reading out the azimuth angle CCDdown of the photoelectric turret, recording the corresponding relation between the focal length of the visible light camera and the angle of view to an array c when the focal length of the visible light camera is CCD_F, measuring the corresponding relation between the focal length of the visible light camera and the angle of view to an array d when the focal length of the visible light camera is CCD_Y= CCDup-CCDdown, and recording the corresponding relation between the focal length of the infrared camera and the angle of view to an array d and storing the infrared camera to an unmanned aerial vehicle-mounted photoelectric turret recorder.
S202, rough matching of the angle of view, namely controlling the motion of a visible light camera or an infrared camera by a camera control board through a PID algorithm to enable the angle of view of a single pixel of a visible light image to be equal to the angle of view of a single pixel of an infrared image, wherein the method comprises the following specific operations:
If each pixel of the visible light image and the infrared image has the same field angle, the focal length relationship between the visible lens and the infrared lens is deduced as follows:
Visible light pixel size: Infrared pixel size: The focal length of the visible lens is fvi, and the focal length of the infrared lens is fir;
the angular resolution θ can be expressed by the following equation θ=pixel size/focal length;
the visible light image angle resolution is θvis= dvis/fvis, and the infrared image angle resolution is;
To make θvis equal to θir, the following equation can be established:; I.e. in case of a visible light picture element size of 2.5 μm and an infrared picture element size of 15 μm, i.e. in case of a visible focal length of 1/6 of the infrared focal length, the angle of view of the individual picture elements of the visible light image is equal to the angle of view of the individual picture elements of the infrared image.
The visible light camera and the infrared camera adopt an external triggering mode, so that the exposure of the visible light and the infrared light at the same time is ensured.
The method for adjusting the angle of view of the single pixel of the visible light image to be equal to the angle of view of the single pixel of the infrared image, namely adjusting the visible focal length to be 1/6 of the infrared focal length, comprises the following steps:
When the unmanned aerial vehicle executes a task, if the unmanned aerial vehicle takes a visible lens as a main lens, tracking infrared is fused, the following steps are executed, namely the visible lens is taken as a main lens, when a ground station sends a large-view-field small-view-field instruction, only the visible lens responds, after the ground station is controlled, the visible lens is in a static state, at the moment, the focal length value gCCD of visible light is read out, the infrared focal length of a single pixel under the condition of the same view field is calculated according to the focal length value gCCD of the visible light, the size of a visible pixel and the size of an infrared pixel, and a camera control board controls the repeated motion of the infrared camera through a PID algorithm until the focal length of the visible light is 1/6 of the infrared focal length;
When the unmanned aerial vehicle executes a task, if the unmanned aerial vehicle takes the infrared lens as a main lens, the infrared lens is used as a main lens, when a ground station sends a large-view-field small-view-field instruction, only the infrared lens responds, after the ground station is controlled, the infrared lens is in a static state, at the moment, the focal length value gIR of the infrared lens is read out, according to the focal length value gIR of the infrared lens, the visible light focal length of a single pixel under the condition of the same view field is calculated according to the size of the visible pixel and the size of the infrared pixel, and a camera control board controls the repeated movement of the visible light camera through a PID algorithm until the visible light focal length is 1/6 of the infrared focal length.
S203, extracting and registering image descriptors, namely extracting heterogeneous image descriptors of a visible light image and an infrared image, matching the descriptors, and storing the projection relation of each pixel coordinate of the visible light image and the infrared image as a matrix A;
The heterogeneous image descriptors of the visible light image and the infrared image comprise phase consistency descriptors and gradient descriptors, the method for matching the descriptors is to extract universal characteristic descriptors of the visible light image and the infrared image by combining the phase consistency descriptors and the gradient descriptors and match the universal characteristic descriptors by adopting a method of weighting correlation distances (Weighted Correlation Distance, WCD), and the specific method is as follows:
S2031, calculating a phase consistency descriptor of the heterologous image, wherein the phase consistency descriptor is used for measuring the phase of a specific frequency component in the heterologous image, and for each pixel, the phase can be calculated by the following formula:
;
wherein, theIs the pixel coordinates, w is a different scale,The phase at the scale w, M represents the number of dimensions (typically a value of 10);
s2032, calculating gradient descriptors of the heterologous images;
gradient direction descriptors of the heterologous image are as follows:
;;
wherein, theAndThe gradients in x-direction and y-direction for the scale m, respectively, I being the brightness of the image, whereby the gradient descriptor of the heterologous image is expressed as:
;
In the formula,A direction representing the gradient, i.e. a direction representing the brightness variation of the image at the point (x, y);
S2033. setting a statistical scale to m, and taking (x, y) pixels as the center, and a gradient histogram of 10×10 pixels around is expressed as follows:
;
S2034, taking the image target size as a reference, taking M/2 scales upwards and M/2 scales downwards, wherein the M scales are altogether. The m-th dimensionPhase consistency descriptor belowGradient descriptor (gradient amplitude)Gradient histogramConstitutes a descriptor vector at the mth scale:
;
The visible light image and the infrared image are operated according to the method, and each characteristic point in the obtained visible light image and infrared image respectively generates a rich multi-scale descriptor vectorAndThe descriptor effectively describes local structural information of the visible image and the infrared image and provides a solid basis for feature matching.
S2035, matching by adopting a method of weighting correlation distances (Weighted Correlation Distance, WCD), wherein the expression is as follows:
;
wherein WCD represents a weighted correlation distance, i.e., a weighted sum of differences between the visible image and the infrared image descriptor vector at scale 1 to scale M, M is the scale of the descriptor (M is a value ranging from 1 to M), M represents the number of dimensions (typically 10); AndRepresenting multi-scale descriptor vectors at an mth scale in the visible light image and the infrared image, respectively; Is the weight of the m-th scale, expressed as:; Is the descriptor vector of the mth scaleAndIs a variance of (2);
When the WCD value is greater than 0.5, it is consideredAndAnd storing the projection relation of each pixel coordinate in the visible light image and the infrared image as a matrix A.
The principle is briefly described that after the step S202, the visual field of the visible infrared field is basically consistent in the Y direction in theory, but the registration operation is required because of the calibration error of the visual field angle in the step S1 and the focus control error in the step S202, and the visible infrared field still has slight difference. The calculation of the universal feature descriptors of the visible light image and the infrared image is a key step in the matching of the visible light image and the infrared image, and provides a unique vector representation for each feature point of the visible light image and the infrared image.
The Weighted Correlation Distance (WCD) approach takes into account not only the euclidean distance between feature vectors, but also the correlation between feature dimensions. This approach is particularly applicable where there is an inherent relationship between those feature dimensions.
Because the visible infrared adopts an external triggering mode, the exposure of the visible infrared at the same time is ensured, and after the exposure, a new frame of visible image and infrared image reach an onboard computer, and the real-time registration of the visible infrared can be completed only by calling the matrix A, so that the real-time performance of the registration is effectively ensured.
S3, extracting local contrast and information entropy according to the registered and fused images, dynamically adjusting the fusion proportion of the visible light image and the infrared image through a fusion tracking method to generate a fusion tracking image (figure 4), inputting the fusion tracking image into a related tracking algorithm to track a target, and carrying out target searching area offset according to the parameters of ground speed compensation to ensure that the target is searched with maximum probability;
The fusion tracking method specifically comprises the following steps:
S301, calculating local contrast of a visible light image and an infrared image, carrying out normalization processing on the local contrast of the visible light image and the infrared image to obtain weight of the local contrast of the visible light image and the infrared image, and specifically comprising the following sub-steps:
S3011, determining a local area taking the feature as the center, applying an average filter to the local area to obtain local average brightness, calculating the standard deviation of pixel values in the local area and local average brightness deviation, and taking the maximum value of the standard deviation as local contrast.
Local standard deviationIs a local areaThe measurement of the fluctuation of the inner pixel value, and the calculation formulas of the local standard deviation of the visible light image and the infrared image are respectively as follows:
;
;
In the formula,Is the pixel value of the visible image at coordinates (x, y),Is the pixel value of the infrared image at coordinates (x, y),Is the average pixel value of the local area qf in the visible or infrared image, |qf| represents the total number of pixels of the local area qf in the visible or infrared image.
The local standard deviation is obtained by summing the squares of the deviations of the pixel values from their mean value uf in the local area q f and taking the square root. This value reflects the degree of dispersion of pixel intensities within the feature region and is an indicator of the complexity of the local texture.
The local contrast of a visible or infrared image is defined as the maximum of its local standard deviation:
CCDLocalContrast=max();
IRLocalContrast=max();
S3012, carrying out normalization processing on local contrast of the visible light image and the infrared image to enable the sum of CCDLocalContrast and IRLocalContrast to be 1, and obtaining weight of the local contrast of the visible light image and the infrared image:
;
In the formula,Representing visible light image weights based on local contrast,Representing the infrared image weights based on local contrast.
S302, calculating information entropy of the visible light image and information entropy of the infrared image, and carrying out normalization processing on the information entropy of the visible light image and the information entropy of the infrared image to obtain weight of the information entropy of the visible light image and the information entropy of the infrared image, wherein the method specifically comprises the following sub-steps:
s3021, representing the number of times a gray value i appears in a visible light image or an infrared image by using a histogram, wherein the gray value is for each pixel point in the gray imageOr (b)The following operations are performed:
;
In the formula,AndRepresenting a visible histogram and an infrared histogram; i represents a gray value; Representing indication function whenWhen (1);Representing the gray value of a pixel point with coordinates (x, y) in the visible image,Gray values of pixel points with coordinates of (x, y) in the infrared image are represented;
s3022, carrying out normalization processing, converting the histogram into probability distribution, and dividing the frequency of each gray value by the total pixel number of the image:
;
;
In the formula,AndThe probability distribution for each gray value of the visible image and the infrared image respectively,AndThe width and height of the visible light image respectively,AndThe width and height of the infrared image, respectively.
S3023, multiplying the probability of each gray value by the logarithm based on 2 by using a shannon information entropy formula, accumulating all 256 possible gray values, and calculating a visible light image information entropy Hccd and an infrared image information entropy Hir;
;
;
The information entropy ranges from 0 to a maximum of 8 bits, which reflects the amount of information of the image from completely unordered (the gray values of each pixel are random and the probability is equal) to completely ordered (all pixels are the same gray values), hccd =0, hir=0 when all pixels have the same gray values (i.e. the image is completely uniform), and the information entropy Hccd =8, hir=8, reaches a maximum when the probability of each gray value occurrence is equal.
S3024, carrying out normalization processing on information entropy of the visible light image and the infrared image so that the sum of Hccd and Hir is 1, and obtaining weight of the information entropy of the visible light image and the information entropy of the infrared image:
;
In the formula,Represents visible light image weights based on information entropy,Representing the infrared image weight based on the information entropy.
S303, converting an RGB format file of a visible light image into YUV, superposing a Y-channel brightness value Yccd of the visible light image and a Y-channel brightness value Yir of an infrared image according to a matrix A, carrying out fusion tracking pixel by pixel, and generating a fusion tracking image Yfused, wherein the expression is as follows:
;
Wherein Yfused represents a fusion tracking image, yccd represents a Y-channel brightness value of a visible light image, and Yir represents a Y-channel brightness value of an infrared image; representing visible light image weights based on local contrast,Representing the infrared image weights based on local contrast; represents visible light image weights based on information entropy,Representing the infrared image weight based on the information entropy.
In step S3, the relevant tracking algorithm specifically includes the following steps:
s304, initializing a target area, namely determining an initial area of a target in the fused tracking image through a target detection algorithm, wherein the initial area is used as a starting point of the tracking algorithm;
S305, calculating template matching and correlation, namely searching for a target by calculating the correlation between the template and each possible position in the image by using an initial area of the target as the template, wherein the correlation can be calculated by the following formula:
;
Where R (x, y) is the correlation score at the fused image location (x, y), T (I, j) is the pixel value of the fused template image, I (x+i, y+j) is the pixel value of the target fused image, and μT and μI are the average of the template and target region, respectively.
Because the embedded airborne computer has limited computing capability, the searching of the whole frame of all areas cannot be performed, and the traditional method is to search the limited areas near the appearance position of the target of the previous frame.
S306, peak detection and target positioning, namely determining a new position of the target by searching a local maximum value in the correlation diagram, and if the peak value is higher than a preset threshold value, considering that the target is successfully tracked at the position, and updating a model of the target according to the tracking result, wherein the model comprises the shape, the size and the appearance characteristics of the target.
In step S3, the method for performing the target search area offset according to the parameters of the ground speed compensation specifically includes the following steps:
s307, calculating the distance delta Poffset of the unmanned aerial vehicle moving at the ground speed C in delta t time according to the ground speed compensation value C, wherein the expression is as follows:;
S308, determining the center of an initial search area as a target position Pcurrent observed by the unmanned aerial vehicle at present, and moving the search area according to delta Poffset, wherein the center of a new search area is as follows:
;
S309, performing a related tracking algorithm in the new search area, searching for a target with maximum probability, and calculating the target miss distance delta Perror.
S4, inputting the ground speed compensation parameter into a speed closed-loop control loop by the servo system, and stably tracking a target by combining the off-target quantity of related tracking, wherein the method specifically comprises the following sub-steps:
s401, the servo system comprises two stages of closed control loops, namely a speed loop and a tracking loop, wherein the speed loop receives the ground speed compensation value C to form a speed loop with the ground speed compensation value;
;
The method comprises the steps of determining the angular rate error of a speed loop, namely the difference between the expected angular rate and the actual angular rate, wherein the difference is measured through an angular rate gyroscope arranged in an optoelectronic turret, the angular rate gyroscope can only measure the change of the angular rate of the optoelectronic turret and cannot measure the change of the linear rate generated by the flight of an unmanned aerial vehicle, C is a ground speed compensation value, namely the change of the linear rate generated by the flight of the unmanned aerial vehicle, delta Pspeed _error+C is the angular rate of the optoelectronic turret+the linear rate generated by the flight of the unmanned aerial vehicle, the error of the speed loop of the optoelectronic turret in actual flight can be comprehensively reflected, the accuracy of the response of the speed loop can be ensured to be improved by adding the ground speed compensation value C to the error term Delta Pspeed _error, kp_speed is the proportional gain of the speed loop, ki_speed is the integral gain of the speed loop, and Delta Pspeed _error is the control quantity output of the speed loop and is used for completing the speed loop.
S402, the error input of a tracking loop of the servo system is the off-target quantity delta Perror, and the direction of the photoelectric turret is adjusted according to the off-target quantity delta Perror so as to reduce deviation and realize stable tracking:
;
Where Δpcorrection is the tracking loop control quantity output for completing tracking loop closed loop to achieve accurate tracking of the target, Δperror represents the off-target quantity, and Kp, ki and Kd are the proportional, integral and differential gains, respectively, of the tracking loop of the servo control system.
The method is characterized in that a ground speed compensation mechanism is introduced, the ground speed of the unmanned aerial vehicle is calculated in real time and is used as an input parameter of a tracking algorithm to predict the dynamic change of the target, and the stable tracking of the target is realized. The method comprises the steps of acquiring the flying speed and the flying direction of an unmanned aerial vehicle through a navigation system of the unmanned aerial vehicle, and calculating a ground speed vector according to the azimuth angle and the pitch angle of an onboard photoelectric turret, wherein the ground speed vector is then used for adjusting a related tracking algorithm, so that the target tracking system can accurately predict the change of the target position in the flying process of the unmanned aerial vehicle.
The invention also comprises an image registration algorithm based on the multi-scale descriptor and the optimized search strategy, and the algorithm can run on an onboard computer in real time, so that the accuracy of target tracking is further improved. In addition, a fusion tracking method is provided, and the method combines local contrast and information entropy, and dynamically adjusts the fusion proportion of the visible light image and the infrared image so as to maximize the information quantity of the fusion image.
The invention has the advantages that the invention is not only suitable for image registration under different resolution ratios and varifocal conditions, but also obviously reduces the workload of registration calibration and improves the efficiency of image registration. By means of the fusion tracking technology, accuracy and robustness of target tracking are enhanced, requirements of unmanned aerial vehicle real-time image processing are met, and the method has wide application prospects and practical application values.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.