Summary of the invention
The embodiment of the present invention is designed to provide a kind of image interfusion method, device and readable storage medium storing program for executing, by doubleChannel convolutional neural networks obtain infrared and visible images blending images, calculation of the convolutional neural networks as deep learningMethod can automatically select characteristics of image, improve the unicity of feature extraction, avoid existing infrared image and melt with visible imagesThe defect of conjunction method.Specific technical solution is as follows:
In order to achieve the above objectives, the embodiment of the invention provides a kind of image interfusion methods, comprising:
Infrared image is registrated with visible images, the first image and the second image after being registrated, wherein instituteIt is that visible images are the visible images that state the first image, which be parts of images, second image in the infrared image,In parts of images；
The first image and second image are input in trained convolutional neural networks, by the convolutionClassification the first shot chart of output and the second shot chart after neural metwork training；
The respective pixel of first shot chart and second shot chart is compared, binary map is obtained；
Based on the binary map, the first image and second image, the first blending image is obtained；
The first structure similarity graph of the first image and first blending image is calculated, and calculates the second imageWith the second structural similarity figure of first blending image；
Obtain the disparity map of the first structure similarity graph and the second structural similarity figure；
Based on the disparity map, the first image and second image, the second blending image is obtained.
In a kind of implementation, the respective pixel to first shot chart and second shot chart comparesCompared with the step of obtaining binary map, comprising:
For the first pixel on first shot chart, judge whether the pixel value greater than the second pixel, whereinFirst pixel is any one pixel on first shot chart, and second pixel is second scorePixel corresponding with first pixel on figure；
If it is, the pixel value of third pixel is 1 in the binary map；Otherwise, the pixel value of third pixelIt is 0, wherein the third pixel is the pixel in the binary map with the first pixel corresponding position.
In a kind of implementation, first blending image embodies formula are as follows:
F1(x, y)=D1(x,y)A(x,y)+(1-D1(x,y)B(x,y))
Wherein, D1For binary map, A is the first image, and B is the second image, F1For the first blending image, x, y are to constitute pixelThe coordinate value of point.
In a kind of implementation, the difference for obtaining the first structure similarity graph and the second structural similarity figureThe step of different figure, comprising:
Obtain the difference of the first structure similarity graph and the second structural similarity figure；
Using the absolute value of the difference as the difference of the first structure similarity graph and the second structural similarity figureDifferent figure.
It is described to be based on the disparity map, the first image and second image in a kind of implementation, obtain secondThe step of blending image includes:
Based on target area, region unrelated with target in the disparity map is removed, obtains target's feature-extraction image；
According to the target's feature-extraction image, the first image and second image, the second blending image is obtained.
In a kind of implementation, second blending image embodies formula are as follows:
F2(x, y)=D2(x,y)A(x,y)+(1-D2(x,y)B(x,y))
Wherein, D2For target's feature-extraction image, A is the first image, and B is the second image, and x, y are the seat for constituting pixelScale value, F2For the second blending image.
By the binary map as decision diagram, initial fusion image is obtained using Weighted Fusion rule, is finally mentioned using SSIMThe notable figure for taking out target area, merges again, obtains final blending image；
In a kind of implementation, the training step of the convolutional neural networks, comprising:
The first quantity original image having a size of 32 × 32 is extracted from the first image set, and is added in the second image setSecond quantity visible images；
The original image and the visible images are converted into grayscale image, and the above gray level image is cut into 16 ×16 sub-block, as high resolution graphics image set；
Gaussian Blur processing is carried out to the first quantity original image that the first image is concentrated, and the second image is addedThe infrared light image for the second quantity concentrated, then first quantity is opened into original image and the second quantity infrared light imageIt is cut into 16 × 16 sub-block, as fuzzy graph image set.
Convolutional neural networks structure is trained on the fuzzy graph image set and high resolution graphics image set made.
In a kind of implementation, the convolutional neural networks are binary channels network, each channel is by 5 layers of convolutionNeural network is constituted, including 3 convolutional layers, and 1 maximum pond layer and 1 full articulamentum, last output layer are 1Softmax classifier.
In addition, the embodiment of the invention also provides a kind of image fusion devices, comprising:
Registration module, for being registrated to infrared image with visible images, the first image after being registrated andTwo images, wherein the first image is that parts of images, second image in the infrared image are that visible images areParts of images in the visible images；
Categorization module, for the first image and second image to be input to trained convolutional neural networksIn, classification the first shot chart of output and the second shot chart after convolutional neural networks training；
Comparison module is compared for the respective pixel to first shot chart and second shot chart, obtainsBinary map；
First Fusion Module obtains first and melts for being based on the binary map, the first image and second imageClose image；
Computing module, for calculating the first structure similarity graph of the first image Yu first blending image, withAnd calculate the second structural similarity figure of the second image and first blending image；
Module is obtained, for obtaining the disparity map of the first structure similarity graph and the second structural similarity figure；
Second Fusion Module obtains second and melts for being based on the disparity map, the first image and second imageClose image.
And a kind of readable storage medium storing program for executing is provided, and it is stored thereon with computer program, it is real when which is executed by processorThe step of incumbent item of image fusion method.
Using a kind of image interfusion method provided in an embodiment of the present invention, device and readable storage medium storing program for executing, pass through convolution mindInfrared and visible images blending images are obtained through network, characteristics of image is automatically selected, improves the unicity of feature extraction, keep awayThe defect of existing infrared image and visible light image fusion method is exempted from.For binary segmentation there is no completely by target area withBackground area is accurately divided, and the case where shade occurs so as to cause the blending image in later period, according to infrared and visible light source figureConspicuousness target area figure is obtained as the difference with the structural similarity of original fusion image, secondary fusion steps is taken to changeKind fused image quality, the fusion method based on conspicuousness can keep the integrality of prominent target area, and improve fusion figureThe visual quality of picture, so as to preferably serve subsequent image understanding and identification etc..
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, completeSite preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based onEmbodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every otherEmbodiment shall fall within the protection scope of the present invention.
It should be noted that in the image processing arts, the thermoradiation efficiency in infrared image with target is larger, and canLight-exposed image grayscale differs greatly even opposite；Infrared image background gray scale is low without apparent thermal sensation effect contrast, withVisible images are compared, and spectral information is lacked, but equally include detailed information.Therefore, only have when being merged to imageMore go the information for retaining original image that could further improve syncretizing effect.
Referring to Fig. 1, the embodiment of the invention provides a kind of image interfusion methods, include the following steps:
S101 is registrated infrared image with visible images, the first image and the second image after being registrated,In, the first image is that parts of images, second image in the infrared image are that visible images are described visibleParts of images in light image.
It should be noted that geometrical registration refer to different time, different-waveband, different remote sensor systems are obtained sameThe image (data) in one area, through geometric transformation make corresponding image points in position with the operation that is overlapped completely in orientation.SpecificallyGeometrical registration process is the prior art, and this will not be repeated here for the embodiment of the present invention.
It is understood that sliding window is the commonly used image processing tool in image procossing, specifically, sliding windowThe size of mouth can be 3*3,5*5 either 16*16 etc., and the embodiment of the present invention is not specifically limited herein.
Illustratively, by taking the first image as an example, the sliding window of 16*16 can be opened from first pixel in the upper left cornerBegin, as first central pixel point of 16*16 sliding window, then successively moves the 16*16 sliding window.So theThe chance of pixel centered on any one pixel in one image has, then and so on, for the second imageBe in this way, so any one central pixel point in the first image can be calculated according to this principle, and it is right in the second imageAnswer the structural similarity of central pixel point.
Sliding window is defined having a size of 16 × 16, step-length 1, the infrared image being registrated and visible images of input are distinguishedIt is done from left to right in infrared image and visible images, slide from top to bottom obtains the first image of infrared image sub-blockVA, as shown in Figure 2；The second image of visible images sub-block VB, as shown in Figure 3.
The first image and second image are input in trained convolutional neural networks, by institute by S102Classification exports the first shot chart and the second shot chart after stating convolutional neural networks training.
It should be noted that convolutional neural networks are a kind of depth feed forward-fuzzy controls in machine learning, have becomeIt is applied to image recognition to function.Convolutional neural networks are a kind of feedforward neural networks, and artificial neuron can respond surrounding cells,It can carry out large-scale image procossing, including convolutional layer and pond layer.
In a kind of implementation, the training step of the convolutional neural networks, comprising: from the first image set extract having a size of32 × 32 the first quantity original image, and the second quantity visible images being added in the second image set；By the originalBeginning image and the visible images are converted into grayscale image, and the above gray level image is cut into 16 × 16 sub-block, as heightResolution chart image set；Gaussian Blur processing is carried out to the first quantity original image that the first image is concentrated, and is added theThe infrared light image of the second quantity in two image sets, then first quantity original image and second quantity is infraredLight image is cut into 16 × 16 sub-block, as fuzzy graph image set.
Illustratively, 2000 original clear images having a size of 32 × 32 are extracted from Cifar-10 image set, and addedEnter 200 visible images in TNO_Image_Fusion_Datase image set, is then converted into grayscale image and image is completePortion is cut into 16 × 16 sub-block, as high resolution graphics image set；Secondly high to all being carried out from Cifar-10 image subblockThis Fuzzy Processing (since infrared light image background area is low compared with visible images resolution ratio), and TNO_Image_ is added200 infrared light images (sub-block for having been entirely cut into 16 × 16) in Fusion_Datase image set, as fuzzy graph image set.
Using binary channels network, each channel is made of 5 layers of convolutional neural networks, including 3 convolutional layers, and 1Maximum pond layer and 1 full articulamentum, last output layer are 1 softmax classifiers.Input picture block size is 16× 16, the convolution kernel size of convolutional layer is set as 3 × 3, and step-length is set as 1；Maximum pond layer convolution kernel size 2 × 2, step-length 2 swashFunction living is Relu.Momentum and weight decaying are set to 0.9 and 0.0005, learning rate 0.0001.
It is understood that the first image is input in trained convolutional neural networks, by the convolutionNeural network is trained each of the first image pixel, obtains the score to each pixel, thus rightAll pixels point in first image obtains the score of all pixels point after being trained, thus the first score after being trainedScheme SA, similarly, the corresponding second shot chart S of the second image can be obtainedB.Detailed process is shown in Figure 4, in convolutional Neural netNetwork exports the image after training after convolution twice, maximum pond, convolution sum connect entirely.
S103 is compared the respective pixel of first shot chart and second shot chart, obtains binary map.
Specifically, judging whether the pixel greater than the second pixel for the first pixel on first shot chartValue, wherein first pixel is any one pixel on first shot chart, and second pixel is describedPixel corresponding with first pixel on second shot chart；If it is, the third pixel in the binary mapPixel value is 1；Otherwise, the pixel value of third pixel be 0, wherein the third pixel be the binary map on it is describedThe pixel of first pixel corresponding position.
For binary map T, the first shot chart and the second shot chart are subjected to individual element comparison, if any one pixelPoint, position are (m, n), value SAPixel point value be greater than SBRespective pixel point value, then the pixel is corresponding in binary mapValue is 1 at position (m, n), conversely, then the pixel in the corresponding position of binary map obtains 0, shown in following formula, illustratively,Based on Fig. 2 and Fig. 3, the binary map obtained after through neural network shown in Fig. 4 is as shown in Figure 5.
The binary map of a target area and background area is thus obtained, wherein white area indicates infrared imageTarget area, black region is background area, which can be used as the decision diagram of image co-registration.
S104 is based on the binary map, the first image and second image, obtains the first blending image.
First image and the second image are weighted according to binary map can obtain initial fusion as a result, initial fusion purposeThat the background area of the target area of infrared image and high-resolution visible images is integrated into an image, based on Fig. 2,Fig. 3 and Fig. 5 obtains the first blending image as shown in FIG. 6.
In a kind of implementation, first blending image embodies formula are as follows:
F1(x, y)=D1(x,y)A(x,y)+(1-D1(x,y)B(x,y))
Wherein, D1For binary map, A is the first image, and B is the second image, F1For the first blending image, x, y are to constitute pixelThe coordinate value of point.
S105 calculates the first structure similarity graph of the first image and first blending image, and calculating theSecond structural similarity figure of two images and first blending image.
There are very strong relevance between infrared image and visible images pixel, there is a large amount of among these relevancesStructural information, image structure similarity SSIM (structural similarity index) are that one kind is used to assess image matterThe index of amount.From the perspective of image construction, structural information is defined as brightness and contrast by structural similarity index, with thisTo reflect the structural of objects in images.For two images C and D, then the similarity measure function of two images is defined as:
Wherein, μa, μbIt is the average gray of image C and D, σa, μbIt is the standard deviation of image C and D, σabIt is image C and DCovariance, C1, C2, C3It is minimum normal number, it is therefore an objective to unstable caused by when avoiding denominator close to 0.α, beta, gamma > 0 are to useTo adjust brightness, contrast, the weight of structure function.
Therefore, the first image A and the first blending image F is calculated1First structure similarity graph SAF, illustratively, it is based onFig. 2 and Fig. 6 obtains first structure similarity graph as shown in Figure 7, calculates the second image B and the first blending image F1The second knotStructure similarity graph SBF, the second structural similarity figure as shown in Figure 8 is obtained based on Fig. 3 and Fig. 6.
S106 obtains the disparity map of the first structure similarity graph and the second structural similarity figure.
In a kind of implementation, the difference for obtaining the first structure similarity graph and the second structural similarity figureThe step of different figure, comprising: obtain the difference of the first structure similarity graph and the second structural similarity figure；By the differenceDisparity map of the absolute value of value as the first structure similarity graph and the second structural similarity figure.Specifically, firstStructural similarity figure and the second structural similarity figure disparity map are as follows:
S=| SAF-SBF|
Wherein, first structure similarity graph SAF, the second structural similarity figure SBF, S is disparity map, illustratively, based on figureThe disparity map that 7 and Fig. 8 is obtained is as shown in Figure 9.
S107 is based on the disparity map, the first image and second image, obtains the second blending image.
Since the first blending image that initial fusion obtains completely does not divide target area accurately with background area,Cause the blending image in later period shade occur, therefore takes secondary fusion steps to improve fused image quality.
It is described to be based on the disparity map, the first image and second image in a kind of implementation, obtain secondThe step of blending image includes: to be removed region unrelated with target in the disparity map based on target area, obtained target signatureExtract image；According to the target's feature-extraction image, the first image and second image, the second fusion figure is obtainedPicture.
Illustratively, it is based on disparity map shown in Fig. 9, obtains target's feature-extraction image as shown in Figure 10.
In a kind of implementation, second blending image embodies formula are as follows:
F2(x, y)=D2(x,y)A(x,y)+(1-D2(x,y)B(x,y))
Wherein, D2For target's feature-extraction image, A is the first image, and B is the second image, and x, y are the seat for constituting pixelScale value, F2For the second blending image.
Regard secondary fusion as infrared image and visual image fusion based on conspicuousness Objective extraction.Disparity map SContain the salient region of infrared image.Using morphological images processing method, area unrelated with target in disparity map is removedDomain obtains target's feature-extraction figure, it is to be appreciated that target area is the target person extracted by infrared sensorTherefore infrared figure enhances the conspicuousness of target area, so as to improve the detailed information retained in blending image, such as schemeShown in 11, the second blending image based on Figure 10 and Fig. 2, Fig. 3 acquisition.
Using the thought of binary segmentation, obtain infrared merging figure with visible images by binary channels convolutional neural networksPicture, algorithm of the convolutional neural networks as deep learning, can automatically select characteristics of image, improve the unicity of feature extraction,Avoiding the defect of existing infrared image and visible light image fusion method, (majority needs that manual designs extract feature and feature mentionsTake it is single, be easily lost).Secondly, completely do not divide target area accurately with background area for binary segmentation, thusThe blending image in later period is caused the case where shade occur, according to infrared and visible light source image and original fusion image structureThe difference of similitude obtains conspicuousness target area figure, and secondary fusion steps is taken to improve fused image quality, based on aobviousThe fusion method of work property can keep the integrality of prominent target area, and improve the visual quality of blending image, so as to moreGood serves subsequent image understanding and identification etc..
In addition, the embodiment of the invention also provides a kind of image fusion devices, comprising:
Registration module, for being registrated to infrared image with visible images, the first image after being registrated andTwo images, wherein the first image is that parts of images, second image in the infrared image are that visible images areParts of images in the visible images；
Categorization module, for the first image and second image to be input to trained convolutional neural networksIn, classification the first shot chart of output and the second shot chart after convolutional neural networks training；
Comparison module is compared for the respective pixel to first shot chart and second shot chart, obtainsBinary map；
First Fusion Module obtains first and melts for being based on the binary map, the first image and second imageClose image；
Computing module, for calculating the first structure similarity graph of the first image Yu first blending image, withAnd calculate the second structural similarity figure of the second image and first blending image；
Module is obtained, for obtaining the disparity map of the first structure similarity graph and the second structural similarity figure；
Second Fusion Module obtains second and melts for being based on the disparity map, the first image and second imageClose image.
And a kind of readable storage medium storing program for executing is provided, and it is stored thereon with computer program, it is real when which is executed by processorThe step of incumbent item of image fusion method.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is allAny modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present inventionIt is interior.