Disclosure of Invention
The invention mainly aims to provide an image complement method based on a multi-stage complex neural network, which aims to solve the technical problem of lower quality of image complement results in the prior art.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
an image complement method based on a multi-stage complex neural network comprises the following steps:
A1, establishing a multi-stage complex neural network generator, wherein the generator comprises at least three stages, and each stage is processed by the following modules:
a one-stage network of input image I and mask M, converting the input to the complex domain through real-to-complex blocks (RCB), then extracting features through Complex Convolution Blocks (CCB), the intermediate feature storage being used for jump connection, the latter half converting the features back to the real domain through Complex Transpose Convolution Blocks (CTCB) and complex-to-real blocks (CRB), forming a preliminary complement image;
And a two-stage network for inputting the output image of one stage, directly carrying out image refinement on the output image of one stage without using jump connection, and further refining the image through a Complex Convolution Block (CCB), a Complex Transpose Convolution Block (CTCB) and a complex to real block (CRB).
A three-stage network for inputting two-stage output images, similar to one-stage, for final image refinement and complementation by real-to-complex blocks (RCB), complex Convolution Blocks (CCB), complex Transpose Convolution Blocks (CTCB), and complex-to-real blocks (CRB);
A2, performing countermeasure training on the multi-stage complex neural network generator and the simple real neural network discriminator by using a training set, and training out the multi-stage complex neural network by using images and masks in the training set, wherein the generator is responsible for repairing missing parts in the images, and the discriminator evaluates the consistency of the image characteristics output by the generator and the original drawings so as to optimize the complement quality of the generator;
And A3, image complement, namely utilizing the trained multi-stage complex neural network generator to complement the new input image and the mask, and outputting a complete complement image.
Further, in the three-stage network, the real-to-complex block respectively generates a real part and an imaginary part of a complex number of an input image through two paths, the paths use Conv3 convolution layers with convolution kernel size of 3 to extract features, then sequentially pass through a ReLU activation layer and a BatchNorm standardization layer, and are repeated twice in total, and after the complex-to-real block directly connects the real part and the complex number of the data, the real-to-complex block sequentially passes through Conv3 convolution layers, batchNorm standardization layers, a ReLU activation layer, a Conv3 convolution layer and a Tanh activation layer to restore to a real image.
Further, the complex convolution blocks in the one-stage and three-stage networks comprise complex convolution layers, complex activation functions and complex batch normalization layers, wherein the convolution kernel of the complex convolution layers has a size of 4, a step length of 2 and a filling of 1, and the complex convolution blocks have a downsampling function besides extracting features.
Further, the first half of the one-stage and three-stage network is composed of 8 complex convolution blocks, the second half includes 8 complex transposed convolution blocks, the parameters of which are the same as those of the complex convolution blocks, only the convolution is changed into transposed convolution, the up-sampling function is provided outside the extracted features, and the second half and the first half of the network are symmetrical.
Further, in the first half part of the two-stage network, the first complex convolution block has a size of 7, the step length is 1, features are extracted, the subsequent block has a size of 3, the step length is 2, the padding is 1, the two-stage network has feature extraction and up-sampling functions, the number of the two-stage network is 2, the middle part comprises 3 complex residual convolution blocks, the convolution blocks sequentially comprise a complex convolution layer, a complex activation function and a complex standardization layer, only features are extracted, and the second half part comprises complex transposition convolution blocks symmetrical to the first half part.
In step A2, the complete image and the mask of the training data set are used as the input of the network, the multi-level and multi-scale feature extraction is performed through the complex convolution block, then the features are restored to the original picture through the complex transpose convolution layer and the like, and the final network model is obtained through multiple times of training.
Further, in step A3, for each complete image, a corresponding mask is provided to represent the region to be complemented, and the regions are transmitted into the trained multi-stage complex neural network. And finally, outputting the completed picture by the network.
A computer readable storage medium storing a computer program which when executed by a processor implements the image complement method based on a multi-stage complex neural network.
A computer program product comprising a computer program which when executed by a processor implements the image complement method based on a multi-stage complex neural network.
In some embodiments of the invention, the method comprises the following steps:
The first step is to build a multi-stage Complex neural network consisting of a Complex transform Block (R-C), a Complex convolution Block (Complex-Valued ConvolutionBlock, CCB), a Complex transpose convolution Block (Complex-Valued Transposed ConvolutionBlock, CTCB), and a Complex residual convolution Block (Complex-Valued Convolution Residual Block, CCRB). The one-stage and three-stage networks extract feature information of various sizes through complex convolution blocks of the first half and transfer to the second half using a jump connection. Each complex transpose convolution block in the second half receives the characteristic information output by the previous module and corresponding to the first half, and further processes the characteristic information, so that the information of the bottom layer and the high layer is fully utilized. And the second step is to use the image and the mask in the training set to conduct the countermeasure training on the generator and the discriminator. In this process, the generator is responsible for patching missing parts in the image, while the arbiter evaluates the output quality of the generator, thereby guiding the generator to extract image features at multiple stages using a complex neural network, complement occlusion regions, and improve their details and quality. And thirdly, complementing the input image and the mask by using a trained generator to obtain an output complete image.
The first step specifically comprises the steps of establishing a network structure based on a multi-stage complex neural network, wherein all network heads and tails are real-to-complex blocks and complex-to-real blocks respectively. The real-to-complex block generates a real part and an imaginary part of a complex number through two paths, respectively, of an input image. The paths are each characterized using Conv3 layer with convolution kernel size 3 and then sequentially passed through the ReLU activation layer and BatchNorm normalization layer, repeating twice in total. The real part of the data is directly connected with the complex part by the complex to real number block, and then sequentially passes through Conv3, batchNorm, reLU, conv and the Tanh activation layer. The intermediate body of the first stage and the third stage network consists of complex convolution blocks and complex transposed convolution blocks. The complex convolution blocks are composed of complex convolution layers, complex activation functions and complex batch standardization layers, wherein the complex convolution layers have the convolution kernel size of 4, the step length of 2 and the step length of 1, and the complex convolution blocks have the downsampling function besides the feature extraction. The first half part of the network is composed of 8 complex convolution blocks, the second half part is composed of 8 complex transposed convolution blocks, the parameters of the first half part are the same as those of the complex convolution blocks, the convolution is changed into transposed convolution, and the up-sampling function is realized outside the extracted features. The back half of the network is symmetrical to the front half. In the first half of the second stage network body, the first complex convolution block is 7 in size and the step size is 1 to extract features. The subsequent block has the size of 3, the step length of 2 and the filling of 1, and has the functions of feature extraction and up-down sampling, and the number of the blocks is 2. The middle part is composed of 3 complex residual convolution blocks, which are sequentially composed of a complex convolution layer, a complex activation function and a complex standardization layer, and only the features are extracted. The latter half consists of complex transposed convolution blocks symmetrical to the former half.
The second step specifically comprises taking the complete image and the mask of the training data set as the input of the network, extracting multi-level and multi-scale features through a complex convolution block, restoring the features back to the original picture through a complex transposition convolution layer and the like, and obtaining a final network model through multiple training.
The third step specifically includes providing, for each complete image, a corresponding mask to represent the region to be complemented, both of which are passed together into the trained multi-stage complex neural network. And finally, outputting the completed complete picture by the network.
The invention has the following beneficial effects:
The invention provides an image complement method based on a multi-stage complex neural network, which can realize high-quality image complement effect. The method comprises the steps of firstly using a mode of combining a multi-stage network and a complex neural module in an image complement method, adding a real number-complex conversion block at the head and the tail of each stage of network, extracting multi-level and multi-scale features by using a complex convolution module in the first half, constructing the network by stacking complex transposed convolution blocks in the second half, and gradually restoring an image by using the extracted features. In addition, the multi-stage complex neural network provided by the invention can give consideration to rough repair and fine optimization of the image through the stage networks of different receptive fields, and is more suitable for image completion. The multi-level and multi-scale characteristics extracted by different receptive field modules are utilized, so that the characteristic extraction capability is better, and high-quality image complement results can be obtained. Furthermore, the method of the invention has fewer parameters with the same feature numbers.
Other advantages of embodiments of the present invention are further described below.
Detailed Description
The following describes embodiments of the present invention in detail. It should be emphasized that the following description is merely exemplary in nature and is in no way intended to limit the scope of the invention or its applications.
The image complement method mainly comprises the following steps of establishing a Multi-stage Complex neural network (MS-CN) based on a Complex neural network basic structure as an image complement generator, and taking a simple real neural network as a discriminator. The generator contains a three-phase network of Real-to-Complex conversion blocks (Real-ComplexBlock, R-C), complex convolution blocks (Complex-ValuedConvolutionBlock, CCB), complex transpose convolution blocks (Complex-ValuedTransposed ConvolutionBlock, CTCB), and Complex residual convolution blocks (Complex-ValuedConvolution Residual Block, CCRB). The method comprises the steps of generating an image complement preliminary result through a complex neural network of a large receptive field in one stage, refining the preliminary result through the complex neural network of a small receptive field in two stages, and refining the two-stage result through the complex neural network of the large receptive field in three stages. And secondly, forming countermeasure training in a generator and a discriminator by utilizing the data set, and pre-training a multi-stage complex neural network with reasonable weight. And thirdly, receiving the image and the mask as model input, and complementing the model by using a multi-stage complex neural network obtained by countermeasure training to obtain an output complete image.
Referring to fig. 1 to 3, an embodiment of the present invention provides an image complement method based on a multi-stage complex neural network, which includes the following steps:
A1, establishing a multi-stage complex neural network generator, wherein the generator comprises at least three stages, and each stage is processed by the following modules:
a one-stage network of input image I and mask M, converting the input to the complex domain through real-to-complex blocks (RCB), then extracting features through Complex Convolution Blocks (CCB), the intermediate feature storage being used for jump connection, the latter half converting the features back to the real domain through Complex Transpose Convolution Blocks (CTCB) and complex-to-real blocks (CRB), forming a preliminary complement image;
And a two-stage network for inputting the output image of one stage, directly carrying out image refinement on the output image of one stage without using jump connection, and further refining the image through a Complex Convolution Block (CCB), a Complex Transpose Convolution Block (CTCB) and a complex to real block (CRB).
A three-stage network for inputting two-stage output images, similar to one-stage, for final image refinement and complementation by real-to-complex blocks (RCB), complex Convolution Blocks (CCB), complex Transpose Convolution Blocks (CTCB), and complex-to-real blocks (CRB);
A2, performing countermeasure training on the multi-stage complex neural network generator and the simple real neural network discriminator by using a training set, and training out the multi-stage complex neural network by using images and masks in the training set, wherein the generator is responsible for repairing missing parts in the images, and the discriminator evaluates the consistency of the image characteristics output by the generator and the original drawings so as to optimize the complement quality of the generator;
And A3, image complement, namely utilizing the trained multi-stage complex neural network generator to complement the new input image and the mask, and outputting a complete complement image.
In step A1, the first half of the one-stage and three-stage networks converts the image and mask from the real number domain to the complex number domain by means of real number to complex number blocks (RCBs), then performs feature extraction by means of several CCBs, and stores intermediate features for jump connection. First, CTCB of the second half connects the output feature of the last step with the intermediate feature corresponding to the first half in the feature dimension, and then further restores the feature. The complex features are finally converted to images by complex to real blocks (CRBs).
The two-phase network needs to meet a low receptive field and therefore does not use a jump connection. The image refinement is only performed by passing the one-stage output image through the real-to-complex block, the complex convolution block, the complex transpose convolution block, and the complex-to-real block in sequence.
The task of the discriminator is mainly used for training, and has no direct relation with image complement, so that a simple real neural network is adopted, and the discriminator consists of a real convolution block. The function of this is to determine whether the output of the generator has the same characteristics as the original to provide a part of the optimization objective.
In a preferred embodiment, as shown in FIG. 2, step A1 specifically comprises creating a multi-stage complex neural network structure comprising a real-to-complex conversion block, a complex convolution block, a complex transpose convolution block, and a complex residual convolution block. Wherein the real-to-complex conversion blocks of the three-stage network are divided into two types, real-to-complex blocks and complex-to-real blocks. The real number to complex number block respectively generates a real number part and an imaginary number part of a complex number through two paths, the paths use Conv3 convolution layers with convolution kernel size of 3 to extract features, then sequentially pass through a ReLU activation layer and a BatchNorm standardization layer, and repeat twice in total, and after the complex number to real number block directly connects the real part and the complex number of data, the real number to real number block sequentially passes through Conv3 convolution layers, batchNorm standardization layers, a ReLU activation layer, a Conv3 convolution layer and a Tanh activation layer to restore to a real number image.
As shown in fig. 2, in the preferred embodiment, the complex convolution blocks in the one-stage and three-stage networks each include a complex convolution layer, a complex activation function and a complex batch normalization layer, wherein the complex convolution layer has a convolution kernel size of 4, a step size of 2, and a padding of 1, and has a downsampling function in addition to extracting features. Further, the first half of the one-stage and three-stage network is composed of 8 complex convolution blocks, the second half includes 8 complex transposed convolution blocks, the parameters of which are the same as those of the complex convolution blocks, only the convolution is changed into transposed convolution, the up-sampling function is provided outside the extracted features, and the second half and the first half of the network are symmetrical.
In the first half of the two-stage network, as shown in fig. 2, the first complex convolution block has a size of 7, a step size of 1, extracts features, the subsequent block has a size of 3, a step size of 2, and a fill of 1, has feature extraction and upsampling functions, and is 2 in number, the middle part comprises 3 complex residual convolution blocks, which sequentially comprise a complex convolution layer, a complex activation function, and a complex normalization layer, extracts features only, and the second half comprises a complex transpose convolution block symmetrical to the first half.
In step A2, the training set is used for performing countermeasure training on the generator and the discriminator, a multi-stage complex neural network is trained by using images and masks in the training set, and completion is completed through rough repair and fine optimization.
In step A2, as shown in fig. 1 and fig. 3, the complete image and the mask of the training data set are used as the input of the network, the multi-level and multi-scale feature extraction is performed through the complex convolution block, then the features are restored to the original picture through the complex transposed convolution layer and the like, and the final network model is obtained through multiple training.
In step A3, the input image and the mask are complemented by using the trained complex neural network, so as to obtain a complete image.
As shown in fig. 1 and 3, in step A3, for each complete image, a corresponding mask is provided to represent the region to be complemented, and the regions are transmitted into the trained multi-stage complex neural network together. And finally, outputting the completed picture by the network.
Specific embodiments of the present invention and examples of its algorithms are further described below.
An image complement method based on multi-stage complex neural network. A three-stage network structure is composed of real-to-complex blocks and complex-to-real blocks. The front and back parts of the network main body are respectively composed of 8 complex convolution blocks and complex transposed convolution blocks, and jump connection exists before and after the network to realize feature transfer. The two-stage network structure comprises the same part of the network head and tail as a three-stage network. The front and back parts of the network main body are respectively composed of 3 complex convolution blocks and complex transpose convolution blocks. The middle part of the network consists of 3 complex residual convolution blocks for further processing of the extracted features.
Real to complex block as shown in fig. 2 (light green arrow mark), the block includes two identical paths that produce real and imaginary parts of complex features, respectively. Each path allows the picture to sequentially pass through the real convolution layers Conv3, batchNorm standardization layer and ReLU activation layer with the convolution kernel size of 3, the step length of 1 and the padding of 1, and repeat twice. Complex to real block as shown in fig. 2 (light yellow arrow mark), the module connects the real part of the data with the complex, and then sequentially passes through a Conv3 convolution layer, batchNorm standardization layer, a ReLU activation layer, a Conv3 convolution layer and a Tanh activation layer.
Complex convolution block as shown in fig. 2 (light blue arrow mark), the block is composed of complex convolution layer, complex normalization layer and complex activation layer. The convolution kernel of a three-stage complex convolution layer has the size of 4, the step length of 2 and the filling of 1, and simultaneously has the feature extraction and downsampling functions. The complex convolution block structure of two stages is the same, but the first convolution kernel is 7 in size, 1 in step length and 0 in filling, the last two convolution kernels are 3 in size, 2 in step length and 1 in filling, and the feature extraction and downsampling functions are achieved.
A complex transpose convolution block, as shown in fig. 2 (pink arrow labeled), is composed of a complex transpose convolution layer, a complex normalization layer, and a complex activation layer. The convolution kernel in a three-stage network is 4 in size, 2 in step size, 1 filled in, and the block will accept as input the outputs of the previous block and the corresponding block in the first half of the network and process. In the two-stage network, the first 2 convolution kernels are 3 in size, 2 in step size, 1 in filling, 7 in size, 1 in step size and 0 in filling.
Complex residual convolution block as shown in fig. 2 (red-brown arrow mark), the block is composed of complex convolution layer, complex normalization layer and complex activation layer. The convolution kernel size is 3, the step size is 1, and the padding is 1. The module adds a residual addition operation during the calculation.
The network training part is that a first-stage network firstly utilizes the first half part of a complex neural network to extract multi-level and multi-scale characteristics of the image and transmits the multi-level and multi-scale characteristics to the corresponding second half part of the network through jump connection. Each module in the second half then roughly restores the approximate pattern of the image in combination with the features of the two paths. The two-three stage network further refines and supplements the detail part of the image on different receptive fields respectively. The method can combine the advantages of complex representation and different receptive field networks to realize good complementing effect on the image to be complemented. The operation flow of the network is as follows:
O=Net3(Net2(Net1(I,M),M),M)
where Neti represents the phase i network, I, M represents the input image and mask, respectively.
The network is ultimately trained with the loss function. The three-phase network has a loss function, where the Net1 loss functionLambdag takes on a value of 0.1.
Pixel lossIs defined as follows:
Where O1 is the output of the one-stage network, as indicated by dot product, the1 representation a 1-norm of the one-dimensional model, lambdah takes a value of 6.
Countering lossesWherein D represents a network of discriminators.
M1＝I⊙(1-M)+O1⊙M
The training loss of Net2 is defined as follows, where λtv、λper and λsty take values of 0.1, 0.05 and 120.
In the middle ofPixel loss, defined by Net2, is representedAnd so on.
In the middle ofDefinition is as follows, M2 defines the value of the corresponding coordinate in the image, by analogy with M1, M2 (i, j).
In the middle ofAnd (3) withFor two losses defined on VGG-16, the following formula is:
Where Fi (·) represents the feature map of the i-th layer (i ε {5,10,17 }) when data is input to the pretraining network VGG-16 for computation, and Gi(·)＝Fi(·)Fi(·)T.
Training loss of Net3 defines L3 analogized from L2.
Gradient optimization uses an Adam optimizer with self-adaptive learning rate, and for three sub-networks and discriminators of a generator, each sub-network and discriminators has a respective Adam optimizer, and optimal network parameters can be obtained through training.
And (3) the image complement is realized by inputting the original image and the corresponding mask into a trained multi-stage complex neural network model, so that a high-quality complete complement image can be obtained.
In summary, the present invention provides an image complement method based on a multi-stage complex neural network, which innovatively uses a combination mode of the multi-stage network and the complex neural module to complement images, adds a real-complex conversion block at the head and tail of each stage of network, extracts multi-level and multi-scale features by using a complex convolution module in the first half, constructs a network by stacking complex transpose convolution blocks in the second half, and gradually restores images by using the extracted features. By using networks of different receptive fields at different stages, the invention can realize rough patching and fine optimization of images at the same time. The multi-level and multi-scale characteristics extracted by different receptive field modules are utilized, so that the characteristic extraction capability is better, and high-quality image complement junction can be obtained. The method not only improves the quality of image completion, but also can use fewer parameters under the condition of the same feature number due to the efficient feature extraction capability, so that the model is more compact and efficient. Therefore, the invention has wide application prospect in the application fields of image enhancement, medical image processing and the like, and has important significance for promoting the development of the computer vision field due to the high-quality image complement effect.
The embodiments of the present invention also provide a storage medium storing a computer program which, when executed, performs at least the method as described above.
The embodiment of the invention also provides a control device which comprises a processor and a storage medium for storing a computer program, wherein the processor is used for executing at least the method when executing the computer program.
The embodiments of the present invention also provide a processor executing a computer program, at least performing the method as described above.
The storage medium may be implemented by any type of non-volatile storage device, or combination thereof. The nonvolatile Memory may be a Read Only Memory (ROM), a programmable Read Only Memory (PROM, programmable Read-Only Memory), an erasable programmable Read Only Memory (EPROM, erasable Programmable Read-Only Memory), an electrically erasable programmable Read Only Memory (EEPROM, ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory), a magnetic random access Memory (FRAM, ferromagnetic Random Access Memory), a Flash Memory (Flash Memory), a magnetic surface Memory, an optical disk, or a compact disk-Only Memory (CD-ROM, compact Disc Read-Only Memory), and the magnetic surface Memory may be a magnetic disk Memory or a tape Memory. The storage media described in embodiments of the present invention are intended to comprise, without being limited to, these and any other suitable types of memory.
In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be additional divisions of actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described as separate components may or may not be physically separate, and components displayed as units may or may not be physical units, may be located in one place, may be distributed on a plurality of network units, and may select some or all of the units according to actual needs to achieve the purpose of the embodiment.
In addition, each functional unit in each embodiment of the present invention may be integrated in one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of hardware plus a form of software functional unit.
It will be appreciated by those of ordinary skill in the art that implementing all or part of the steps of the above method embodiments may be implemented by hardware associated with program instructions, where the above program may be stored in a computer readable storage medium, where the program when executed performs the steps comprising the above method embodiments, where the above storage medium includes various media that may store program code, such as a removable storage device, a Read-Only Memory (ROM), a random access Memory (RAM, randomAccess Memory), a magnetic disk or an optical disk.
Or the above-described integrated units of the invention may be stored in a computer-readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solutions of the embodiments of the present invention may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. The storage medium includes various media capable of storing program codes such as a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk.
The methods disclosed in the method embodiments provided by the invention can be arbitrarily combined under the condition of no conflict to obtain a new method embodiment.
The features disclosed in the several product embodiments provided by the invention can be combined arbitrarily under the condition of no conflict to obtain new product embodiments.
The features disclosed in the embodiments of the method or the apparatus provided by the invention can be arbitrarily combined without conflict to obtain new embodiments of the method or the apparatus.
The foregoing is a further detailed description of the invention in connection with the preferred embodiments, and it is not intended that the invention be limited to the specific embodiments described. It will be apparent to those skilled in the art that several equivalent substitutions and obvious modifications can be made without departing from the spirit of the invention, and the same should be considered to be within the scope of the invention.