Disclosure of Invention
In order to overcome the disadvantages of the prior art, the present invention provides an image exposure correction method and system based on artificial intelligence.
In order to achieve the above purpose, the invention adopts the following technical scheme, and an image exposure correction method based on artificial intelligence specifically comprises the following steps:
acquiring an image to be subjected to exposure correction and an image subjected to exposure correction;
image exposure correction neural network training: taking an image to be subjected to exposure correction as the input of an image exposure correction neural network, taking the image subjected to exposure correction as the output of the image exposure correction neural network, and carrying out neural network training;
the loss function formula included in the image exposure correction neural network training is as follows:
in the formula:
the number of pixels of the network input image,
representing network input image
The net output value of the individual pixels,
representing network input image
The image tag value of an individual pixel,
representing network input image
The adaptive exposure weight for each pixel,
in order to map the coefficients of the image,
is a multi-scale spectrum difference loss function;
and outputting the exposure corrected image after the neural network training is finished.
Further, the method for obtaining the adaptive exposure weight comprises the following steps: and converting each exposure image into a Lab color space through color space conversion, and normalizing a brightness channel in the Lab color space to obtain the self-adaptive exposure weight.
Further, the adaptive exposure weight expression:
in the formula:
represents the average value of the luminance of the pixels of the image to be corrected for exposure,
representing the image to be corrected for exposure
Luminance of each pixel and
the standard deviation of (a) is determined,
representing the image to be corrected for exposure
The luminance channel of each pixel normalizes the luminance value.
Further, the calculation formula of the multi-scale spectrum difference loss function is as follows:
in the formula:
the number of classes representing the resolution of the feature map,
is the number of channels of the feature map,
is shown as
In resolution of
The width of the feature map of each channel,
is shown as
In resolution of
The height of the profile of an individual channel,
is shown as
In resolution of
The characteristic diagram of each channel is shown,
is shown as
In resolution of
Exposure corrected images of the individual channels, Σ denotes performing a pixel summation operation on the difference spectrogram,
is a scale factor that is a function of,
is to perform a fast fourier transform on the image.
Further, the image exposure correction neural network includes a generator and a discriminator;
the generator is used for learning the data characteristics of the training set and generating similar data with the characteristics of the training set under the guidance of the discriminator;
the discriminator is used for distinguishing whether the input data is real or false data generated by the generator and feeding back the data to the generator.
Further, the generator is structured as an encoder-decoder; the input of the encoder is an image after image processing to be subjected to exposure correction, and the output is a characteristic diagram; the input of the decoder is the characteristic diagram output by the encoder, and the output is the characteristic diagram with different resolutions and the image after exposure correction.
Further, the image to be subjected to exposure correction is shot by a mobile phone, and the image subjected to exposure correction is shot by a digital camera.
An artificial intelligence based image exposure correction system comprising:
an input unit that inputs an acquired image to be subjected to exposure correction;
the processing unit is used for inputting the image to be subjected to exposure correction input by the input unit into trained image exposure correction neural network training for processing, and acquiring the image to be subjected to exposure correction after exposure correction;
the loss function formula included in the image exposure correction neural network training is as follows:
in the formula:
the number of pixels of the network input image,
representing network input image
The net output value of the individual pixels,
representing network input image
The image tag value of an individual pixel,
representing network input image
The adaptive exposure weight for each pixel,
in order to map the coefficients of the image,
is a multi-scale spectrum difference loss function;
and an output unit that outputs the exposure-corrected image processed by the processing unit.
The invention has the beneficial effects that:
1. the method of the present invention trains a neural network in an end-to-end manner using back propagation. Second, it is generic and can be added to any existing framework without additional overhead.
2. The invention calculates the frequency of different frequencies
So that the network can see artifacts at different scales, learning a scale-independent representation, which is an ideal feature in the wild image. Ultimately the loss function improves the performance of the exposure correction by reducing noise, blur and other impurities, such as color artifacts. Multi-rulerThe degree spectrum difference loss function can definitely guide the network to learn the real frequency component of the correct exposure image distribution and ignore the noise frequency of the poor exposure input.
3. The invention combines the depth map to obtain the plane in the image, and can provide plane information for the neural network, thereby eliminating the influence caused by different reflected light components of different objects.
4. The invention adopts a self-adaptive exposure weight method, can allocate larger weight to a dark area during image long exposure, allocate larger weight to a bright area during image short exposure, and finally obtain the exposure weight of each pixel position of each image, wherein the larger the weight is, the more the brightness of the position needs to be compensated, so that the network can better correct different exposure areas of the image.
Detailed Description
The invention is described in detail below with reference to the figures and examples.
In the description of the present invention, it is to be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience of description and simplicity of description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention.
The terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of the present invention, "a plurality" means two or more unless otherwise specified.
Example 1
The specific scene aimed by the invention is a mobile phone photographing link, and the image may be darkened, noisy and fuzzy due to the weak light condition in the photographing process.
In order to solve the above technical problem, the present invention provides an artificial intelligence based image exposure correction method, as shown in fig. 1. The method specifically comprises the following steps:
the first step is to acquire the image. Firstly, shooting by a mobile phone under a severe illumination condition to obtain an image to be subjected to exposure correction, and then collecting depth information of the shot image to be subjected to exposure correction by a depth camera to obtain a depth map. And secondly, acquiring an exposure-corrected image by using a digital camera, and using the exposure-corrected image as a label output by the neural network. Because digital cameras have a higher exposure setting range, it is possible to take a picture of better quality under poor lighting conditions.
And then constructing a neural network to realize the correction of image exposure.
The image exposure correction neural network is an anti-neural network for generating an exposure corrected image. The antagonistic neural network comprises a generator and a discriminator. The generator is used for reconstructing an image which accords with the real distribution of the training set data as much as possible under the guidance of the discriminator by learning the characteristics of the training set data, thereby generating similar data with the characteristics of the training set. The discriminator is used for extracting the characteristics of the image, the input of the discriminator is the image generated by the generator after exposure correction and the image shot by the digital camera for the same scene, and the discriminator is responsible for distinguishing whether the input image is real or false image generated by the generator and feeding back the input image to the generator. The two networks are alternately trained, and the capability is synchronously improved until the data generated by the generated network can be in a false state and reach a certain balance with the capability of the identified network.
The generator is in encoder-decoder structure and can adopt
The network model is equal, the input of the encoder isAnd outputting a fused image obtained by performing correlation (joint operation) on the RGB image to be subjected to exposure correction and the depth map as a feature map. Then inputting the feature maps into a decoder, outputting feature maps with different resolutions after fitting and up-sampling, wherein the empirical value of the resolution is 5, the last layer of the feature maps with different resolutions is 3 (the RGB images are three channels), and the final decoder outputs an image after exposure correction, and assuming that the input resolution of the image is 512 × 512, the generated 4 resolution images are respectively 32,64,128 and 256, and the original image is 512 × 512 resolution, and the total is 5.
The method comprises the following steps of constructing a loss function of a neural network, optimizing network parameters through the constructed loss function, reducing noise, blur and other impurities, improving the performance of exposure correction, wherein the constructed loss function adopts a multi-scale spectrum difference loss and exposure weighted mean square error loss function, and the formula is as follows:
in the formula:
the number of pixels of the network input image,
representing network input image
The net output value of the individual pixels,
representing network input image
The image tag value of an individual pixel,
representing a networkInput image to
The adaptive exposure weight for each pixel,
in order to map the coefficients of the image,
is a multi-scale spectral difference loss function.
The method for obtaining the self-adaptive exposure weight in the multi-scale spectrum difference loss and exposure weighting mean square error loss function comprises the following steps:
and carrying out exposure weight calculation on the brightness channel of each exposure image. Firstly, color space conversion is carried out on each exposure image, and the exposure image is converted into a Lab color space, wherein L is a brightness channel, and the following calculation is carried out on the brightness channel. The L component in Lab color space is used to represent the brightness of the pixel, and the value range is [0,100], which means from pure black to pure white, the L channel needs to be normalized.
The adaptive exposure weight is:
in the formula:
represents the average value of the luminance of the pixels of the image to be corrected for exposure,
representing the image to be corrected for exposure
Luminance of each pixel and
the standard deviation of (a) is determined,
representing the image to be corrected for exposure
The luminance channel of each pixel normalizes the luminance value.
By this formula, the luminance value is made closer to (1-
) The larger the weighted value of the pixel is, the closer to (1-
) The dark areas in the long-exposure image and the bright areas in the short-exposure image can be well highlighted by the formula. I.e. when the whole image is bright (long exposure), dark areas are given a greater weight; when the entire image is dark (short exposure), a weight of a bright area is assigned.
The exposure weight of each pixel position of each image is finally obtained, the larger the weight is, the more the brightness value of the position is compensated, and the obtained image is called an adaptive exposure weight map.
The process is to convert each exposure image into a Lab color space through a color space, and normalize a brightness channel in the Lab color space to obtain the self-adaptive exposure weight. The adaptive weight calculation method has the advantage that the dark areas can be assigned with larger weight when the image is exposed for a long time, and the bright areas can be assigned with larger weight when the image is exposed for a short time.
The calculation formula of the multi-scale spectrum difference loss function in the multi-scale spectrum difference loss and exposure weighted mean square error loss function is as follows:
in the formula:
the number of categories representing the resolution of the feature map, i.e. 5 resolution feature maps,
is the number of channels of the feature map, 3,
is shown as
In resolution of
The width of the feature map of each channel,
is shown as
In resolution of
The height of the profile of an individual channel,
is shown as
In resolution of
The characteristic diagram of each channel is shown,
is shown as
In resolution of
The different resolutions of the exposure-corrected images of the channels can be realized by adopting down-sampling, sigma represents the pixel summation operation of the spectrogram after the difference,
is a scale factor, since the feature size decreases by a factor of 2, the feature size is a scale factor
The image is subjected to fast Fourier transform, and the data is transformed to a uniform value range through logarithmic transformation.
Computing label images and predictive feature maps during neural network training
And then computing the label image
And predicting feature maps
Average of absolute differences. The proposed multi-scale spectral difference loss function can explicitly guide the network to learn the true frequency components of the distribution of correctly exposed images and ignore the noise frequency of poorly exposed inputs.
The multi-scale spectrum difference loss function has the following advantages: first, it is differentiable and therefore suitable for training neural networks in an end-to-end manner using back-propagation. Second, it is generic and can be added to any existing framework without additional overhead. Third, by calculating the resolution at different levels
The network can see artifacts at different scales, learning a scale-independent representation, which is an ideal feature in the image. This loss function improves the performance of exposure correction by reducing noise, blur, and other impurities.
Therefore, the neural network can be optimized according to the objective function through an optimization method such as a gradient descent method, and finally exposure correction of the image is achieved through a generator of the antagonistic neural network.
Example 2
The specific scene aimed by the invention is a mobile phone photographing link, and the image may be darkened, noisy and fuzzy due to the weak light condition in the photographing process.
As shown in fig. 2, the present invention provides an artificial intelligence based image exposure correction system, comprising:
the input unit is used for inputting the image shot by the camera of the mobile phone or the image stored in the memory of the mobile phone into the input unit;
the processing unit is used for inputting the image input by the input unit into trained image exposure correction neural network training for processing and acquiring an image after exposure correction obtained after processing;
and the output unit is used for displaying the image after exposure correction processed by the processing unit on a display screen of the mobile phone or storing the image into a memory of the mobile phone.
The above embodiments are merely illustrative of the present invention, and should not be construed as limiting the scope of the present invention, and all designs identical or similar to the present invention are within the scope of the present invention.