技术领域technical field
本发明涉及图像分辨率重建技术领域，特别是涉及一种基于卷积神经网络的图像超分辨率重建方法及系统。The present invention relates to the technical field of image resolution reconstruction, in particular to an image super-resolution reconstruction method and system based on a convolutional neural network.
背景技术Background technique
随着数字时代的到来，以图像为媒介的信息处理方式已经被广泛应用于各种场景中。图像分辨率的高低可以影响到信息获取的完善程度，高分辨率图像可以提供更多的数据信息。近年来，基于深度学习的多个网络模型在图像超分辨率重建领域的精度和计算性能方面相比于传统方法上有了很大的提升。但这些网络模型里低分辨率图像在预处理阶段通常会先使用双三次插值的方法放大为与目标图像同尺寸的高分辨率图像。这意味着在网络训练过程中需要在高分辨率图像空间中进行操作，导致网络计算复杂度的增加。虽然加深网络深度可以有效提升图像的重建精度，但仅增加网络层数又会导致梯度爆炸和梯度消失问题。With the advent of the digital age, image-based information processing methods have been widely used in various scenarios. The level of image resolution can affect the degree of perfection of information acquisition, and high-resolution images can provide more data information. In recent years, multiple network models based on deep learning have greatly improved the accuracy and computational performance in the field of image super-resolution reconstruction compared with traditional methods. However, the low-resolution images in these network models are usually enlarged into high-resolution images of the same size as the target image by using the bicubic interpolation method in the preprocessing stage. This means that operations in the high-resolution image space are required during network training, leading to an increase in the computational complexity of the network. Although deepening the network depth can effectively improve the image reconstruction accuracy, only increasing the number of network layers will lead to gradient explosion and gradient disappearance.
针对上述问题，近些年出现的方法大可归为三类。第一类是基于插值的方法，基于低分辨率图像中像素点利用各种类型的核函数去估计高分辨率图像中相邻区域内未知像素点的值，从而实现图像重建工作，典型算法包括最近邻插值、双线性插值及双立方插值。第二类是基于重建的方法，从多幅低分辨率图像中提取特征信息来建立常规模型，再通过模型恢复图像中丢失的高频信息，包括空间域和频域两类，常用的空域法包括非均匀插值法、凸集投影法、迭代反投影法和最大后验概率法。第三类是基于学习的方法，通过对低分辨率图像与高分辨率图像之间的映射模型进行不断训练优化，再将低分辨图像作为输入映射成高分辨率图像，主要包括基于机器学习的方法和基于深度学习的方法，其中典型基于机器学习的方法有基于样本学习、基于局部线性嵌入、支持向量回归、基于稀疏编码、基于锚点邻域回归及基于线性最小二乘函数等，而基于深度学习的方法包括卷积神经网络(SRCNN)、多通道的卷积神经网络、VGG卷积网络及生成对抗网络等。例如，SRCNN先利用双三次插值放大图像，再将预处理的图像输入传统卷积神经网络进行图像重建。SRCNN对于图像的重建效果有显著提升，且由于神经网络训练过程中参数会自适应调整，因此节省了大量的人工调参时间。In response to the above problems, the methods that have emerged in recent years can be classified into three categories. The first type is based on interpolation methods, based on the pixels in the low-resolution image, various types of kernel functions are used to estimate the value of unknown pixels in the adjacent area of the high-resolution image, so as to achieve image reconstruction. Typical algorithms include Nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. The second type is based on the reconstruction method, which extracts feature information from multiple low-resolution images to establish a conventional model, and then restores the lost high-frequency information in the image through the model, including two types of spatial domain and frequency domain. The commonly used spatial domain method Including non-uniform interpolation, convex set projection, iterative back-projection, and maximum a posteriori probability. The third category is based on learning methods, through continuous training and optimization of the mapping model between low-resolution images and high-resolution images, and then mapping low-resolution images into high-resolution images as input, mainly including machine learning-based methods and methods based on deep learning, in which typical machine learning-based methods are based on sample learning, based on local linear embedding, support vector regression, based on sparse coding, based on anchor neighborhood regression and based on linear least squares function, etc., and based on Deep learning methods include convolutional neural network (SRCNN), multi-channel convolutional neural network, VGG convolutional network, and generative confrontation network. For example, SRCNN first uses bicubic interpolation to enlarge the image, and then feeds the preprocessed image into the traditional convolutional neural network for image reconstruction. SRCNN has significantly improved the image reconstruction effect, and because the parameters will be adaptively adjusted during the neural network training process, it saves a lot of time for manual parameter adjustment.
虽然基于深度学习的重建方法可以通过加深网络层数的方式来提高算法重建性能，但随着网络层数的加深会出现梯度消失和梯度爆炸等问题，而且网络的加深也会导致计算量的加大，这些均会降低图像重建的质量。Although the reconstruction method based on deep learning can improve the reconstruction performance of the algorithm by deepening the number of network layers, problems such as gradient disappearance and gradient explosion will appear as the number of network layers deepens, and the deepening of the network will also lead to an increase in the amount of calculation. These will reduce the quality of image reconstruction.
发明内容Contents of the invention
本发明的目的是提供一种基于卷积神经网络的图像超分辨率重建方法及系统，能够提高图像的重建质量。The purpose of the present invention is to provide an image super-resolution reconstruction method and system based on a convolutional neural network, which can improve image reconstruction quality.
为实现上述目的，本发明提供了如下方案：To achieve the above object, the present invention provides the following scheme:
一种基于卷积神经网络的图像超分辨率重建方法，所述方法采用七个网络层，所述网络层包括第一网络层、第二网络层、第三网络层、第四网络层、第五网络层、第六网络层和第七网络层，包括：An image super-resolution reconstruction method based on a convolutional neural network, the method adopts seven network layers, and the network layers include a first network layer, a second network layer, a third network layer, a fourth network layer, and a network layer The fifth network layer, the sixth network layer and the seventh network layer, including:
获取待重建的图像；Obtain the image to be reconstructed;
对所述待重建的图像进行预处理，得到预处理后的图像；Preprocessing the image to be reconstructed to obtain a preprocessed image;
对所述预处理后的图像的特征进行提取，得到特征图像集；Extracting features of the preprocessed image to obtain a feature image set;
对所述特征图像集进行非线性映射，得到全局特征映射集；performing nonlinear mapping on the feature image set to obtain a global feature map set;
根据所述全局特征映射集进行重建，得到重建后的高分辨率图像。Reconstruction is performed according to the global feature map set to obtain a reconstructed high-resolution image.
可选的，所述对所述待重建的图像进行预处理，得到预处理后的图像，具体包括：Optionally, the preprocessing the image to be reconstructed to obtain the preprocessed image specifically includes:
对所述待重建的图像进行色彩空间转换，得到预处理后的亮度通道图像。Perform color space conversion on the image to be reconstructed to obtain a preprocessed brightness channel image.
可选的，所述对所述预处理后的图像的特征进行提取，得到特征图像集，具体包括：Optionally, the feature extraction of the preprocessed image to obtain a feature image set specifically includes:
将所述预处理后的图像作为所述第一网络层的输入，进行特征提取，得到第一层特征图像集；Using the preprocessed image as the input of the first network layer to perform feature extraction to obtain a first layer feature image set;
将所述第一层特征图像集作为所述第二网络层的输入，进行特征提取，得到第二层特征图像集；所述第一网络层和所述第二网络层为卷积层。The feature image set of the first layer is used as the input of the second network layer to perform feature extraction to obtain the feature image set of the second layer; the first network layer and the second network layer are convolutional layers.
可选的，所述对所述特征图像集进行非线性映射，得到全局特征映射集，具体包括：Optionally, performing nonlinear mapping on the feature image set to obtain a global feature map set specifically includes:
将所述第一层特征图像集和所述第二层特征图像集分别作为所述第三网络层的输入，进行映射操作，得到第三特征图像集；Using the feature image set of the first layer and the feature image set of the second layer as the input of the third network layer respectively, and performing a mapping operation to obtain a third feature image set;
将所述第三特征图像集作为所述第四网络层的输入，进行映射操作，得到第四特征图像集；Using the third feature image set as the input of the fourth network layer, performing a mapping operation to obtain a fourth feature image set;
将所述第三特征图像集和所述第四特征图像集作为所述第五网络层的输入，进行映射操作，得到第五特征图像集；Using the third feature image set and the fourth feature image set as the input of the fifth network layer, performing a mapping operation to obtain a fifth feature image set;
将所述第五特征图像集作为所述第六网络层的输入，进行映射操作，得到第六特征图像集，所述第六特征图像集为全局特征映射集；所述第三网络层、所述第四网络层、所述第五网络层和所述第六网络层为残差层。Using the fifth feature image set as the input of the sixth network layer, performing a mapping operation to obtain a sixth feature image set, the sixth feature image set is a global feature map set; the third network layer, the The fourth network layer, the fifth network layer and the sixth network layer are residual layers.
可选的，所述根据所述全局特征映射集进行重建，得到重建后的高分辨率图像，具体包括：Optionally, the reconstruction according to the global feature map set to obtain a reconstructed high-resolution image specifically includes:
根据所述全局特征映射集对所述第七网络层进行重建，得到重建后的高分辨率图像；所述第七网络层为亚像素卷积层。Reconstructing the seventh network layer according to the global feature map set to obtain a reconstructed high-resolution image; the seventh network layer is a sub-pixel convolution layer.
一种基于卷积神经网络的图像超分辨率重建系统，所述系统采用七个网络层，所述网络层包括第一网络层、第二网络层、第三网络层、第四网络层、第五网络层、第六网络层和第七网络层，包括：An image super-resolution reconstruction system based on a convolutional neural network, the system adopts seven network layers, and the network layers include a first network layer, a second network layer, a third network layer, a fourth network layer, a The fifth network layer, the sixth network layer and the seventh network layer, including:
获取模块，用于获取待重建的图像；An acquisition module, configured to acquire an image to be reconstructed;
预处理模块，用于对所述待重建的图像进行预处理，得到预处理后的图像；A preprocessing module, configured to preprocess the image to be reconstructed to obtain a preprocessed image;
特征提取模块，用于对所述预处理后的图像的特征进行提取，得到特征图像集；A feature extraction module, configured to extract features of the preprocessed image to obtain a feature image set;
映射模块，用于对所述特征图像集进行非线性映射，得到全局特征映射集；A mapping module, configured to perform nonlinear mapping on the feature image set to obtain a global feature map set;
重建模块，用于根据所述全局特征映射集进行重建，得到重建后的高分辨率图像。A reconstruction module, configured to perform reconstruction according to the global feature map set to obtain a reconstructed high-resolution image.
可选的，所述预处理模块具体包括：Optionally, the preprocessing module specifically includes:
预处理单元，用于对所述待重建的图像进行色彩空间转换，得到预处理后的亮度通道图像。A preprocessing unit, configured to perform color space conversion on the image to be reconstructed to obtain a preprocessed brightness channel image.
可选的，所述特征提取模块具体包括：Optionally, the feature extraction module specifically includes:
第一次特征提取单元，用于将所述预处理后的图像作为所述第一网络层的输入，进行特征提取，得到第一层特征图像集；The first feature extraction unit is used to use the preprocessed image as the input of the first network layer to perform feature extraction to obtain the first layer feature image set;
第二次特征提取单元，用于将所述第一层特征图像集作为所述第二网络层的输入，进行特征提取，得到第二层特征图像集；所述第一网络层和所述第二网络层为卷积层。The second feature extraction unit is used to use the feature image set of the first layer as the input of the second network layer to perform feature extraction to obtain a feature image set of the second layer; the first network layer and the second network layer The second network layer is a convolutional layer.
可选的，所述映射模块具体包括：Optionally, the mapping module specifically includes:
第一映射操作单元，用于将所述第一层特征图像集和所述第二层特征图像集分别作为所述第三网络层的输入，进行映射操作，得到第三特征图像集；A first mapping operation unit, configured to use the feature image set of the first layer and the feature image set of the second layer as the input of the third network layer respectively, and perform a mapping operation to obtain a third feature image set;
第二映射操作单元，用于将所述第三特征图像集作为所述第四网络层的输入，进行映射操作，得到第四特征图像集；A second mapping operation unit, configured to use the third feature image set as the input of the fourth network layer, and perform a mapping operation to obtain a fourth feature image set;
第三映射操作单元，用于将所述第三特征图像集和所述第四特征图像集作为所述第五网络层的输入，进行映射操作，得到第五特征图像集；A third mapping operation unit, configured to use the third feature image set and the fourth feature image set as inputs of the fifth network layer to perform a mapping operation to obtain a fifth feature image set;
第四映射操作单元，用于将所述第五特征图像集作为所述第六网络层的输入，进行映射操作，得到第六特征图像集，所述第六特征图像集为全局特征映射集；所述第三网络层、所述第四网络层、所述第五网络层和所述第六网络层为残差层。The fourth mapping operation unit is configured to use the fifth feature image set as the input of the sixth network layer to perform a mapping operation to obtain a sixth feature image set, and the sixth feature image set is a global feature map set; The third network layer, the fourth network layer, the fifth network layer and the sixth network layer are residual layers.
可选的，所述重建模块具体包括：Optionally, the reconstruction module specifically includes:
重建单元，用于根据所述全局特征映射集对所述第七网络层进行重建，得到重建后的高分辨率图像；所述第七网络层为亚像素卷积层。The reconstruction unit is configured to reconstruct the seventh network layer according to the global feature map set to obtain a reconstructed high-resolution image; the seventh network layer is a sub-pixel convolution layer.
根据本发明提供的具体实施例，本发明公开了以下技术效果：According to the specific embodiments provided by the invention, the invention discloses the following technical effects:
当网络深度加深时，由于不同的网络层的学习速度差异较大，且当后面的网络层学习效果更好时，前面的网络层的参数会出现停滞不变的情况，即梯度消失。而且此时前面网络层的参数梯度可能会变大，即梯度爆炸等情况的发生。因此，本发明使用残差网络层替代部分卷积层解决上述情况，残差网络层可以通过捷径连接的方式将低层网络的特征信息直接传送到高层网络，实现特征信息的重复利用，从而避免网络由于层数加深导致梯度爆炸或者梯度消失的情况。When the network depth is deepened, because the learning speed of different network layers varies greatly, and when the learning effect of the later network layer is better, the parameters of the previous network layer will stagnate, that is, the gradient will disappear. And at this time, the parameter gradient of the previous network layer may become larger, that is, the occurrence of gradient explosion. Therefore, the present invention uses the residual network layer to replace part of the convolutional layer to solve the above situation. The residual network layer can directly transmit the feature information of the low-level network to the high-level network through a shortcut connection, so as to realize the repeated use of feature information, thereby avoiding network The situation where the gradient explodes or the gradient disappears due to the deepening of the number of layers.
本发明使用亚像素卷积层作为重构层可以将低分辨率的特征图像通过像素组合排列的形式形成高分辨率图像，并且由于在网络中图像计算均为低分辨率图像空间，因此，可以有效降低网络的重建时间。The present invention uses the sub-pixel convolutional layer as the reconstruction layer to form the low-resolution feature image into a high-resolution image in the form of pixel combination arrangement, and since the image calculation in the network is all in the low-resolution image space, it can Effectively reduce the rebuilding time of the network.
总之，本发明通过网络层数的加深，即构建两层特征提取层、四层非线性映射层以及一层重构层，能够使图像的重建质量相比于传统方法有显著的提高；其中，在纹理细节和文字信息方法效果较为显著。In a word, the present invention, by deepening the number of network layers, that is, constructing two layers of feature extraction layers, four layers of nonlinear mapping layers and one layer of reconstruction layer, can significantly improve the quality of image reconstruction compared with traditional methods; among them, The effect of the method on texture details and text information is more significant.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without paying creative labor.
图1为本发明基于卷积神经网络的图像超分辨率重建方法流程图；Fig. 1 is the flow chart of the image super-resolution reconstruction method based on the convolutional neural network of the present invention;
图2为本发明基于卷积神经网络的图像超分辨率重建系统结构图；Fig. 2 is the structural diagram of the image super-resolution reconstruction system based on the convolutional neural network of the present invention;
图3为本发明的网络结构模型图；Fig. 3 is a network structure model figure of the present invention;
图4为本发明的结构模型对应的卷积核参数设置。FIG. 4 shows the convolution kernel parameter settings corresponding to the structural model of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
本发明的目的是提供一种基于卷积神经网络的图像超分辨率重建方法及系统，能够提高图像的重建质量。The purpose of the present invention is to provide an image super-resolution reconstruction method and system based on a convolutional neural network, which can improve image reconstruction quality.
为使本发明的上述目的、特征和优点能够更加明显易懂，下面结合附图和具体实施方式对本发明作进一步详细的说明。In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.
实施例1：Example 1:
图1为本发明基于卷积神经网络的图像超分辨率重建方法流程图。如图1所示，一种基于卷积神经网络的图像超分辨率重建方法，所述方法采用七个网络层，所述网络层包括第一网络层、第二网络层、第三网络层、第四网络层、第五网络层、第六网络层和第七网络层，包括：Fig. 1 is a flow chart of the image super-resolution reconstruction method based on convolutional neural network in the present invention. As shown in Figure 1, a kind of image super-resolution reconstruction method based on convolutional neural network, described method adopts seven network layers, and described network layer comprises the first network layer, the second network layer, the third network layer, The fourth network layer, the fifth network layer, the sixth network layer and the seventh network layer, including:
步骤101：获取待重建的图像。Step 101: Obtain an image to be reconstructed.
步骤102：对所述待重建的图像进行预处理，得到预处理后的图像，具体包括：Step 102: Perform preprocessing on the image to be reconstructed to obtain a preprocessed image, specifically including:
对所述待重建的图像进行色彩空间转换，得到预处理后的亮度通道图像。Perform color space conversion on the image to be reconstructed to obtain a preprocessed brightness channel image.
步骤103：对所述预处理后的图像的特征进行提取，得到特征图像集，具体包括：Step 103: Extracting the features of the preprocessed image to obtain a feature image set, specifically including:
将所述预处理后的图像作为所述第一网络层的输入，进行特征提取，得到第一层特征图像集。The preprocessed image is used as an input of the first network layer to perform feature extraction to obtain a feature image set of the first layer.
将所述第一层特征图像集作为所述第二网络层的输入，进行特征提取，得到第二层特征图像集；所述第一网络层和所述第二网络层为卷积层。The feature image set of the first layer is used as the input of the second network layer to perform feature extraction to obtain the feature image set of the second layer; the first network layer and the second network layer are convolutional layers.
步骤104：对所述特征图像集进行非线性映射，得到全局特征映射集，具体包括：Step 104: Perform nonlinear mapping on the feature image set to obtain a global feature map set, specifically including:
将所述第一层特征图像集和所述第二层特征图像集分别作为所述第三网络层的输入，进行映射操作，得到第三特征图像集。The feature image set of the first layer and the feature image set of the second layer are respectively used as the input of the third network layer, and a mapping operation is performed to obtain a third feature image set.
将所述第三特征图像集作为所述第四网络层的输入，进行映射操作，得到第四特征图像集。The third feature image set is used as an input of the fourth network layer, and a mapping operation is performed to obtain a fourth feature image set.
将所述第三特征图像集和所述第四特征图像集作为所述第五网络层的输入，进行映射操作，得到第五特征图像集。The third feature image set and the fourth feature image set are used as the input of the fifth network layer, and a mapping operation is performed to obtain a fifth feature image set.
将所述第五特征图像集作为所述第六网络层的输入，进行映射操作，得到第六特征图像集，所述第六特征图像集为全局特征映射集；所述第三网络层、所述第四网络层、所述第五网络层和所述第六网络层为残差层。Using the fifth feature image set as the input of the sixth network layer, performing a mapping operation to obtain a sixth feature image set, the sixth feature image set is a global feature map set; the third network layer, the The fourth network layer, the fifth network layer and the sixth network layer are residual layers.
步骤105：根据所述全局特征映射集进行重建，得到重建后的高分辨率图像，具体包括：Step 105: Reconstruct according to the global feature map set to obtain a reconstructed high-resolution image, specifically including:
根据所述全局特征映射集对所述第七网络层进行重建，得到重建后的高分辨率图像；所述第七网络层为亚像素卷积层。Reconstructing the seventh network layer according to the global feature map set to obtain a reconstructed high-resolution image; the seventh network layer is a sub-pixel convolution layer.
实施例2：Example 2:
图2为本发明基于卷积神经网络的图像超分辨率重建系统结构图。如图2所示，一种基于卷积神经网络的图像超分辨率重建系统，所述系统采用七个网络层，所述网络层包括第一网络层、第二网络层、第三网络层、第四网络层、第五网络层、第六网络层和第七网络层，包括：FIG. 2 is a structural diagram of an image super-resolution reconstruction system based on a convolutional neural network in the present invention. As shown in Figure 2, a kind of image super-resolution reconstruction system based on convolutional neural network, described system adopts seven network layers, and described network layer comprises the first network layer, the second network layer, the third network layer, The fourth network layer, the fifth network layer, the sixth network layer and the seventh network layer, including:
获取模块201，用于获取待重建的图像。The acquiring module 201 is configured to acquire an image to be reconstructed.
预处理模块202，用于对所述待重建的图像进行预处理，得到预处理后的图像。The preprocessing module 202 is configured to preprocess the image to be reconstructed to obtain a preprocessed image.
特征提取模块203，用于对所述预处理后的图像的特征进行提取，得到特征图像集。The feature extraction module 203 is configured to extract features of the preprocessed image to obtain a feature image set.
映射模块204，用于对所述特征图像集进行非线性映射，得到全局特征映射集。The mapping module 204 is configured to perform nonlinear mapping on the feature image set to obtain a global feature map set.
重建模块205，用于根据所述全局特征映射集进行重建，得到重建后的高分辨率图像。The reconstruction module 205 is configured to perform reconstruction according to the global feature map set to obtain a reconstructed high-resolution image.
所述预处理模块202具体包括：The preprocessing module 202 specifically includes:
预处理单元，用于对所述待重建的图像进行色彩空间转换，得到预处理后的亮度通道图像。A preprocessing unit, configured to perform color space conversion on the image to be reconstructed to obtain a preprocessed brightness channel image.
所述特征提取模块203具体包括：The feature extraction module 203 specifically includes:
第一次特征提取单元，用于将所述预处理后的图像作为所述第一网络层的输入，进行特征提取，得到第一层特征图像集。The first feature extraction unit is configured to use the preprocessed image as the input of the first network layer to perform feature extraction to obtain a feature image set of the first layer.
第二次特征提取单元，用于将所述第一层特征图像集作为所述第二网络层的输入，进行特征提取，得到第二层特征图像集；所述第一网络层和所述第二网络层为卷积层。The second feature extraction unit is used to use the feature image set of the first layer as the input of the second network layer to perform feature extraction to obtain a feature image set of the second layer; the first network layer and the second network layer The second network layer is a convolutional layer.
所述映射模块204具体包括：The mapping module 204 specifically includes:
第一映射操作单元，用于将所述第一层特征图像集和所述第二层特征图像集分别作为所述第三网络层的输入，进行映射操作，得到第三特征图像集。The first mapping operation unit is configured to use the feature image set of the first layer and the feature image set of the second layer as the input of the third network layer respectively, and perform a mapping operation to obtain a third feature image set.
第二映射操作单元，用于将所述第三特征图像集作为所述第四网络层的输入，进行映射操作，得到第四特征图像集。The second mapping operation unit is configured to use the third feature image set as an input of the fourth network layer, and perform a mapping operation to obtain a fourth feature image set.
第三映射操作单元，用于将所述第三特征图像集和所述第四特征图像集作为所述第五网络层的输入，进行映射操作，得到第五特征图像集。A third mapping operation unit, configured to use the third feature image set and the fourth feature image set as inputs of the fifth network layer to perform a mapping operation to obtain a fifth feature image set.
第四映射操作单元，用于将所述第五特征图像集作为所述第六网络层的输入，进行映射操作，得到第六特征图像集，所述第六特征图像集为全局特征映射集；所述第三网络层、所述第四网络层、所述第五网络层和所述第六网络层为残差层。The fourth mapping operation unit is configured to use the fifth feature image set as the input of the sixth network layer to perform a mapping operation to obtain a sixth feature image set, and the sixth feature image set is a global feature map set; The third network layer, the fourth network layer, the fifth network layer and the sixth network layer are residual layers.
所述重建模块205具体包括：The reconstruction module 205 specifically includes:
重建单元，用于根据所述全局特征映射集对所述第七网络层进行重建，得到重建后的高分辨率图像；所述第七网络层为亚像素卷积层。The reconstruction unit is configured to reconstruct the seventh network layer according to the global feature map set to obtain a reconstructed high-resolution image; the seventh network layer is a sub-pixel convolution layer.
实施例3：Example 3:
本发明实施例3构建了一种包含卷积层、残差网络层以及亚像素卷积层等三种网络层的网络结构。初始低分辨率图像经过预处理后，将Y通道图像作为亚像素卷积神经网络的输入数据，通过对网络模型不断训练最后完成重建。本发明所提出的网络模型具体参数设置，共包含7个网络层，定义每一层网络表示为Conv(input，output，filter)，其中input为输入通道数，output为输出通道数，filter为卷积核的大小。Embodiment 3 of the present invention constructs a network structure including three network layers including a convolutional layer, a residual network layer, and a sub-pixel convolutional layer. After the initial low-resolution image is preprocessed, the Y-channel image is used as the input data of the sub-pixel convolutional neural network, and the reconstruction is completed through continuous training of the network model. The specific parameter settings of the network model proposed by the present invention include 7 network layers in total, and each layer of network is defined as Conv (input, output, filter), where input is the number of input channels, output is the number of output channels, and filter is the volume The size of the core.
图3为本发明的网络结构模型图。如图3所示，在特征提取阶段中，Conv1和Conv2均使用64个5×5的卷积核来使低维度特征信息尽量丰富。非线性映射阶段，Conv3、Conv4、Conv5和Conv6使用32个卷积核可以将冗余的数据丢弃，即可以减少参数，提高程序的计算效率。在Conv5和Conv6中，卷积核的大小由5×5降低为3×3可以进一步提高程序的计算效率，而且此时使用较小的卷积核不会造成纹理信息大量丢失的情况，且经过多次非线性映射可以将低维度的特征图像映射为高维度特征图像。在图像重构阶段，Conv7的卷积核的数量将由32降为r2个，这是为了完成重排过程中特征图像的像素点与高分辨率图像区域对应。Fig. 3 is a network structure model diagram of the present invention. As shown in Figure 3, in the feature extraction stage, both Conv1 and Conv2 use 64 5×5 convolution kernels to enrich the low-dimensional feature information as much as possible. In the nonlinear mapping stage, Conv3, Conv4, Conv5, and Conv6 use 32 convolution kernels to discard redundant data, which can reduce parameters and improve the computational efficiency of the program. In Conv5 and Conv6, reducing the size of the convolution kernel from 5×5 to 3×3 can further improve the computational efficiency of the program, and using a smaller convolution kernel at this time will not cause a large loss of texture information, and after Multiple nonlinear mappings can map low-dimensional feature images to high-dimensional feature images. In the image reconstruction stage, the number of convolution kernels of Conv7 will be reduced from 32 to r2. This is to complete the pixel points of the feature image in the rearrangement process corresponding to the high-resolution image area.
详细操作如下所示：The detailed operation is as follows:
(1)图像预处理阶段，由于颜色空间的不同，对图像的处理结果会产生很大的影响，特别是空间域的图像相关性较强，因此需要对图像进行色彩空间转换。经分析发现，在图像重建领域中，图像的亮度对图像重建结构会产生极大的影响，而RGB空间色彩模式难以区分亮度信息，因此在处理过程中将初始图像的RGB空间色彩模式转换为YCbCr模式。图4为本发明的结构模型对应的卷积核参数设置，设置内容包括输入通道、输出通道和卷积核尺寸。(1) In the image preprocessing stage, due to the difference in color space, it will have a great impact on the image processing results, especially the image correlation in the spatial domain is strong, so it is necessary to convert the color space of the image. After analysis, it is found that in the field of image reconstruction, the brightness of the image will have a great impact on the image reconstruction structure, and the RGB space color mode is difficult to distinguish the brightness information, so the RGB space color mode of the original image is converted to YCbCr in the process of processing model. Fig. 4 is the convolution kernel parameter setting corresponding to the structural model of the present invention, and the setting content includes input channel, output channel and convolution kernel size.
(2)图像特征提取阶段，将经过预处理的亮度通道图像Y作为Conv1的输入与64个5×5的卷积核进行第一次特征提取，输出第一层低维度特征图像集F1(Y)。具体操作如公式1所示：(2) In the image feature extraction stage, the preprocessed luminance channel image Y is used as the input of Conv1 and 64 5×5 convolution kernels for the first feature extraction, and the first layer of low-dimensional feature image set F1 ( Y). The specific operation is shown in Formula 1:
Fi(Y)＝Tanh(Wi*Y+Bi) (1)Fi (Y)＝Tanh(Wi *Y+Bi ) (1)
其中，Wi，Bi分为为网络的权重和偏差。Wi的大小为ci×ni×fi×fi，其中ci表示输入图像的通道数，ni为网络的卷积核数量，即输出通道数，fi×fi为卷积核的大小，Tanh(x)为特征提取后的激活函数Tanh，在卷积层中均使用Tanh函数激活。Among them, Wi and Bi are divided into the weight and bias of the network. The size of Wi is ci ×ni ×fi ×fi , where ci represents the number of channels of the input image, ni is the number of convolution kernels of the network, that is, the number of output channels, and fi ×fi is the convolution The size of the kernel, Tanh(x) is the activation function Tanh after feature extraction, and the Tanh function is used for activation in the convolutional layer.
将特征图像集F1(Y)作为Conv2的输入与64个5×5的卷积核进行第二次特征提取，并获得特征图像集F2(Y)，二次特征提取可以使低维度的特征信息尽可能充分，具体实现公式为公式1，此时F1(Y)和F2(Y)均为64通道特征图像集。The feature image set F1 (Y) is used as the input of Conv2 and 64 5×5 convolution kernels for the second feature extraction, and the feature image set F2 (Y) is obtained. The secondary feature extraction can make the low-dimensional The feature information is as sufficient as possible, and the specific realization formula is formula 1, at this time, both F1 (Y) and F2 (Y) are 64-channel feature image sets.
(3)非线性映射阶段，Conv3、Conv4、Conv5和Conv6为非线性映射层。将特征图像集F1(Y)和F2(Y)分别作为残差层Conv3的恒等映射分支和卷积分支与32个5×5的卷积核进行卷积计算，输出特征图像集F3(Y)；(3) Non-linear mapping stage, Conv3, Conv4, Conv5 and Conv6 are non-linear mapping layers. The feature image sets F1 (Y) and F2 (Y) are respectively used as the identity mapping branch and the convolution branch of the residual layer Conv3 to perform convolution calculations with 32 5×5 convolution kernels, and the output feature image set F3 (Y);
将特征图像集F3(Y)作为Conv4的输入与32个5×5的卷积核进行第二次映射并输出特征图像集F4(Y)，F3(Y)和F4(Y)均为32通道特征图像集；将特征图像集F3(Y)和F4(Y)分为作为残差层Conv5的两个分支与32个3×3的卷积核完成第三次映射操作，由于卷积核的变小，此时的特征图像集F5(Y)维度更高且纹理细节更丰富；最后将特征图像集F5(Y)与32个3×3进行最后一次非线性映射，获得高维度的全局特征映射集F6(Y)。其中，Conv4和Conv6的操作与Conv1类似，为公式1。Conv3和Conv5具体操作如公式2所示：Take the feature image set F3 (Y) as the input of Conv4 and perform the second mapping with 32 5×5 convolution kernels and output the feature image set F4 (Y), F3 (Y) and F4 (Y) Both are 32-channel feature image sets; the feature image sets F3 (Y) and F4 (Y) are divided into two branches as the residual layer Conv5 and 32 3×3 convolution kernels to complete the third mapping operation , due to the smaller convolution kernel, the feature image set F5 (Y) at this time has higher dimensions and richer texture details; finally, the feature image set F5 (Y) and 32 3×3 nonlinearities are performed for the last time Mapping to obtain a high-dimensional global feature map set F6 (Y). Among them, the operation of Conv4 and Conv6 is similar to that of Conv1, which is formula 1. The specific operations of Conv3 and Conv5 are shown in formula 2:
Fi(Y)＝ReLU[0，Wi*(k1*Fi-1(Y)+k2*Fi-2(Y))+Bi] (2)Fi (Y)＝ReLU[0，Wi *(k1 *Fi-1 (Y)+k2 *Fi-2 (Y))+Bi ] (2)
其中，k1和k2为i-1层和i-2层的特征图在本层中图像通道中所占的比例，为保持数据一致性，k1+k2＝1。W3的大小为(k1×ni-1+k2×ni-2)×ni×fi×fi。ReLU(x)为激活函数ReLU函数。Among them, k1 and k2 are the proportions of the feature maps of layer i-1 and layer i-2 in the image channels of this layer. In order to maintain data consistency, k1 +k2 =1 . The size of W3 is (k1 ×ni-1 +k2 ×ni-2 )×ni ×fi ×fi . ReLU(x) is the activation function ReLU function.
(4)图像重构阶段，在亚像素卷积层Conv7中利用全局特征映射集F6(Y)进行重构，得到最终的高分辨率图像ISR。经过卷积层和残差层后图像的通道变成r2个，通过将这些图像通过以亚像素卷积层的方式进行重排列来完成图像的重建工作。具体操作如公式3所示：(4) In the image reconstruction stage, the global feature map set F6 (Y) is used for reconstruction in the sub-pixel convolutional layer Conv7 to obtain the final high-resolution image ISR . After the convolutional layer and the residual layer, the channels of the image become r2 , and the reconstruction of the image is completed by rearranging these images in a sub-pixel convolutional layer. The specific operation is shown in formula 3:
ISR＝PS[(Wi*Fi-1(Y)+Bi)] (3)ISR ＝PS[(Wi *Fi-1 (Y)+Bi )] (3)
其中，Wi的大小为r2·c×ni×fi×fi，c为初始低分辨率图像的输入通道数，PS为亚像素卷积操作，表示将通道图像根据像素位置进行重新排列，对图像的特征进行整合，ISR为重建完成的高分辨率图像。Among them, the size of Wi is r2 ·c×ni ×fi ×fi , c is the number of input channels of the initial low-resolution image, PS is the sub-pixel convolution operation, which means that the channel image is reconstructed according to the pixel position Arrangement, integrate the features of the image, andISR is the reconstructed high-resolution image.
采用本发明的方法，具有下列优点：Adopt method of the present invention, have following advantage:
1、通过网络层数的加深，即构建两层特征提取层、四层非线性映射层以及一层重构层，能够使图像的重建质量相比于传统方法有显著的提高，其中在纹理细节和文字信息方法效果较为显著。1. By deepening the number of network layers, that is, constructing two layers of feature extraction layers, four layers of nonlinear mapping layers and one layer of reconstruction layer, the quality of image reconstruction can be significantly improved compared with traditional methods, among which texture details The effect of text information method is more obvious.
2、当网络深度加深时，由于不同的网络层的学习速度差异较大，且当后面的网络层学习效果更好时，前面的网络层的参数会出现停滞不变的情况，即梯度消失。而且此时前面网络层的参数梯度可能会变大，即梯度爆炸等情况的发生。因此使用残差网络层替代部分卷积层解决上述情况，残差网络层可以通过捷径连接的方式将低层网络的特征信息直接传送到高层网络，实现特征信息的重复利用，从而避免网络由于层数加深导致梯度爆炸或者梯度消失的情况。2. When the network depth is deepened, because the learning speed of different network layers is quite different, and when the learning effect of the later network layer is better, the parameters of the previous network layer will stagnate, that is, the gradient will disappear. And at this time, the parameter gradient of the previous network layer may become larger, that is, the occurrence of gradient explosion. Therefore, the residual network layer is used to replace part of the convolutional layer to solve the above situation. The residual network layer can directly transmit the feature information of the low-level network to the high-level network through a shortcut connection, so as to realize the repeated use of feature information, thereby avoiding the network due to the number of layers. Deepening causes the gradient to explode or disappear.
3、本发明使用亚像素卷积层作为重构层可以将低分辨率的特征图像通过像素组合排列的形式形成高分辨率图像，并且由于在网络中图像计算均为低分辨率图像空间，因此可以有效降低网络的重建时间。3. The present invention uses a sub-pixel convolutional layer as a reconstruction layer to form a high-resolution image by combining and arranging low-resolution feature images in the form of pixels, and since the image calculation in the network is all in the low-resolution image space, therefore It can effectively reduce the reconstruction time of the network.
本说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的系统而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.
本文中应用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的一般技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处。综上所述，本说明书内容不应理解为对本发明的限制。In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the present invention Thoughts, there will be changes in specific implementation methods and application ranges. In summary, the contents of this specification should not be construed as limiting the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910439720.0ACN110163803A (en) | 2019-05-24 | 2019-05-24 | A kind of image super-resolution rebuilding method and system based on convolutional neural networks |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910439720.0ACN110163803A (en) | 2019-05-24 | 2019-05-24 | A kind of image super-resolution rebuilding method and system based on convolutional neural networks |
| Publication Number | Publication Date |
|---|---|
| CN110163803Atrue CN110163803A (en) | 2019-08-23 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910439720.0APendingCN110163803A (en) | 2019-05-24 | 2019-05-24 | A kind of image super-resolution rebuilding method and system based on convolutional neural networks |
| Country | Link |
|---|---|
| CN (1) | CN110163803A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110545426A (en)* | 2019-08-29 | 2019-12-06 | 西安电子科技大学 | Spatial Scalable Video Coding Method Based on Coding Damage Repair CNN |
| CN110826467A (en)* | 2019-11-22 | 2020-02-21 | 中南大学湘雅三医院 | An electron microscope image reconstruction system and method thereof |
| CN111445392A (en)* | 2020-03-20 | 2020-07-24 | Oppo广东移动通信有限公司 | Image processing method and device, computer readable storage medium and electronic device |
| CN112907446A (en)* | 2021-02-07 | 2021-06-04 | 电子科技大学 | Image super-resolution reconstruction method based on packet connection network |
| CN113222816A (en)* | 2021-04-30 | 2021-08-06 | Oppo广东移动通信有限公司 | Super-resolution network structure, super-resolution module, equipment and reconstruction method |
| CN114494006A (en)* | 2020-10-26 | 2022-05-13 | 中国移动通信有限公司研究院 | Training method and device for image reconstruction model, electronic equipment and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107977932A (en)* | 2017-12-28 | 2018-05-01 | 北京工业大学 | It is a kind of based on can differentiate attribute constraint generation confrontation network face image super-resolution reconstruction method |
| CN108734659A (en)* | 2018-05-17 | 2018-11-02 | 华中科技大学 | A kind of sub-pix convolved image super resolution ratio reconstruction method based on multiple dimensioned label |
| CN109615582A (en)* | 2018-11-30 | 2019-04-12 | 北京工业大学 | A face image super-resolution reconstruction method based on attribute description generative adversarial network |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107977932A (en)* | 2017-12-28 | 2018-05-01 | 北京工业大学 | It is a kind of based on can differentiate attribute constraint generation confrontation network face image super-resolution reconstruction method |
| CN108734659A (en)* | 2018-05-17 | 2018-11-02 | 华中科技大学 | A kind of sub-pix convolved image super resolution ratio reconstruction method based on multiple dimensioned label |
| CN109615582A (en)* | 2018-11-30 | 2019-04-12 | 北京工业大学 | A face image super-resolution reconstruction method based on attribute description generative adversarial network |
| Title |
|---|
| 刘月峰 等: "基于改进卷积神经网络的单幅图像超分辨率重建方法", 《计算机应用》* |
| 王旺 等: "改进的基于卷积神将网络的图像超分辨率方法", 《计算机应用与软件》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110545426A (en)* | 2019-08-29 | 2019-12-06 | 西安电子科技大学 | Spatial Scalable Video Coding Method Based on Coding Damage Repair CNN |
| CN110545426B (en)* | 2019-08-29 | 2021-04-20 | 西安电子科技大学 | Spatial Scalable Video Coding Method Based on Coding Damage Repair CNN |
| CN110826467A (en)* | 2019-11-22 | 2020-02-21 | 中南大学湘雅三医院 | An electron microscope image reconstruction system and method thereof |
| CN110826467B (en)* | 2019-11-22 | 2023-09-29 | 中南大学湘雅三医院 | Electron microscope image reconstruction system and method thereof |
| CN111445392A (en)* | 2020-03-20 | 2020-07-24 | Oppo广东移动通信有限公司 | Image processing method and device, computer readable storage medium and electronic device |
| CN111445392B (en)* | 2020-03-20 | 2023-09-15 | Oppo广东移动通信有限公司 | Image processing method and device, computer readable storage medium and electronic equipment |
| CN114494006A (en)* | 2020-10-26 | 2022-05-13 | 中国移动通信有限公司研究院 | Training method and device for image reconstruction model, electronic equipment and storage medium |
| CN114494006B (en)* | 2020-10-26 | 2025-04-15 | 中国移动通信有限公司研究院 | Image reconstruction model training method, device, electronic device and storage medium |
| CN112907446A (en)* | 2021-02-07 | 2021-06-04 | 电子科技大学 | Image super-resolution reconstruction method based on packet connection network |
| CN112907446B (en)* | 2021-02-07 | 2022-06-07 | 电子科技大学 | An image super-resolution reconstruction method based on packet connection network |
| CN113222816A (en)* | 2021-04-30 | 2021-08-06 | Oppo广东移动通信有限公司 | Super-resolution network structure, super-resolution module, equipment and reconstruction method |
| Publication | Publication Date | Title |
|---|---|---|
| CN110033410B (en) | Image reconstruction model training method, image super-resolution reconstruction method and device | |
| CN108921786B (en) | Image super-resolution reconstruction method based on residual convolutional neural network | |
| CN110163803A (en) | A kind of image super-resolution rebuilding method and system based on convolutional neural networks | |
| CN108765296B (en) | An Image Super-Resolution Reconstruction Method Based on Recurrent Residual Attention Network | |
| CN108537733B (en) | Super-resolution reconstruction method based on multi-path deep convolutional neural network | |
| CN111915487B (en) | Face super-resolution method and device based on hierarchical multi-scale residual fusion network | |
| CN108805814B (en) | Image super-resolution reconstruction method based on multi-band deep convolutional neural network | |
| CN107240066A (en) | Image super-resolution rebuilding algorithm based on shallow-layer and deep layer convolutional neural networks | |
| CN108647775B (en) | Super-resolution image reconstruction method based on full convolution neural network single image | |
| CN109859110B (en) | Panchromatic Sharpening Method of Hyperspectral Image Based on Spectral Dimension Control Convolutional Neural Network | |
| CN111951164B (en) | An image super-resolution reconstruction network structure and image reconstruction effect analysis method | |
| CN110322402B (en) | Medical image super-resolution reconstruction method based on dense mixed attention network | |
| CN116977651B (en) | Image denoising method based on double-branch and multi-scale feature extraction | |
| CN109934771B (en) | Super-resolution reconstruction method of unsupervised remote sensing image based on recurrent neural network | |
| CN105550989B (en) | The image super-resolution method returned based on non local Gaussian process | |
| CN113781308A (en) | Image super-resolution reconstruction method, device, storage medium and electronic device | |
| CN112767243B (en) | Method and system for realizing super-resolution of hyperspectral image | |
| CN105447840A (en) | Image super-resolution method based on active sampling and Gaussian process regression | |
| CN112889084A (en) | Method, system and computer readable medium for improving color quality of image | |
| CN106447609A (en) | Image super-resolution method based on depth convolutional neural network | |
| CN114677271A (en) | Hyperspectral image super-resolution reconstruction method based on depth image prior | |
| Li et al. | RGSR: A two-step lossy JPG image super-resolution based on noise reduction | |
| CN115641260A (en) | Image processing method, module, device, storage medium and program product | |
| CN112819724B (en) | A Scanned Document Image Enhancement Method Based on CNN | |
| CN118967467A (en) | Hyperspectral image fusion method based on high-frequency information extraction and multi-feature matrix |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20190823 | |
| RJ01 | Rejection of invention patent application after publication |