CN115564655A - Video super-resolution reconstruction method, system and medium based on deep learning - Google Patents
Video super-resolution reconstruction method, system and medium based on deep learningDownload PDFInfo
- Publication number
- CN115564655A CN115564655ACN202211392882.1ACN202211392882ACN115564655ACN 115564655 ACN115564655 ACN 115564655ACN 202211392882 ACN202211392882 ACN 202211392882ACN 115564655 ACN115564655 ACN 115564655A
- Authority
- CN
- China
- Prior art keywords
- module
- video
- super
- resolution
- branch
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明涉及视频处理技术领域，特别是涉及一种基于深度学习的视频超分辨率重建方法、系统及介质。The present invention relates to the technical field of video processing, in particular to a video super-resolution reconstruction method, system and medium based on deep learning.
背景技术Background technique
分辨率是一组用于评估图像中蕴含细节信息丰富程度的性能参数，包括时间分辨率、空间分辨率及色阶分辨率等，体现了成像系统实际所能反映物体细节信息的能力。相较于低分辨率图像，高分辨率图像通常包含更大的像素密度、更丰富的纹理细节及更高的可信赖度。但在实际上情况中，受采集设备与环境、网络传输介质与带宽、视频退化模型本身等诸多因素的约束，通常并不能直接得到具有边缘锐化、无成块模糊的理想高分辨率图像。提升图像分辨率的最直接的做法是对采集系统中的光学硬件进行改进，但是由于制造工艺难以大幅改进并且制造成本十分高昂，因此物理上解决图像低分辨率问题往往代价太大。Resolution is a set of performance parameters used to evaluate the richness of detailed information contained in an image, including temporal resolution, spatial resolution, and color scale resolution, etc., reflecting the ability of the imaging system to actually reflect the detailed information of an object. Compared with low-resolution images, high-resolution images usually contain greater pixel density, richer texture details, and higher reliability. However, in actual situations, due to the constraints of many factors such as acquisition equipment and environment, network transmission medium and bandwidth, and video degradation model itself, it is usually not possible to directly obtain ideal high-resolution images with edge sharpening and no block blur. The most direct way to improve the image resolution is to improve the optical hardware in the acquisition system, but because the manufacturing process is difficult to greatly improve and the manufacturing cost is very high, it is often too expensive to physically solve the problem of low image resolution.
视频的超分辨率重建技术指的是将给定的低分辨率图像通过特定的算法恢复成相应的高分辨率视频。与图像超分相比，视频超分可以利用相邻多帧的信息，以达到更好的超分效果。传统的超分算法，如插值等会使得高分辨率的视频图像出现边缘模糊，效果不佳。Video super-resolution reconstruction technology refers to restoring a given low-resolution image to a corresponding high-resolution video through a specific algorithm. Compared with image super-resolution, video super-resolution can use the information of adjacent multiple frames to achieve better super-resolution effects. Traditional super-resolution algorithms, such as interpolation, will blur the edges of high-resolution video images, and the effect is not good.
发明内容Contents of the invention
本发明的目的是提供一种基于深度学习的视频超分辨率重建方法、系统及介质，可提高高分辨率视频图像的质量。The purpose of the present invention is to provide a video super-resolution reconstruction method, system and medium based on deep learning, which can improve the quality of high-resolution video images.
为实现上述目的，本发明提供了如下方案：To achieve the above object, the present invention provides the following scheme:
一种基于深度学习的视频超分辨率重建方法，包括：A video super-resolution reconstruction method based on deep learning, comprising:
构建超分模型；所述超分模型为以待训练视频各帧对应的图像为输入，以所述待训练视频各帧对应的超分辨率图像为输出，以频率损失函数最小为目标对BasicVSR模型进行训练得到的；所述BasicVSR模型的前向分支和后向分支均包括GDFN模块；Build a super-resolution model; the super-resolution model is to input the image corresponding to each frame of the video to be trained, to output the super-resolution image corresponding to each frame of the video to be trained, and to target the BasicVSR model with the minimum frequency loss function Obtained by training; the forward branch and the backward branch of the BasicVSR model both include the GDFN module;
获取待处理视频；Get the video to be processed;
将所述待处理视频的各帧图像输入所述超分模型得到所述待处理视频各帧图像对应的超分辨率图像；Inputting each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed;
根据所述待处理视频各帧图像对应的超分辨率图像得到所述待处理视频对应的超分辨率视频。The super-resolution video corresponding to the video to be processed is obtained according to the super-resolution image corresponding to each frame image of the video to be processed.
可选的，所述BasicVSR模型包括前向分支、后向分支和上采样分支；所述前向分支的输出端与所述后向分支的输出端均与所述上采样分支的输入端连接。Optionally, the BasicVSR model includes a forward branch, a backward branch and an upsampling branch; the output end of the forward branch and the output end of the backward branch are both connected to the input end of the upsampling branch.
可选的，所述前向分支包括N个前向传播模块；所述后向分支包括N个后向传播模块；所述上采样分支包括N个上采样传播模块；N为大于1的正整数；Optionally, the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; N is a positive integer greater than 1 ;
第i个前向传播模块的第一输入端与第i-1个前向传播模块的第一输出端连接；所述第i个前向传播模块的第二输入端用于输入所述待处理视频的第i帧图像和第i-1帧图像；所述第i个前向传播模块的第一输出端与所述第i+1个前向传播模块的第一输入端连接；所述第i个前向传播模块的第二输出端与第i个上采样模块的第一输入端连接；The first input end of the i-th forward propagation module is connected to the first output end of the i-1th forward propagation module; the second input end of the i-th forward propagation module is used to input the to-be-processed The i-th frame image and the i-1th frame image of the video; the first output end of the i-th forward propagation module is connected to the first input end of the i+1-th forward propagation module; the first The second output end of the i forward propagation module is connected to the first input end of the i th upsampling module;
第i个后向传播模块的第一输入端与第i+1个后向传播模块的第一输出端连接；所述第i个后向传播模块的第二输入端用于输入所述待处理视频的第i帧图像和第i-1帧图像；所述第i个后向传播模块的第一输出端与所述第i-1个后向传播模块的第一输入端连接；所述第i个后向传播模块的第二输出端与第i个上采样模块的第二输入端连接。The first input end of the i-th backpropagation module is connected to the first output end of the i+1th backpropagation module; the second input end of the i-th backpropagation module is used to input the to-be-processed The i-th frame image and the i-1th frame image of the video; the first output end of the i-th backward propagation module is connected to the first input end of the i-1th backward propagation module; the first The second output end of the i backpropagation module is connected to the second input end of the i th upsampling module.
可选的，所述前向传播模块和所述后向传播模块均包括光流估计模块、空间扭曲模块和深度残差块，所述光流估计模块、所述空间扭曲模块、所述GDFN模块和所述深度残差块依次连接。Optionally, both the forward propagation module and the backward propagation module include an optical flow estimation module, a space warping module and a depth residual block, the optical flow estimation module, the space warping module, and the GDFN module and the depth residual block are sequentially connected.
可选的，所述频率损失函数具体为：Optionally, the frequency loss function is specifically:
其中，
一种基于深度学习的视频超分辨率重建系统，包括：A video super-resolution reconstruction system based on deep learning, including:
构建模块，用于构建超分模型；所述超分模型为以待训练视频各帧对应的图像为输入，以所述待训练视频各帧对应的超分辨率图像为输出，以频率损失函数最小为目标对BasicVSR模型进行训练得到的；所述BasicVSR模型的前向分支和后向分支均包括GDFN模块；A building block for constructing a super-resolution model; the super-resolution model is to input the image corresponding to each frame of the video to be trained, to output the super-resolution image corresponding to each frame of the video to be trained, and to minimize the frequency loss function Obtained by training the BasicVSR model for the target; the forward branch and the backward branch of the BasicVSR model both include the GDFN module;
获取模块，用于获取待处理视频；The acquisition module is used to acquire the video to be processed;
超分辨率图像确定模块，用于将所述待处理视频的各帧图像输入所述超分模型得到所述待处理视频各帧图像对应的超分辨率图像；A super-resolution image determination module, configured to input each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed;
超分辨率视频确定模块；根据所述待处理视频各帧图像对应的超分辨率图像得到所述待处理视频对应的超分辨率视频。A super-resolution video determining module; obtain the super-resolution video corresponding to the video to be processed according to the super-resolution image corresponding to each frame image of the video to be processed.
可选的，所述BasicVSR模型包括前向分支、后向分支和上采样分支；所述前向分支的输出端与所述后向分支的输出端均与所述上采样分支的输入端连接。Optionally, the BasicVSR model includes a forward branch, a backward branch and an upsampling branch; the output end of the forward branch and the output end of the backward branch are both connected to the input end of the upsampling branch.
可选的，所述前向分支包括N个前向传播模块；所述后向分支包括N个后向传播模块；所述上采样分支包括N个上采样传播模块；N为大于1的正整数；Optionally, the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; N is a positive integer greater than 1 ;
第i个前向传播模块的第一输入端与第i-1个前向传播模块的第一输出端连接；所述第i个前向传播模块的第二输入端用于输入所述待处理视频的第i帧图像和第i-1帧图像；所述第i个前向传播模块的第一输出端与所述第i+1个前向传播模块的第一输入端连接；所述第i个前向传播模块的第二输出端与第i个上采样模块的第一输入端连接；The first input end of the i-th forward propagation module is connected to the first output end of the i-1th forward propagation module; the second input end of the i-th forward propagation module is used to input the to-be-processed The i-th frame image and the i-1th frame image of the video; the first output end of the i-th forward propagation module is connected to the first input end of the i+1-th forward propagation module; the first The second output end of the i forward propagation module is connected to the first input end of the i th upsampling module;
第i个后向传播模块的第一输入端与第i+1个后向传播模块的第一输出端连接；所述第i个后向传播模块的第二输入端用于输入所述待处理视频的第i帧图像和第i-1帧图像；所述第i个后向传播模块的第一输出端与所述第i-1个后向传播模块的第一输入端连接；所述第i个后向传播模块的第二输出端与第i个上采样模块的第二输入端连接。The first input end of the i-th backpropagation module is connected to the first output end of the i+1th backpropagation module; the second input end of the i-th backpropagation module is used to input the to-be-processed The i-th frame image and the i-1th frame image of the video; the first output end of the i-th backward propagation module is connected to the first input end of the i-1th backward propagation module; the first The second output end of the i backpropagation module is connected to the second input end of the i th upsampling module.
可选的，所述前向传播模块和所述后向传播模块均包括光流估计模块、空间扭曲模块和深度残差块，所述光流估计模块、所述空间扭曲模块、所述GDFN模块和所述深度残差块依次连接。Optionally, both the forward propagation module and the backward propagation module include an optical flow estimation module, a space warping module and a depth residual block, the optical flow estimation module, the space warping module, and the GDFN module and the depth residual block are sequentially connected.
可选的，一种计算机可读存储介质，其存储有计算机程序，所述计算机程序被处理器执行时实现如上述所述的基于深度学习的视频超分辨率重建方法。Optionally, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements the method for video super-resolution reconstruction based on deep learning as described above.
根据本发明提供的具体实施例，本发明公开了以下技术效果：本发明使用GDFN模块来达到更好的特征融合效果，可提高高分辨率视频图像的质量。According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: the present invention uses the GDFN module to achieve a better feature fusion effect, and can improve the quality of high-resolution video images.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without paying creative labor.
图1为本发明实施例提供的一种基于深度学习的视频超分辨率重建方法的流程图；Fig. 1 is a flow chart of a deep learning-based video super-resolution reconstruction method provided by an embodiment of the present invention;
图2为BasicVSR模型的具体架构图；Figure 2 is a specific architecture diagram of the BasicVSR model;
图3为前向传播模块的具体结构图；Fig. 3 is the specific structural diagram of forward propagation module;
图4为后向传播模块的具体结构图；Fig. 4 is the specific structural diagram of backward propagation module;
图5为GDFN模块的具体结构图；Fig. 5 is the specific structural diagram of GDFN module;
图6为视频超分系统的具体结构图。FIG. 6 is a specific structural diagram of the video super-resolution system.
具体实施方式detailed description
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
为使本发明的上述目的、特征和优点能够更加明显易懂，下面结合附图和具体实施方式对本发明作进一步详细的说明。In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.
随着深度学习的兴起，基于深度学习的视频超分辨率技术发展越来越迅速，本发明提出了一种基于深度学习的视频超分辨率重建方法，本发明中的超分模型使用循环网络架构进行视频帧信息间的信息传播，同时使用GDFN模块提升特征融合的效果，加入频率损失函数优化网络使超分模型同时具有良好的性能、低参数量和计算效率高的优点。With the rise of deep learning, the video super-resolution technology based on deep learning is developing more and more rapidly. The present invention proposes a video super-resolution reconstruction method based on deep learning. The super-resolution model in the present invention uses a recurrent network architecture Carry out information dissemination between video frame information, and use the GDFN module to improve the effect of feature fusion, and add frequency loss function to optimize the network so that the super-resolution model has the advantages of good performance, low parameter amount and high computational efficiency.
本发明实施例提供了一种基于深度学习的视频超分辨率重建方法，如图1所示，包括：The embodiment of the present invention provides a video super-resolution reconstruction method based on deep learning, as shown in Figure 1, including:
步骤101：构建超分模型；所述超分模型为以待训练视频各帧对应的图像为输入，以所述待训练视频各帧对应的超分辨率图像为输出，以频率损失函数最小为目标对BasicVSR模型进行训练得到的；所述BasicVSR模型的前向分支和后向分支均包括GDFN模块。Step 101: Construct a super-resolution model; the super-resolution model takes the image corresponding to each frame of the video to be trained as input, takes the super-resolution image corresponding to each frame of the video to be trained as output, and takes the minimum frequency loss function as the goal Obtained by training the BasicVSR model; both the forward branch and the backward branch of the BasicVSR model include a GDFN module.
步骤102：获取待处理视频。Step 102: Obtain the video to be processed.
步骤103：将所述待处理视频的各帧图像输入所述超分模型得到所述待处理视频各帧图像对应的超分辨率图像。Step 103: Input each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed.
步骤104：根据所述待处理视频各帧图像对应的超分辨率图像得到所述待处理视频对应的超分辨率视频。Step 104: Obtain the super-resolution video corresponding to the video to be processed according to the super-resolution image corresponding to each frame image of the video to be processed.
在实际应用中，所述BasicVSR模型包括前向分支、后向分支和上采样分支；所述前向分支的输出端与所述后向分支的输出端均与所述上采样分支的输入端连接。In practical applications, the BasicVSR model includes a forward branch, a backward branch, and an upsampling branch; the output of the forward branch and the output of the backward branch are connected to the input of the upsampling branch .
在实际应用中，如图2所示，所述前向分支包括N个前向传播模块；所述后向分支包括N个后向传播模块；所述上采样分支包括N个上采样传播模块；N为大于1的正整数。In practical applications, as shown in Figure 2, the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; N is a positive integer greater than 1.
第i个前向传播模块的第一输入端与第i-1个前向传播模块的第一输出端连接，用于输入第i-1个前向传播模块输出的第i-1帧图像的前向传播特征
第i个后向传播模块的第一输入端与第i+1个后向传播模块的第一输出端连接，用于输入第i+1个后向传播模块输出的第i+1帧图像的后向传播特征
在实际应用中，如图3和图4所示，所述前向传播模块和所述后向传播模块均包括光流估计模块、空间扭曲模块和深度残差块，所述光流估计模块、所述空间扭曲模块、所述GDFN模块和所述深度残差块依次连接。In practical applications, as shown in Figure 3 and Figure 4, the forward propagation module and the backward propagation module both include an optical flow estimation module, a space warping module and a depth residual block, the optical flow estimation module, The space warping module, the GDFN module and the depth residual block are sequentially connected.
以第i个前向传播模块为例对前向传播的具体工作流程进行介绍：首先通过光流估计模块计算xi-1与xi的前向光流信息
以第i个后向传播模块为例对后向传播的具体工作流进行介绍：首先通过光流估计模块计算xi-1与xi的后向光流信息
之后将
在实际应用中，GDFN模块具体结构如图5所示，特征融合模块GDFN使用depth-wiseconvolution从空间相邻的像素位置编码信息，可用于学习有效地融合特征。输入的特征在经过归一化操作(Norm)后，按通道均分为两部分，各自通过1x1卷积和3x3卷积，其中一个分支通过GELU激活函数激活后，与另一分支进行元素积，在通过一个1x1卷积恢复通道后和原始输入相加，得到最终的结果。In practical applications, the specific structure of the GDFN module is shown in Figure 5. The feature fusion module GDFN uses depth-wise convolution to encode information from spatially adjacent pixel positions, which can be used to learn to fuse features effectively. After the normalization operation (Norm), the input feature is divided into two parts according to the channel, each of which passes through 1x1 convolution and 3x3 convolution. After one branch is activated by the GELU activation function, it performs an element product with the other branch. After restoring the channel through a 1x1 convolution, it is added to the original input to get the final result.
在实际应用中，本发明使用频率损失函数帮助获取更多的图像细节信息。使用一些常用的损失函数时，为了减小损失值，模型往往倾向于使得视频图像变得更加平滑。这些细节部分往往对应频率信号的高频部分，因此通过频率损失函数减少频率空间的差异，获得更加清晰锐利的视频。所述频率损失函数具体为：In practical applications, the present invention uses a frequency loss function to help obtain more image detail information. When using some commonly used loss functions, in order to reduce the loss value, the model tends to make the video image smoother. These details often correspond to the high-frequency part of the frequency signal, so the difference in frequency space is reduced through the frequency loss function to obtain a clearer and sharper video. The frequency loss function is specifically:
其中，
本发明实施例针对上述方法还提供了一种基于深度学习的视频超分辨率重建系统，包括：The embodiment of the present invention also provides a deep learning-based video super-resolution reconstruction system for the above method, including:
构建模块，用于构建超分模型；所述超分模型为以待训练视频各帧对应的图像为输入，以所述待训练视频各帧对应的超分辨率图像为输出，以频率损失函数最小为目标对BasicVSR模型进行训练得到的；所述BasicVSR模型的前向分支和后向分支均包括GDFN模块。A building block for constructing a super-resolution model; the super-resolution model is to input the image corresponding to each frame of the video to be trained, to output the super-resolution image corresponding to each frame of the video to be trained, and to minimize the frequency loss function It is obtained by training the BasicVSR model for the target; both the forward branch and the backward branch of the BasicVSR model include a GDFN module.
获取模块，用于获取待处理视频。The acquisition module is used to acquire the video to be processed.
超分辨率图像确定模块，用于将所述待处理视频的各帧图像输入所述超分模型得到所述待处理视频各帧图像对应的超分辨率图像。A super-resolution image determining module, configured to input each frame image of the video to be processed into the super-resolution model to obtain a super-resolution image corresponding to each frame image of the video to be processed.
超分辨率视频确定模块；根据所述待处理视频各帧图像对应的超分辨率图像得到所述待处理视频对应的超分辨率视频。A super-resolution video determining module; obtain the super-resolution video corresponding to the video to be processed according to the super-resolution image corresponding to each frame image of the video to be processed.
在实际应用中，所述BasicVSR模型包括前向分支、后向分支和上采样分支；所述前向分支的输出端与所述后向分支的输出端均与所述上采样分支的输入端连接。In practical applications, the BasicVSR model includes a forward branch, a backward branch, and an upsampling branch; the output of the forward branch and the output of the backward branch are connected to the input of the upsampling branch .
在实际应用中，所述前向分支包括N个前向传播模块；所述后向分支包括N个后向传播模块；所述上采样分支包括N个上采样传播模块；N为大于1的正整数。In practical applications, the forward branch includes N forward propagation modules; the backward branch includes N backward propagation modules; the upsampling branch includes N upsampling propagation modules; N is a positive value greater than 1 integer.
第i个前向传播模块的第一输入端与第i-1个前向传播模块的第一输出端连接；所述第i个前向传播模块的第二输入端用于输入所述待处理视频的第i帧图像和第i-1帧图像；所述第i个前向传播模块的第一输出端与所述第i+1个前向传播模块的第一输入端连接；所述第i个前向传播模块的第二输出端与第i个上采样模块的第一输入端连接。The first input end of the i-th forward propagation module is connected to the first output end of the i-1th forward propagation module; the second input end of the i-th forward propagation module is used to input the to-be-processed The i-th frame image and the i-1th frame image of the video; the first output end of the i-th forward propagation module is connected to the first input end of the i+1-th forward propagation module; the first The second output end of the i forward propagation module is connected to the first input end of the i th upsampling module.
第i个后向传播模块的第一输入端与第i+1个后向传播模块的第一输出端连接；所述第i个后向传播模块的第二输入端用于输入所述待处理视频的第i帧图像和第i-1帧图像；所述第i个后向传播模块的第一输出端与所述第i-1个后向传播模块的第一输入端连接；所述第i个后向传播模块的第二输出端与第i个上采样模块的第二输入端连接。The first input end of the i-th backpropagation module is connected to the first output end of the i+1th backpropagation module; the second input end of the i-th backpropagation module is used to input the to-be-processed The i-th frame image and the i-1th frame image of the video; the first output end of the i-th backward propagation module is connected to the first input end of the i-1th backward propagation module; the first The second output end of the i backpropagation module is connected to the second input end of the i th upsampling module.
在实际应用中，所述前向传播模块和所述后向传播模块均包括光流估计模块、空间扭曲模块和深度残差块，所述光流估计模块、所述空间扭曲模块、所述GDFN模块和所述深度残差块依次连接。In practical applications, both the forward propagation module and the backward propagation module include an optical flow estimation module, a space warping module and a depth residual block, the optical flow estimation module, the space warping module, the GDFN Modules and the deep residual block are sequentially connected.
本发明实施例还提供了一种计算机可读存储介质，其存储有计算机程序，所述计算机程序被处理器执行时实现上述实施例所述的基于深度学习的视频超分辨率重建方法。An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, implements the video super-resolution reconstruction method based on deep learning described in the above-mentioned embodiments.
本发明实施例还提供了一种视频超分系统：The embodiment of the present invention also provides a video super-resolution system:
为了更好地展示模型的超分性能，本发明使用开放神经网络交换(ONNX)将模型进行转换，从而可以在未安装模型依赖库的环境中进行视频超分任务。使用pyqt搭建去系统界面。如图6所示，图中select为视频选择按钮，点击select选择需要处理的视频文件，OriginalVideo为原始视频播放器，Modified Video为超分视频播放器，model a、modelb和model c为超分算法选择按钮，按下按钮将使用对应的超分算法进行视频处理并播放，即可得到对应的视频。In order to better demonstrate the super-resolution performance of the model, the present invention uses the Open Neural Network Exchange (ONNX) to convert the model, so that the video super-resolution task can be performed in an environment where the model-dependent library is not installed. Use pyqt to build the system interface. As shown in Figure 6, select in the figure is the video selection button, click select to select the video file to be processed, OriginalVideo is the original video player, Modified Video is the super-resolution video player, model a, modelb and model c are super-resolution algorithms Select the button, press the button to use the corresponding super-resolution algorithm for video processing and playback, and the corresponding video can be obtained.
本发明基于BasicVSR模型改进，相比现有的BasicVSR，本发明模型使用GDFN模块来达到更好的特征融合效果。使用频率损失函数减少超分结果高频分量的损失。The present invention is based on the improvement of the BasicVSR model. Compared with the existing BasicVSR, the model of the present invention uses the GDFN module to achieve a better feature fusion effect. Use the frequency loss function to reduce the loss of high-frequency components of super-resolution results.
本说明书中各个实施例采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似部分互相参见即可。对于实施例公开的系统而言，由于其与实施例公开的方法相对应，所以描述的比较简单，相关之处参见方法部分说明即可。Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for relevant details, please refer to the description of the method part.
本文中应用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的一般技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处。综上所述，本说明书内容不应理解为对本发明的限制。In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the present invention Thoughts, there will be changes in specific implementation methods and application ranges. In summary, the contents of this specification should not be construed as limiting the present invention.
Claims (10)
Translated fromChinese
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211392882.1ACN115564655B (en) | 2022-11-08 | Video super-resolution reconstruction method, system and medium based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211392882.1ACN115564655B (en) | 2022-11-08 | Video super-resolution reconstruction method, system and medium based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115564655Atrue CN115564655A (en) | 2023-01-03 |
| CN115564655B CN115564655B (en) | 2025-10-10 |
Family
ID=
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118037549A (en)* | 2024-04-11 | 2024-05-14 | 华南理工大学 | Video enhancement method and system based on video content understanding |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583112A (en)* | 2020-04-29 | 2020-08-25 | 华南理工大学 | Method, system, device and storage medium for video super-resolution |
| CN112767250A (en)* | 2021-01-19 | 2021-05-07 | 南京理工大学 | Video blind super-resolution reconstruction method and system based on self-supervision learning |
| CN112991183A (en)* | 2021-04-09 | 2021-06-18 | 华南理工大学 | Video super-resolution method based on multi-frame attention mechanism progressive fusion |
| CN114332561A (en)* | 2021-12-24 | 2022-04-12 | 腾讯科技（深圳）有限公司 | Training method, device, equipment and medium for super-resolution model |
| CN114418845A (en)* | 2021-12-28 | 2022-04-29 | 北京欧珀通信有限公司 | Image resolution improving method and device, storage medium and electronic equipment |
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111583112A (en)* | 2020-04-29 | 2020-08-25 | 华南理工大学 | Method, system, device and storage medium for video super-resolution |
| CN112767250A (en)* | 2021-01-19 | 2021-05-07 | 南京理工大学 | Video blind super-resolution reconstruction method and system based on self-supervision learning |
| CN112991183A (en)* | 2021-04-09 | 2021-06-18 | 华南理工大学 | Video super-resolution method based on multi-frame attention mechanism progressive fusion |
| CN114332561A (en)* | 2021-12-24 | 2022-04-12 | 腾讯科技（深圳）有限公司 | Training method, device, equipment and medium for super-resolution model |
| CN114418845A (en)* | 2021-12-28 | 2022-04-29 | 北京欧珀通信有限公司 | Image resolution improving method and device, storage medium and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| KELVIN C.K ET AL.: "BasicVSR: The search for essential components in video super-resolution and beyond", 《ARXIV》, 7 April 2021 (2021-04-07), pages 1 - 5* |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118037549A (en)* | 2024-04-11 | 2024-05-14 | 华南理工大学 | Video enhancement method and system based on video content understanding |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2024178979A1 (en) | Single-image defogging method based on detail restoration | |
| CN113727141B (en) | Interpolation device and method for video frames | |
| CN114881921B (en) | De-occlusion imaging method and device based on event and video fusion | |
| CN110889809B9 (en) | Image processing method and device, electronic equipment and storage medium | |
| CN111985281B (en) | Image generation model generation method and device and image generation method and device | |
| CN111062883B (en) | Image processing method and device, computer readable medium and electronic device | |
| CN111860363A (en) | A video image processing method and device, electronic device, and storage medium | |
| CN115065796B (en) | Method and device for generating video intermediate frame | |
| US20240362747A1 (en) | Methods for generating image super-resolution data set, image super-resolution model and training method | |
| CN113538287A (en) | Video enhancement network training method, video enhancement method and related device | |
| CN113992920A (en) | Video compressed sensing reconstruction method based on deep expansion network | |
| WO2023010750A1 (en) | Image color mapping method and apparatus, electronic device, and storage medium | |
| Xin et al. | Video face super-resolution with motion-adaptive feedback cell | |
| CN113012073A (en) | Training method and device for video quality improvement model | |
| WO2023160426A1 (en) | Video frame interpolation method and apparatus, training method and apparatus, and electronic device | |
| CN114245117A (en) | Multi-sampling rate multiplexing network reconstruction method, device, equipment and storage medium | |
| CN113658128A (en) | Image blurring degree determining method, data set constructing method and deblurring method | |
| CN115564655A (en) | Video super-resolution reconstruction method, system and medium based on deep learning | |
| CN115564655B (en) | Video super-resolution reconstruction method, system and medium based on deep learning | |
| CN114240750B (en) | Video resolution enhancement method and device, storage medium and electronic device | |
| CN112598578B (en) | Super-resolution reconstruction system and method for nuclear magnetic resonance image | |
| CN109862299A (en) | Resolution processing method and device | |
| CN115861048A (en) | Image super-resolution method, device, equipment and storage medium | |
| CN115103118B (en) | High dynamic range image generation method, device, equipment and readable storage medium | |
| Dar et al. | Modular admm-based strategies for optimized compression, restoration, and distributed representations of visual data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |