CN115439369A - Image noise reduction method and device, electronic equipment, storage medium - Google Patents

Abstract

Translated fromChinese

本公开实施例是关于一种图像降噪方法及装置、电子设备、存储介质，涉及影像技术领域，该图像降噪方法包括：获取当前图像以及所述当前图像的参考帧图像的时域降噪图像，并对所述时域降噪图像进行分解得到多层时域图像；对所述当前图像进行空域降噪得到所述当前图像对应的空域降噪图像，并对所述空域降噪图像进行分解得到多层空域图像；将每层时域图像和每层空域图像进行匹配，确定所述时域降噪图像的对齐图像；将所述空域降噪图像以及所述对齐图像进行融合，获取所述当前图像对应的降噪图像。本公开实施例中的技术方案，能够提高图像降噪效果。

Embodiments of the present disclosure relate to an image noise reduction method and device, electronic equipment, and storage media, and relate to the field of image technology. The image noise reduction method includes: acquiring a current image and a temporal domain noise reduction of a reference frame image of the current image image, and decompose the time domain noise reduction image to obtain a multi-layer time domain image; perform spatial domain noise reduction on the current image to obtain a spatial domain noise reduction image corresponding to the current image, and perform spatial noise reduction on the spatial domain noise reduction image decompose to obtain a multi-layer spatial domain image; match each layer of time domain image and each layer of spatial domain image to determine the alignment image of the temporal noise reduction image; fuse the spatial domain noise reduction image and the alignment image to obtain the Describe the noise-reduced image corresponding to the current image. The technical solutions in the embodiments of the present disclosure can improve the image noise reduction effect.

Description

Translated fromChinese

图像降噪方法及装置、电子设备、存储介质Image noise reduction method and device, electronic equipment, storage medium

技术领域technical field

本公开涉及影像技术领域，具体而言，涉及一种图像降噪方法及装置、电子设备以及计算机可读存储介质。The present disclosure relates to the field of image technology, and in particular, to an image noise reduction method and device, electronic equipment, and a computer-readable storage medium.

背景技术Background technique

在图像处理过程中，图像降噪处理是提高图像质量的常用方式。In image processing, image noise reduction is a common way to improve image quality.

相关技术中，可将匹配的所有图像块转换到频域，在频域对各个频率分量再做一次频域降噪以降低高频噪声，然后再反变换到空域得到降噪后的图像。上述方式中，计算量较大且精准性较差。并且，可能导致部分区域出现伪影的问题，从而使得降噪效果较差。In related technologies, all matched image blocks can be converted to the frequency domain, frequency domain noise reduction is performed on each frequency component in the frequency domain to reduce high-frequency noise, and then inversely transformed to the spatial domain to obtain a noise-reduced image. In the above method, the amount of calculation is large and the accuracy is poor. Moreover, it may cause the problem of artifacts in some areas, which makes the noise reduction effect poor.

需要说明的是，在上述背景技术部分公开的信息仅用于加强对本公开的背景的理解，因此可以包括不构成对本领域普通技术人员已知的现有技术的信息。It should be noted that the information disclosed in the above background section is only for enhancing the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art.

发明内容Contents of the invention

本公开的目的在于提供一种图像降噪方法及装置、电子设备、存储介质，进而至少在一定程度上克服由于相关技术的限制和缺陷而导致的降噪效果较差的问题。The purpose of the present disclosure is to provide an image noise reduction method and device, an electronic device, and a storage medium, so as to overcome the problem of poor noise reduction effect caused by limitations and defects of related technologies at least to a certain extent.

本公开的其他特性和优点将通过下面的详细描述变得显然，或部分地通过本公开的实践而习得。Other features and advantages of the present disclosure will become apparent from the following detailed description, or in part, be learned by practice of the present disclosure.

根据本公开的第一方面，提供一种图像降噪方法，包括：获取当前图像以及所述当前图像的参考帧图像的时域降噪图像，并对所述时域降噪图像进行分解得到多层时域图像；对所述当前图像进行空域降噪得到所述当前图像对应的空域降噪图像，并对所述空域降噪图像进行分解得到多层空域图像；将每层时域图像和每层空域图像进行匹配，确定所述时域降噪图像的对齐图像；将所述空域降噪图像以及所述对齐图像进行融合，获取所述当前图像对应的降噪图像。According to the first aspect of the present disclosure, an image noise reduction method is provided, including: acquiring a temporal domain noise reduction image of a current image and a reference frame image of the current image, and decomposing the temporal domain noise reduction image to obtain multiple layer time domain image; performing spatial denoising on the current image to obtain a spatial denoising image corresponding to the current image, and decomposing the spatial denoising image to obtain a multi-layer air domain image; combining each layer of time domain image and each Match the spatial domain images of the layers to determine the alignment image of the temporal noise reduction image; fuse the spatial noise reduction image and the alignment image to obtain the noise reduction image corresponding to the current image.

根据本公开的第二方面，提供一种图像降噪装置，包括：第一图像分解模块，用于获取当前图像以及所述当前图像的参考帧图像的时域降噪图像，并对所述时域降噪图像进行分解得到多层时域图像；第二图像分解模块，用于对所述当前图像进行空域降噪得到所述当前图像对应的空域降噪图像，并对所述空域降噪图像进行分解得到多层空域图像；图像匹配模块，用于将每层时域图像和每层空域图像进行匹配，确定所述时域降噪图像的对齐图像；图像融合模块，用于将所述空域降噪图像以及所述对齐图像进行融合，获取所述当前图像对应的降噪图像。According to a second aspect of the present disclosure, an image noise reduction device is provided, including: a first image decomposition module, configured to acquire a temporal noise reduction image of a current image and a reference frame image of the current image, and analyze the temporal Decompose the domain noise reduction image to obtain a multi-layer time domain image; the second image decomposition module is used to perform spatial domain noise reduction on the current image to obtain a spatial domain noise reduction image corresponding to the current image, and perform spatial domain noise reduction on the spatial domain noise reduction image Decompose to obtain a multi-layer space domain image; an image matching module is used to match each layer of time domain images with each layer of space domain images, and determine the alignment image of the time domain noise reduction image; an image fusion module is used to combine the space domain images The noise-reduced image and the aligned image are fused to obtain a noise-reduced image corresponding to the current image.

根据本公开的第三方面，提供一种电子设备，包括：处理器；以及存储器，用于存储所述处理器的可执行指令；其中，所述处理器配置为经由执行所述可执行指令来执行上述第一方面的图像降噪方法及其可能的实现方式。According to a third aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to Execute the image noise reduction method of the above first aspect and its possible implementation.

根据本公开的第四方面，提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述第一方面的图像降噪方法及其可能的实现方式。According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the image noise reduction method of the above-mentioned first aspect and possible implementations thereof are implemented.

本公开实施例中提供的技术方案中，一方面，先通过每层时域图像和每层空域图像对图像做精细对齐，再通过对齐图像以及空域降噪图像融合实现对当前图像的时域降噪，避免了相关技术中只能在部分区域实现比较好的降噪效果的局限性，避免了容易产生的伪影问题，增加了降噪范围，提高了降噪的全面性和可靠性。另一方面，通过每层空域图像和每层时域图像进行匹配，通过分层匹配能够增加图像匹配时的搜索范围且提高匹配的准确性，避免了时域和空域之间的转换，减小了图像匹配时的计算量，提升了降噪效果，提高了降噪图像的图像质量。In the technical solutions provided in the embodiments of the present disclosure, on the one hand, the images are finely aligned through each layer of time domain images and each layer of spatial domain images, and then the temporal domain reduction of the current image is realized through the fusion of aligned images and spatial domain noise reduction images. Noise, avoiding the limitation of the related technology that can only achieve better noise reduction effect in some areas, avoiding the easy-to-produce artifact problem, increasing the noise reduction range, and improving the comprehensiveness and reliability of noise reduction. On the other hand, through the matching of each layer of spatial domain images and each layer of time domain images, the search range of image matching can be increased and the accuracy of matching can be increased through layered matching, which avoids the conversion between time domain and space domain and reduces The amount of calculation during image matching is reduced, the noise reduction effect is improved, and the image quality of the noise reduction image is improved.

应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

附图说明Description of drawings

此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Apparently, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings according to these drawings without creative efforts.

图1示出了可以应用本公开实施例的图像降噪方法的应用场景的示意图。Fig. 1 shows a schematic diagram of an application scenario to which the image noise reduction method of the embodiment of the present disclosure can be applied.

图2示意性示出本公开实施例一种图像降噪方法的示意图。Fig. 2 schematically shows a schematic diagram of an image noise reduction method according to an embodiment of the present disclosure.

图3示意性示出本公开实施例中对时域降噪图像进行分解的示意图。Fig. 3 schematically shows a schematic diagram of decomposing a temporal-domain noise-reduced image in an embodiment of the present disclosure.

图4示意性示出本公开实施例中对空域降噪图像进行分解的示意图。Fig. 4 schematically shows a schematic diagram of decomposing a spatial noise reduction image in an embodiment of the present disclosure.

图5示意性示出本公开实施例中获取对齐图像的流程示意图。Fig. 5 schematically shows a flow chart of acquiring aligned images in an embodiment of the present disclosure.

图6示意性示出本公开实施例的第一次块匹配的流程示意图。Fig. 6 schematically shows a schematic flowchart of the first block matching in the embodiment of the present disclosure.

图7示意性示出本公开实施例的第一次块匹配的具体流程示意图。Fig. 7 schematically shows a specific flowchart of the first block matching in the embodiment of the present disclosure.

图8示意性示出本公开实施例的第二次块匹配的具体流程示意图。Fig. 8 schematically shows a specific flowchart of the second block matching in the embodiment of the present disclosure.

图9示意性示出本公开实施例的第三次块匹配的具体流程示意图。FIG. 9 schematically shows a specific flowchart of the third block matching in the embodiment of the present disclosure.

图10示意性示出本公开实施例中进行图像融合的流程示意图。Fig. 10 schematically shows a schematic flow chart of image fusion in an embodiment of the present disclosure.

图11示意性示出本公开实施例中进行时域降噪的整体流程示意图。FIG. 11 schematically shows an overall flow chart of temporal noise reduction in an embodiment of the present disclosure.

图12示意性示出本公开实施例中一种图像降噪装置的框图。Fig. 12 schematically shows a block diagram of an image noise reduction device in an embodiment of the present disclosure.

图13示意性示出本公开实施例中电子设备的框图。Fig. 13 schematically shows a block diagram of an electronic device in an embodiment of the present disclosure.

具体实施方式detailed description

现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。在下面的描述中，提供许多具体细节从而给出对本公开的实施方式的充分理解。然而，本领域技术人员将意识到，可以实践本公开的技术方案而省略所述特定细节中的一个或更多，或者可以采用其它的方法、组元、装置、步骤等。在其它情况下，不详细示出或描述公知技术方案以避免喧宾夺主而使得本公开的各方面变得模糊。Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details being omitted, or other methods, components, devices, steps, etc. may be adopted. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

此外，附图仅为本公开的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and/or processor means and/or microcontroller means.

相关技术中，可以将匹配的所有图像块转换到频域，在频域对各个频率分量再做一次频域降噪以降低高频噪声，然后再反变换到空域得到降噪后的图像。上述降噪方案中，一方面计算量很大，为了保证降噪效果不得不在较大的搜索窗口内搜索匹配块，然后搜索到的所有参考块都要转化到频域，然后每个频率分量也要转换到频域，在频域降噪完成后再经过两次反变换回到空域才得到最终的降噪图像，如果搜索窗口太小虽然减小了计算量但是降噪效果也会随之下降。另一方面，图像的降噪效果未考虑时域的影响，只对空域做降噪有时候会把高频噪声转换成低频噪声，视频播放时就会看到低频噪声的闪动。In related technologies, all matched image blocks can be converted to the frequency domain, frequency domain noise reduction is performed on each frequency component in the frequency domain to reduce high-frequency noise, and then inversely transformed to the spatial domain to obtain a noise-reduced image. In the above noise reduction scheme, on the one hand, the amount of calculation is very large. In order to ensure the noise reduction effect, the matching blocks have to be searched in a large search window, and then all the searched reference blocks must be transformed into the frequency domain, and then each frequency component is also To convert to the frequency domain, after the noise reduction in the frequency domain is completed, the final noise reduction image is obtained after two inverse transformations and then returned to the space domain. If the search window is too small, the calculation amount will be reduced, but the noise reduction effect will also decrease. . On the other hand, the noise reduction effect of the image does not consider the influence of the time domain. Only the noise reduction in the air domain sometimes converts high-frequency noise into low-frequency noise, and you will see flickering low-frequency noise when the video is played.

为了解决相关技术中的技术问题，本公开实施例中提供了一种图像降噪方法，可以应用于图像降噪过程，以增强图像降噪效果的应用场景。图1示出了可以应用本公开实施例的图像降噪方法及装置的系统架构的示意图。In order to solve the technical problems in the related technologies, the embodiments of the present disclosure provide an image noise reduction method, which can be applied to the image noise reduction process to enhance the image noise reduction effect application scenarios. FIG. 1 shows a schematic diagram of a system architecture to which an image noise reduction method and device according to an embodiment of the present disclosure can be applied.

如图1所示，终端101可以为具有图像处理功能的智能设备，例如可以为智能手机、电脑、平板电脑、智能音箱、智能手表、车载设备、可穿戴设备、监控设备等智能设备。当前图像可以为拍摄得到的图像，也可以为拍摄得到的视频中的每一帧图像，还可以为终端中存储的图像或视频中的某一帧图像。As shown in FIG. 1 , theterminal 101 may be a smart device with an image processing function, such as a smart phone, a computer, a tablet computer, a smart speaker, a smart watch, a vehicle device, a wearable device, a monitoring device and other smart devices. The current image may be a captured image, may also be an image of each frame in a captured video, or may be an image stored in a terminal or a certain frame of an image in a video.

本公开实施例中，终端101可以包括存储器102以及处理器103。存储器用于对图像进行存储，处理器用于对图像进行处理。存储器102中可以存储当前图像104。终端101从存储器102中获取当前图像104以及参考帧图像105，并发送至处理器103中，在处理器103中对所述当前图像进行空域降噪并进行金字塔分解得到多层空域图像；对参考帧图像进行时域降噪以及金字塔分解得到多层时域图像；将多层空域图像以及多层时域图像进行分层对齐，得到对齐图像；将对齐图像以及空域降噪图像进行融合，从而生成时域降噪后的降噪图像106。In this embodiment of the present disclosure, theterminal 101 may include amemory 102 and aprocessor 103 . The memory is used to store images, and the processor is used to process images. Acurrent image 104 may be stored inmemory 102 . Theterminal 101 obtains thecurrent image 104 and thereference frame image 105 from thememory 102, and sends them to theprocessor 103, and in theprocessor 103, the current image is subjected to spatial noise reduction and pyramid decomposition to obtain a multi-layer spatial image; The frame image is subjected to time-domain noise reduction and pyramid decomposition to obtain a multi-layer time-domain image; the multi-layer spatial domain image and the multi-layer time-domain image are layered and aligned to obtain an aligned image; the aligned image and the spatial domain noise reduction image are fused to generate Noise-reducedimage 106 after time-domain noise reduction.

需要说明的是，本公开实施例所提供的图像降噪方法可以由终端101来执行。图像降噪方法也可以设置于终端中。It should be noted that the image noise reduction method provided by the embodiment of the present disclosure may be executed by theterminal 101 . The image noise reduction method can also be set in the terminal.

图2中示意性示出了本公开实施例中的图像降噪方法，具体包括以下步骤：Figure 2 schematically shows an image noise reduction method in an embodiment of the present disclosure, which specifically includes the following steps:

步骤S210，获取当前图像以及所述当前图像的参考帧图像的时域降噪图像，并对所述时域降噪图像进行分解得到多层时域图像；Step S210, acquiring the temporal noise reduction image of the current image and the reference frame image of the current image, and decomposing the temporal noise reduction image to obtain a multi-layer temporal image;

步骤S220，对所述当前图像进行空域降噪得到所述当前图像对应的空域降噪图像，并对所述空域降噪图像进行分解得到多层空域图像；Step S220, performing spatial noise reduction on the current image to obtain a spatial noise reduction image corresponding to the current image, and decomposing the spatial noise reduction image to obtain a multi-layer spatial image;

步骤S230，将每层时域图像和每层空域图像进行匹配，确定所述时域降噪图像的对齐图像；Step S230, matching each layer of time-domain images with each layer of space-domain images to determine an aligned image of the temporal-domain noise-reduced image;

步骤S240，将所述空域降噪图像以及所述对齐图像进行融合，获取所述当前图像对应的降噪图像。Step S240, fusing the spatial noise reduction image and the aligned image to obtain a noise reduction image corresponding to the current image.

本公开实施例中，当前图像可以为当前帧图像，可以将当前图像进行空域降噪得到空域降噪图像，并对空域降噪图像进行分解得到多层空域图像。例如，可以对空域降噪图像进行四层金字塔分解得到多层空域图像，将空域降噪图像A1分解为多层空域图像a0/a1/a2/a3。In the embodiments of the present disclosure, the current image may be a current frame image, and the current image may be denoised in the spatial domain to obtain a denoised image in the spatial domain, and the denoised image in the spatial domain may be decomposed to obtain a multi-layered spatial domain image. For example, the spatial denoising image may be decomposed into a four-layer pyramid to obtain a multi-layer spatial image, and the spatial denoising image A1 may be decomposed into a multi-layer spatial image a0/a1/a2/a3.

与此同时可以获取当前图像的参考帧图像，参考帧图像可以为与当前帧相邻的前一帧图像。可对参考帧图像进行时域降噪得到时域降噪图像，并对时域降噪图像进行分解得到多层时域图像。例如，可以对时域降噪图像B0进行四层金字塔分解得到多层时域图像b0/b1/b2/b3。At the same time, a reference frame image of the current image may be acquired, and the reference frame image may be a previous frame image adjacent to the current frame. Temporal noise reduction can be performed on the reference frame image to obtain a temporal noise reduction image, and the temporal noise reduction image can be decomposed to obtain a multi-layer temporal image. For example, a four-level pyramid decomposition can be performed on the time-domain noise-reduced image B0 to obtain multi-layer time-domain images b0/b1/b2/b3.

进一步地，可以对每层时域图像以及每层空域图像进行金字塔匹配，金字塔匹配的次数为多次，且每次金字塔匹配的尺度不同。在进行每次金字塔匹配时，均可以按照分辨率的排列顺序，将每层空域图像中的匹配块在相同分辨率的每层空域图像中进行图像对齐，以实现多次分层多尺度匹配，得到时域降噪图像的对齐图像。Further, pyramid matching may be performed on each layer of time-domain images and each layer of air-domain images, the number of times of pyramid matching is multiple, and the scale of each pyramid matching is different. When performing each pyramid matching, the matching blocks in each layer of airspace images can be image-aligned in each layer of airspace images of the same resolution according to the order of resolution, so as to achieve multiple layered and multi-scale matching, Get an aligned image of the temporally denoised image.

在此基础上，可以将对齐图像与空域降噪图像融合，得到当前图像的时域降噪后的降噪图像。On this basis, the aligned image can be fused with the spatial noise reduction image to obtain a noise reduction image after temporal noise reduction of the current image.

本公开实施例中，先对图像做精细对齐再通过对齐图像以及空域降噪图像融合实现时域降噪，避免了相关技术中的只能在部分区域实现比较好的降噪效果的局限性，避免了容易产生的伪影问题，能够提高降噪效果。除此之外，通过每层空域图像和每层时域图像进行匹配，通过分层匹配能够增加匹配时的搜索范围且提高匹配的准确性，并且减小了匹配时的计算量，提升了降噪效果，提高了图像质量。In the embodiment of the present disclosure, the images are finely aligned first, and then temporal domain noise reduction is realized by aligning the images and spatial noise reduction image fusion, which avoids the limitation of the related technology that can only achieve better noise reduction effects in some areas. Avoids the artifact problem that is easy to produce, and can improve the noise reduction effect. In addition, through the matching of each layer of spatial domain images and each layer of time domain images, the search range and accuracy of matching can be increased through layered matching, and the amount of calculation during matching is reduced, which improves the accuracy of the matching process. Noise effect improves image quality.

接下来，参考图2所示，对本公开实施例中的图像降噪方法中的每个步骤进行详细说明。Next, each step in the image noise reduction method in the embodiment of the present disclosure will be described in detail with reference to FIG. 2 .

在步骤S210中，获取当前图像以及所述当前图像的参考帧图像的时域降噪图像，并对所述时域降噪图像进行分解得到多层时域图像。In step S210, the temporal noise reduction image of the current image and the reference frame image of the current image is obtained, and the temporal noise reduction image is decomposed to obtain a multi-layer temporal image.

本公开实施例中，当前图像可以为当前帧图像，可以为通过终端的摄像模组对待拍摄物体进行拍摄得到的视频中的每一帧图像，也可以为从终端的存储器中获取的视频中的每一帧图像。In the embodiment of the present disclosure, the current image may be the current frame image, it may be each frame image in the video obtained by shooting the object to be photographed by the camera module of the terminal, or it may be an image in the video obtained from the terminal memory. every frame of image.

参考帧可以是当前图像的前n帧图像中的一帧或多帧，如可以为与当前图像相邻的前一帧。时域降噪图像可以为经过时域降噪得到的降噪图像。基于此，参考帧图像的时域降噪图像可以为前一帧的时域降噪后的图像。时域降噪是一种3D降噪方法，可利用多帧图像在时间上的相关性实现降噪。参考帧图像的时域降噪图像例如可以表示为B0。The reference frame may be one or more frames in the previous n frames of the current image, for example, it may be the previous frame adjacent to the current image. The temporal-domain noise-reduced image may be a noise-reduced image obtained through temporal-domain noise reduction. Based on this, the temporal noise-reduced image of the reference frame image may be the temporal-domain noise-reduced image of the previous frame. Temporal noise reduction is a 3D noise reduction method that can use the temporal correlation of multiple frames of images to achieve noise reduction. The temporal noise-reduced image of the reference frame image can be denoted as B0, for example.

在一些实施例中，为了增加搜索范围，可以将参考帧图像的时域降噪图像进行分解得到多层时域图像。示例性地，可以对参考帧图像的时域降噪图像进行金字塔分解。金字塔分解可以为高斯金字塔分解或者是拉普拉斯金字塔分解，此处以高斯金字塔分解为例进行说明。高斯金字塔的底层为待分解的图像，即参考帧图像的时域降噪图像，每向上一层则是通过与其相邻的下一层的图像进行高斯滤波和下采样得到，此处的下采样例如可以为1/2采样。In some embodiments, in order to increase the search range, the time-domain noise-reduced image of the reference frame image may be decomposed to obtain a multi-layer time-domain image. Exemplarily, pyramid decomposition may be performed on the time-domain noise-reduced image of the reference frame image. The pyramid decomposition can be a Gaussian pyramid decomposition or a Laplacian pyramid decomposition, and the Gaussian pyramid decomposition is taken as an example for illustration here. The bottom layer of the Gaussian pyramid is the image to be decomposed, that is, the temporal noise reduction image of the reference frame image, and each upper layer is obtained by performing Gaussian filtering and downsampling on the image of the next layer adjacent to it. The downsampling here For example, it can be 1/2 sampling.

参考图3中所示的金字塔的结构，金字塔包括分辨率依次降低的底层图像以及至少一个上层图像，底层图像即用于表示参考帧图像的时域降噪图像。至少一个上层图像则根据高斯滤波以及下采样得到。在一些实施例中，上层图像包括顶层图像及中间层图像，顶层图像为分辨率最低的上层图像，上层图像中除了顶层图像以外的图像均为中间层图像。例如图3所示，金字塔包括分辨率不同的多层图像，并且金字塔中任意一层图像的分辨率都比排列在其上一层的图像的分辨率大。即，按照金字塔从下向上的顺序而言，分辨率依次减小，即金字塔中底层图像的分辨率最大，顶层图像的分辨率最小。Referring to the structure of the pyramid shown in FIG. 3 , the pyramid includes bottom images with successively reduced resolutions and at least one upper image. The bottom image is a time-domain noise-reduced image used to represent a reference frame image. At least one upper-layer image is obtained according to Gaussian filtering and down-sampling. In some embodiments, the upper layer image includes a top layer image and a middle layer image, the top layer image is the upper layer image with the lowest resolution, and all the images in the upper layer image except the top layer image are middle layer images. For example, as shown in FIG. 3 , the pyramid includes multiple layers of images with different resolutions, and the resolution of an image at any layer in the pyramid is greater than that of the images arranged at the layer above it. That is, according to the order of the pyramid from bottom to top, the resolution decreases successively, that is, the resolution of the bottom image in the pyramid is the largest, and the resolution of the top image in the pyramid is the smallest.

需要说明的是，中间层图像的数量可以是一个，也可以是多个。当中间层图像的数量为多个时，排列在越靠近顶层图像的中间层图像的分辨率越小。中间层图像的数量具体根据高斯金字塔分解的次数来确定。例如，若进行四次高斯金字塔下采样，则可以得到四层图像，其中包含底层图像、2个中间层图像以及顶层图像。It should be noted that the number of intermediate layer images may be one or multiple. When there are multiple intermediate layer images, the resolution of the intermediate layer images arranged closer to the top layer image is smaller. The number of intermediate layer images is specifically determined according to the number of Gaussian pyramid decompositions. For example, if Gaussian pyramid downsampling is performed four times, four layers of images can be obtained, including a bottom layer image, two middle layer images and a top layer image.

基于此，在进行四次高斯金字塔下采样时，可以先将参考帧图像的时域降噪图像作为金字塔的底层图像，随后对底层图像进行高斯滤波及下采样，以获得分辨率比底层图像的分辨率小的第一中间层图像；对第一中间层图像进行高斯滤波及下采样，以获得分辨率比第一中间层图像的分辨率小的第二中间层图像。进一步对第二中间层图像进行高斯滤波及下采样，以获得分辨率比第二中间层图像的分辨率小的顶层图像。需要说明的是，在一些实施例中，在金字塔中上一层图像的长度及宽度可以为与其相邻的下一层图像的一半。Based on this, when performing four-time Gaussian pyramid downsampling, the temporal noise reduction image of the reference frame image can be used as the bottom image of the pyramid, and then the bottom image is Gaussian filtered and down-sampled to obtain the resolution ratio of the bottom image. A first intermediate layer image with a smaller resolution; Gaussian filtering and downsampling are performed on the first intermediate layer image to obtain a second intermediate layer image with a resolution smaller than that of the first intermediate layer image. Gaussian filtering and down-sampling are further performed on the second middle layer image to obtain a top layer image with a resolution smaller than that of the second middle layer image. It should be noted that, in some embodiments, the length and width of the image in the upper layer of the pyramid may be half of the image in the lower layer adjacent to it.

参考图3中所示，可以将参考帧图像的时域降噪图像B0进行四层高斯金字塔下采样得到多层时域图像b0、b1、b2、b3，其中，多层时域图像b0、b1、b2、b3的分辨率按照从下向上的方向依次减小，即b0的分辨率最大，b3的分辨率最小。Referring to Fig. 3, the time-domain noise reduction image B0 of the reference frame image can be down-sampled by a four-layer Gaussian pyramid to obtain multi-layer time-domain images b0, b1, b2, b3, wherein the multi-layer time-domain images b0, b1 The resolutions of , b2, and b3 decrease sequentially from bottom to top, that is, the resolution of b0 is the largest, and the resolution of b3 is the smallest.

在步骤S220中，对所述当前图像进行空域降噪得到所述当前图像对应的空域降噪图像，并对所述空域降噪图像进行分解得到多层空域图像。In step S220, perform spatial noise reduction on the current image to obtain a spatial noise reduction image corresponding to the current image, and decompose the spatial noise reduction image to obtain a multi-layer spatial image.

本公开实施例中，空域降噪可为一种2D降噪方法，只处理一帧图像内部的噪声。空域降噪可以通过各种具有保持边缘效果的空域降噪算法来实现。空域降噪算法可以包括但不限于NonLocalMean、双边滤波算法、或者其他基于深度学习的空域降噪算法。此处以双边滤波算法为例进行说明。In the embodiment of the present disclosure, the spatial noise reduction may be a 2D noise reduction method, which only processes noise inside a frame of image. Spatial noise reduction can be achieved by various spatial noise reduction algorithms with edge-preserving effects. The spatial denoising algorithm may include but not limited to NonLocalMean, bilateral filtering algorithm, or other deep learning-based spatial denoising algorithms. Here, the bilateral filtering algorithm is taken as an example for illustration.

双边滤波算法是一种非线性滤波器，它可以达到保持边缘、降噪平滑的效果。双边滤波也是采用加权平均的方法，用周边像素亮度值的加权平均代表某个像素的强度，所用的加权平均基于高斯分布。双边滤波器由两部分组成，一部分与像素空间距离相关，另一部分与像素点的像素差值相关。双边滤波器的权重等于空间邻近度因子和亮度相似度因子的乘积。空间邻近度因子为高斯滤波器系数，亮度相似度因子与空间像素差值相关。The bilateral filtering algorithm is a nonlinear filter, which can achieve the effect of maintaining edges and smoothing noise reduction. Bilateral filtering also uses a weighted average method, using the weighted average of the brightness values of surrounding pixels to represent the intensity of a certain pixel, and the weighted average used is based on the Gaussian distribution. The bilateral filter is composed of two parts, one part is related to the pixel space distance, and the other part is related to the pixel difference of the pixel point. The weight of the bilateral filter is equal to the product of the spatial proximity factor and the luminance similarity factor. The spatial proximity factor is a Gaussian filter coefficient, and the brightness similarity factor is related to the spatial pixel difference.

对于当前图像而言，可先进行空域降噪得到空域降噪图像，以初步降低噪声，提高后续匹配的准确度。例如，对当前图像A0进行空域降噪后可得到空域降噪图像A1。For the current image, spatial noise reduction can be performed first to obtain a spatial noise reduction image to initially reduce noise and improve the accuracy of subsequent matching. For example, after the spatial noise reduction is performed on the current image A0, the spatial noise reduction image A1 can be obtained.

本公开实施例中，可以对空域降噪图像进行金字塔分解，得到多层空域图像。金字塔分解可以为高斯金字塔分解或者是拉普拉斯金字塔分解，此处以高斯金字塔分解为例进行说明。高斯金字塔的底层为待分解的图像，即空域降噪图像，每向上一层则是通过高斯滤波和1/2下采样得到。In the embodiment of the present disclosure, a pyramid decomposition may be performed on the spatial domain noise reduction image to obtain a multi-layer spatial domain image. The pyramid decomposition can be a Gaussian pyramid decomposition or a Laplacian pyramid decomposition, and the Gaussian pyramid decomposition is taken as an example for illustration here. The bottom layer of the Gaussian pyramid is the image to be decomposed, that is, the spatial noise reduction image, and each upper layer is obtained by Gaussian filtering and 1/2 downsampling.

参考图4中所示，可以对当前图像对应的空域降噪图像A1进行四层高斯金字塔下采样得到分辨率不同的多层空域图像a0、a1、a2、a3，其中，a0的分辨率最大，a3的分辨率最小。通过将空域降噪图像进行金字塔分解得到对应的多层空域图像，有利于后续进行分层匹配时匹配尽可能大的运动范围以及减小计算量。Referring to Fig. 4, a four-layer Gaussian pyramid downsampling can be performed on the spatial noise reduction image A1 corresponding to the current image to obtain multi-layer spatial images a0, a1, a2, a3 with different resolutions, among which a0 has the largest resolution, a3 has the smallest resolution. By decomposing the spatial noise reduction image into a pyramid to obtain the corresponding multi-layer spatial image, it is beneficial to match the largest possible range of motion and reduce the amount of calculation in the subsequent layered matching.

在步骤S230中，将每层时域图像和每层空域图像进行匹配，确定所述时域降噪图像的对齐图像。In step S230, the time-domain image of each layer is matched with the space-domain image of each layer, and an aligned image of the temporal-domain noise-reduced image is determined.

本公开实施例中，在将多层时域图像和多层空域图像分解为多层图像后，可以将多层时域图像以及多层空域图像进行多次分层多尺度匹配。多次分层多尺度匹配用于对图像进行多次分层对齐。多次分层对齐指的是多次不同尺度的分层对齐。分层对齐可以用于对每层时域图像和每层空域图像均进行图像对齐。分层对齐是指按照分辨率的排列顺序，分别将每个层进行块匹配确定每个匹配块的运动矢量MV(Motion Vector)，从而根据运动矢量将多层时域图像以及多层空域图像中的每一层进行对齐。需要说明的是，每层时域图像和相同分辨率的每层空域图像进行图像对齐。例如，第三层空域图像和第三层时域图像进行图像对齐，第二层空域图像和第二层时域图像进行图像对齐等等。In the embodiment of the present disclosure, after decomposing the multi-layer time-domain image and the multi-layer air-domain image into multi-layer images, the multi-layer time-domain image and the multi-layer air domain image may be subjected to multiple layered and multi-scale matching. Multi-hierarchical multi-scale matching is used for multi-hierarchical alignment of images. Multiple hierarchical alignments refer to multiple hierarchical alignments of different scales. Hierarchical alignment can be used to perform image alignment on each layer of temporal images and each layer of spatial images. Hierarchical alignment refers to performing block matching on each layer in accordance with the order of resolution to determine the motion vector MV (Motion Vector) of each matching block, so that the multi-layer time domain image and the multi-layer air domain image are aligned according to the motion vector. Each layer is aligned. It should be noted that image alignment is performed on each layer of temporal domain images and each layer of spatial domain images with the same resolution. For example, image alignment is performed on the third-layer spatial domain image and the third-layer temporal domain image, image alignment is performed on the second-layer spatial domain image and the second-layer temporal domain image, and so on.

基于此，对多层时域图像以及多层空域图像中的每一层进行多次不同尺度的图像对齐时，多次不同尺度的图像对齐的次数可以根据实际需求进行设定，例如三次、四次等等。尺度可通过匹配块的大小而来表示，且匹配块的大小可以根据实际需求进行确定，例如可以为16×16，或者是为5×5等等。每次图像对齐的尺度可以只要满足图像对齐的尺度随着对齐次数逐渐减小即可。例如，按照金字塔分解将参考帧图像的时域降噪图像以及空域降噪图像分解为四层，且进行了三次图像对齐，第一次匹配块的大小为16×16，第二次匹配块的大小为8×8，第三次匹配块的大小为5×5。通过逐渐减小匹配块的大小，逐渐减小图像对齐的尺度，能够实现更精细的匹配过程，提高匹配的准确性和效率。Based on this, when multiple image alignments of different scales are performed on each layer of the multi-layer time domain image and the multi-layer air domain image, the number of image alignments of multiple different scales can be set according to actual needs, such as three times, four times, etc. time and so on. The scale can be represented by the size of the matching block, and the size of the matching block can be determined according to actual needs, for example, it can be 16×16, or 5×5, and so on. The scale of each image alignment can be satisfied as long as the scale of the image alignment gradually decreases with the number of alignments. For example, according to the pyramid decomposition, the temporal noise reduction image and the spatial noise reduction image of the reference frame image are decomposed into four layers, and image alignment is performed three times. The size of the first matching block is 16×16, and the second matching block’s The size is 8×8, and the size of the third matching block is 5×5. By gradually reducing the size of the matching block and gradually reducing the scale of image alignment, a finer matching process can be achieved, and the accuracy and efficiency of matching can be improved.

在进行分层多尺度的图像对齐时，可按照分辨率的排列顺序，依次对每层空域图像在每层时域图像中进行运动矢量估计，并根据运动矢量进行运动补偿以实现每层时域图像的图像对齐，直至进行多次不同尺度的图像对齐为止，以确定出对齐图像。其中，分辨率的排列顺序可以为分辨率由小到大的顺序，即从金字塔的顶层至底层的顺序。因此，可以按照分辨率由小到大的顺序，依次在每层时域图像中对每层空域图像进行运动矢量估计，确定每层空域图像相对于每层时域图像的移动，从而实现图像对齐。并且，每次进行分层图像对齐时，均是按照分辨率由小到大的顺序执行。例如，均是按照多层时域图像b3、b2、b1、b0的顺序进行运动矢量估计。When performing layered and multi-scale image alignment, the motion vector estimation of each layer of spatial domain images in each layer of time domain images can be performed in sequence according to the order of resolution, and motion compensation is performed according to the motion vectors to achieve each layer of time domain images. The images of the images are aligned until multiple image alignments of different scales are performed to determine an aligned image. Wherein, the arrangement order of the resolutions may be the order of the resolutions from small to large, that is, the order from the top layer to the bottom layer of the pyramid. Therefore, according to the order of resolution from small to large, the motion vector estimation of each layer of spatial domain images can be performed in each layer of time domain images in turn, and the movement of each layer of spatial domain images relative to each layer of time domain images can be determined, so as to achieve image alignment . Moreover, each time layered image alignment is performed, it is performed in order of resolution from small to large. For example, motion vector estimation is performed in the order of multi-layer time domain images b3, b2, b1, b0.

运动矢量用于表示不同图像之间的对应区域的位置变化，运动矢量可以通过块匹配方式来确定。块匹配方式可以将一帧图像划分为多个不重叠的图像块，例如N×N大小的图像块，然后每个图像块按照一定的匹配准则在前一帧图像的搜索窗口内查找最匹配的图像块，得到的位移差称为运动矢量。示例性地，可以将每层时域图像和每层空域图像均划分为多个不重叠的图像块，并假设一个图像块内所有的像素均进行速度相同的平移运动，且所有图像块的位移量相同。基于此，对于每层空域图像中的每一个图像块，均可以在每层时域图像的搜索窗口内寻找与其相似的图像块作为相似块，并认为相似块在每层时域图像中所处的位置就是每层空域图像中的图像块位移前的位置，通过相似块和图像块的坐标的差值可以计算出运动矢量。The motion vector is used to represent the position change of the corresponding area between different images, and the motion vector can be determined through block matching. The block matching method can divide a frame of image into multiple non-overlapping image blocks, such as N×N size image blocks, and then each image block finds the best matching image within the search window of the previous frame image according to certain matching criteria. image block, the resulting displacement difference is called a motion vector. Exemplarily, each layer of time-domain image and each layer of space-domain image can be divided into multiple non-overlapping image blocks, and it is assumed that all pixels in an image block perform translational motion at the same speed, and the displacement of all image blocks same amount. Based on this, for each image block in the spatial domain image of each layer, the image block similar to it can be found in the search window of each layer of time domain image as a similar block, and the similar block is considered to be located in each layer of time domain image The position of is the position of the image block in each layer of the spatial domain image before displacement, and the motion vector can be calculated by the difference between the coordinates of the similar block and the image block.

本公开实施例中，在进行每次分层对齐时，可以使用匹配块来进行运动矢量估计，并且每次分层对齐中每层所使用的匹配块相同，但是不同分层对齐过程中使用的匹配块不同。示例性地，分层对齐的匹配块的大小随着分层对齐次数的增加而减小。In the embodiment of the present disclosure, the matching block can be used for motion vector estimation during each layer alignment, and the matching block used by each layer in each layer alignment is the same, but the matching blocks used in different layer alignment processes The matching blocks are different. Exemplarily, the size of a hierarchically aligned matching block decreases as the number of hierarchical alignments increases.

可以将每层空域图像作为每层参考图像，将每层时域图像作为每层的待对齐图像。进一步，在进行第一次块匹配时，对当前层参考图像中的每个第一匹配块在当前层的待对齐图像中进行运动矢量估计，确定当前层参考图像的运动矢量图，以确定每层参考图像的运动矢量图。当前层可以为每层。The spatial domain image of each layer can be used as a reference image of each layer, and the temporal image of each layer can be used as an image to be aligned of each layer. Further, when performing block matching for the first time, motion vector estimation is performed on each first matching block in the current layer reference image in the current layer image to be aligned, and the motion vector vector of the current layer reference image is determined to determine each The motion vector map of the layer reference image. The current layer can be every layer.

在一些实施例中，每次块匹配过程中，在第三层以第三层空域图像a3为第三层参考图像，以第三层时域图像b3为第三层的待对齐图像，对第三层参考图像在第三层的待对齐图像b3做MV估计，具体估计方法可以包括：对第三层参考图像a3中每个第一匹配块blockA，在第三层的待对齐图像b3中的搜索窗口内搜索与第一匹配块最相似的块作为相似块blockB。其中，第一匹配块blockA是大小为R0×R0的块，R0×R0可以为15×15。搜索窗口的大小可以为W0×W0。假设找到的相似块blockB的坐标为(b_x,b_y)，第一匹配块blockA的坐标为(a_x，a_y)，则第一匹配块blockA在第三层的待对齐图像b3中的运动矢量为二者的坐标向量，可以表示为(b_x-a_x,b_y-a_y)。基于此，可以获取第三层空域图像中所有第一匹配块在第三层的待对齐图像中的运动矢量。按照上述方式进行重复，可以获取每层空域图像中的所有匹配块在每层时域图像中的运动矢量。In some embodiments, in each block matching process, in the third layer, the third layer spatial domain image a3 is used as the third layer reference image, and the third layer time domain image b3 is used as the third layer image to be aligned. The three-layer reference image performs MV estimation on the third-layer image b3 to be aligned. The specific estimation method may include: for each first matching block blockA in the third-layer reference image a3, the MV in the third-layer image b3 to be aligned Search for the block most similar to the first matching block within the search window as the similar block blockB. Wherein, the first matching block blockA is a block with a size of R0×R0, and R0×R0 may be 15×15. The size of the search window may be W0×W0. Suppose the coordinates of the found similar block blockB are (b_x, b_y), and the coordinates of the first matching block blockA are (a_x, a_y), then the motion vector of the first matching block blockA in the image b3 to be aligned on the third layer is two Or coordinate vector, can be expressed as (b_x-a_x, b_y-a_y). Based on this, the motion vectors of all the first matching blocks in the image to be aligned in the third layer in the image to be aligned in the third layer can be obtained. By repeating the above manner, motion vectors of all matching blocks in each layer of spatial domain image in each layer of time domain image can be obtained.

运动补偿描述前面一帧的每个小块移动到当前帧中的某个位置的过程，即用于描述每层待对齐图像中的每个相似块移动至每层空域图像的对应位置的过程。根据每个匹配块的运动矢量从目标层的待对齐图像中取出每个运动矢量处的图像块，逐块组成起来的过程称为运动补偿。运动补偿后得到的图像为对齐图像。Motion compensation describes the process of moving each small block of the previous frame to a certain position in the current frame, that is, it is used to describe the process of moving each similar block in each layer of image to be aligned to the corresponding position of each layer of spatial domain image. According to the motion vector of each matching block, the image block at each motion vector is taken out from the image to be aligned in the target layer, and the process of forming block by block is called motion compensation. The images obtained after motion compensation are aligned images.

举例而言，例如目标层参考图像a0中第一匹配块的运动矢量MV是2，目标层的待对齐图像b0中到第一匹配块的距离为2的图像块为相似块。为了补偿，根据第一匹配块的坐标和运动矢量得到补偿坐标，从目标层的待对齐图像b0中获取补偿坐标对应的图像块，得到新的图像以获取对齐图像。例如，第一匹配块的坐标为(100,100)，运动矢量为(2,2)，则两个坐标相加得到补偿坐标(102,102)，因此可以获取b0中补偿坐标(102,102)对应的图像块，根据这些图像块得到新的图像即实现对齐后的对齐图像，并且将其作为第一补偿图像。第一次运动补偿得到的对齐图像为第一补偿图像B01。本公开实施例中，通过运动补偿，能够对每层时域图像以及每层空域图像实现时域和空域的图像对齐，提高图像对齐的效果。For example, if the motion vector MV of the first matching block in the reference image a0 of the target layer is 2, the image block whose distance to the first matching block is 2 in the image b0 of the target layer to be aligned is a similar block. For compensation, the compensation coordinates are obtained according to the coordinates of the first matching block and the motion vector, the image block corresponding to the compensation coordinates is obtained from the image b0 of the target layer to be aligned, and a new image is obtained to obtain an aligned image. For example, the coordinates of the first matching block are (100,100), and the motion vector is (2,2), then the two coordinates are added to obtain the compensation coordinates (102,102), so the image block corresponding to the compensation coordinates (102,102) in b0 can be obtained, A new image is obtained according to these image blocks, that is, an aligned image after the alignment is achieved, and is used as the first compensation image. The aligned image obtained by the first motion compensation is the first compensated image B01. In the embodiments of the present disclosure, image alignment in the time domain and space domain can be implemented for each layer of time domain images and each layer of space domain images through motion compensation, thereby improving the effect of image alignment.

在此基础上，采用块匹配方式依次对每层空域图像进行运动矢量估计，并根据运动矢量进行运动补偿的过程可以通过三次块匹配实现，进而根据三次块匹配的匹配结果来确定对齐图像。图5中示意性示出了确定对齐图像的流程图，参考图5中所示，主要包括以下步骤：On this basis, the block matching method is used to estimate the motion vector of each layer of spatial domain image in turn, and the process of motion compensation according to the motion vector can be realized by three block matching, and then the alignment image is determined according to the matching result of the three block matching. Fig. 5 schematically shows a flow chart of determining an aligned image, referring to Fig. 5, which mainly includes the following steps:

在步骤S510中，依次对每层空域图像中的第一匹配块，在每层时域图像中通过第一次块匹配进行运动矢量估计，确定每层空域图像的运动矢量图，并根据所述运动矢量图进行运动补偿，以确定第一差分图像。In step S510, for the first matching block in each layer of spatial domain image, motion vector estimation is performed through the first block matching in each layer of time domain image, and the motion vector diagram of each layer of spatial domain image is determined, and according to the The motion vector map is motion compensated to determine the first difference image.

本步骤中，首先可以按照分辨率从小到大的排列顺序，依次对每层空域图像中的第一匹配块在每层时域图像中进行第一次块匹配，通过第一次块匹配进行第一匹配块的运动矢量估计，得到每个第一匹配块的运动矢量，并确定每层空域图像的运动矢量图，并基于运动矢量图进行运动补偿，从而确定出第一差分图像。In this step, firstly, the first matching blocks in each layer of spatial domain images can be sequentially performed on the first matching blocks in each layer of time domain images according to the order of resolution from small to large, and the first block matching can be performed through the first block matching. The motion vector estimation of a matching block obtains the motion vector of each first matching block, and determines the motion vector diagram of each layer of spatial domain images, and performs motion compensation based on the motion vector diagrams, so as to determine the first difference image.

其中，参考图6中所示，第一次块匹配的过程可以包括以下步骤，且图6中的步骤是步骤S510的具体实现方式，其中：Wherein, as shown in FIG. 6, the first block matching process may include the following steps, and the steps in FIG. 6 are specific implementations of step S510, wherein:

步骤S610，将当前层空域图像作为当前层参考图像，将当前层时域图像作为当前层的待对齐图像；Step S610, using the spatial domain image of the current layer as the reference image of the current layer, and using the time domain image of the current layer as the image to be aligned of the current layer;

步骤S620，对当前层参考图像中的每个第一匹配块在所述待对齐图像中进行运动矢量估计，确定当前层参考图像的运动矢量图，以确定每层参考图像的运动矢量图；Step S620, performing motion vector estimation on each first matching block in the reference image of the current layer in the image to be aligned, and determining the motion vector diagram of the reference image of the current layer, so as to determine the motion vector diagram of the reference image of each layer;

步骤S630，根据目标层参考图像的运动矢量图进行运动补偿，确定第一补偿图像；Step S630, performing motion compensation according to the motion vector diagram of the reference image of the target layer, and determining a first compensated image;

步骤S640，基于所述第一补偿图像确定所述第一差分图像。Step S640, determining the first difference image based on the first compensation image.

示例性地，当前层参考图像可以为每层参考图像，当前层的待对齐图像可以为每一层的待对齐图像。在第一次块匹配时，可在当前层的待对齐图像的搜索窗口内，根据当前层参考图像中每个第一匹配块的相似块，确定每个第一匹配块的运动矢量。每层对应的搜索窗口可以相同或不同。在一些实施例中，可以根据当前层参考图像是否为顶层参考图像，选择不同的方式在当前层的待对齐图像中来确定搜索窗口。如果当前层参考图像为顶层参考图像，则直接根据第一预设窗口来确定当前层的搜索窗口。第一预设窗口可以为W0×W0的窗口，且第一预设窗口可以直接配置，与其他参数之间不存在关联关系。如果当前层参考图像不为顶层参考图像，例如为中间层或底层，其运动估计可以在上一层运动估计的基础上进行，则可以结合上一层参考图像的运动矢量图、第一匹配块的坐标以及第二预设窗口来确定当前层的搜索窗口。此处的上一层参考图像指的是与当前层相邻且靠近顶层的参考图像，上一层参考图像的分辨率小于当前层参考图像。例如当前层参考图像为a1，则上一层参考图像为a2。具体地，可根据上一层参考图像中第一匹配块的运动矢量以及第一匹配块的坐标确定中心点，并基于所述中心点确定大小为第二预设窗口，以将其作为当前层的搜索窗口。第二预设窗口可以为W1×W1的窗口，其大小可以与第一预设窗口相同，也可以不同，具体根据实际需求进行配置。Exemplarily, the reference image of the current layer may be a reference image of each layer, and the image to be aligned of the current layer may be the image to be aligned of each layer. During the first block matching, the motion vector of each first matching block may be determined within the search window of the current layer image to be aligned according to the similar blocks of each first matching block in the current layer reference image. The search windows corresponding to each layer can be the same or different. In some embodiments, according to whether the reference image of the current layer is the reference image of the top layer, different methods may be selected to determine the search window in the images to be aligned of the current layer. If the reference image of the current layer is the reference image of the top layer, the search window of the current layer is directly determined according to the first preset window. The first preset window may be a window of W0×W0, and the first preset window may be directly configured without any correlation with other parameters. If the current layer reference image is not the top layer reference image, such as the middle layer or the bottom layer, its motion estimation can be performed on the basis of the motion estimation of the previous layer, then it can be combined with the motion vector diagram of the previous layer reference image, the first matching block coordinates and the second preset window to determine the search window of the current layer. Here, the upper layer reference image refers to the reference image adjacent to the current layer and close to the top layer, and the resolution of the upper layer reference image is smaller than that of the current layer reference image. For example, the reference image of the current layer is a1, and the reference image of the previous layer is a2. Specifically, the center point may be determined according to the motion vector of the first matching block in the reference image of the previous layer and the coordinates of the first matching block, and based on the center point, the size of the second preset window may be determined as the current layer search window. The second preset window may be a W1×W1 window, and its size may be the same as or different from that of the first preset window, which is specifically configured according to actual needs.

举例而言，在第二层中，若第一匹配块blockA的坐标为(a_x,a_y)，根据上一层参考图像得到第一匹配块blockA的运动矢量为(mv_x,mv_y)，则第二层的搜索窗口可以为在第二层的待对齐图像b2中，以(a_x+mv_x,a_y+mv_y)为中心，以W1×W1为大小的范围。通过类似的方式，可以计算在第一层b1以及底层b0的搜索窗口。For example, in the second layer, if the coordinates of the first matching block blockA are (a_x, a_y), and the motion vector of the first matching block blockA is (mv_x, mv_y) obtained according to the reference image of the previous layer, then the second The search window of a layer may be a range in the image b2 to be aligned in the second layer, with (a_x+mv_x, a_y+mv_y) as the center and W1×W1 as the size. In a similar manner, the search windows at the first layer b1 and the bottom layer b0 can be calculated.

在确定出搜索窗口后，可以基于该搜索窗口在每层待对齐图像中进行搜索，得到与第一匹配块相似的相似块。其中可以使用最小SAD(Sum of Absolute Difference，绝对误差和)、最小MAD(Mean Absolute Difference，平均绝对差值)或MSE(Mean SquaredError，平均评分误差)来确定相似块，具体可根据计算需求进行选择。用于搜索相似块的搜索窗口的大小和形状以及搜索每个相似块时偏移量的大小也可以根据实际需要设置，此处不作具体限定。After the search window is determined, a search may be performed in each layer of images to be aligned based on the search window to obtain a similar block similar to the first matching block. Among them, the minimum SAD (Sum of Absolute Difference, absolute error sum), minimum MAD (Mean Absolute Difference, mean absolute difference) or MSE (Mean Squared Error, mean score error) can be used to determine similar blocks, which can be selected according to calculation requirements . The size and shape of the search window used to search for similar blocks and the size of the offset when searching for each similar block can also be set according to actual needs, which are not specifically limited here.

基于相似块，可以根据相似块的坐标以及每个第一匹配块的坐标之间的差值表示的向量，确定出每层参考图像中的每个第一图像块在每层的待对齐图像中的运动矢量。进一步可以根据每层参考图像中每个第一匹配块的运动矢量组成的图，确定所述每层参考图像的运动矢量图。Based on the similar blocks, according to the coordinates of the similar blocks and the vector represented by the difference between the coordinates of each first matching block, it can be determined that each first image block in each layer of reference image is in each layer of the image to be aligned motion vector. Further, the motion vector diagram of the reference image of each layer may be determined according to the graph composed of the motion vectors of each first matching block in the reference image of each layer.

在此基础上，可以基于目标层参考图像的运动矢量图，确定目标层参考图像中每个第一匹配块的运动矢量；根据每个第一匹配块的运动矢量以及每个第一匹配块的坐标确定补偿坐标，从目标层的待对齐图像中获取与所述补偿坐标对应的图像块进行运动补偿，得到第一对齐图像。其中，目标层参考图像可以为底层参考图像，目标层的待对齐图像可以为底层时域图像。可以将每个第一匹配块的运动矢量以及每个第一匹配块的坐标进行相加得到补偿坐标。将底层待补齐图像中补偿坐标对应的图像块提取出来形成新的图像来确定对齐图像，对齐图像则为第一次块匹配得到的第一补偿图像。在进行运动补偿时，可将底层待对齐图像中补偿坐标处对应的图像块(像素块)放置于新的图像中的该位置处，即当前的第一匹配块的位置处进行运动补偿。On this basis, the motion vector of each first matching block in the target layer reference image can be determined based on the motion vector diagram of the target layer reference image; according to the motion vector of each first matching block and the motion vector of each first matching block The coordinates determine the compensation coordinates, and the image block corresponding to the compensation coordinates is acquired from the image to be aligned of the target layer to perform motion compensation to obtain the first aligned image. Wherein, the reference image of the target layer may be a bottom layer reference image, and the image to be aligned of the target layer may be a bottom layer temporal image. The motion vector of each first matching block and the coordinates of each first matching block may be added to obtain the compensation coordinates. The image blocks corresponding to the compensation coordinates in the underlying image to be filled are extracted to form a new image to determine the aligned image, and the aligned image is the first compensated image obtained by the first block matching. When performing motion compensation, the image block (pixel block) corresponding to the compensation coordinate in the underlying image to be aligned can be placed at this position in the new image, that is, the position of the current first matching block for motion compensation.

由于进行运动补偿得到的第一补偿图像比较粗糙，可能存在未完全对齐的情况，例如在纹理区域未对齐。因此可以基于第一补偿图像确定第一差分图像，以获取第一次块匹配阶段的输出结果。示例性地，可以根据所述第一补偿图像和所述目标层参考图像进行差分处理，获取所述第一差分图像。第一差分图像可以用于反映第一补偿图像和目标层参考图像的对齐效果。第一差分图像中数值为预设值的区域代表对齐区域，数值不为预设值的区域表示未对齐区域。预设值例如可以为0，用于表示区域的类型为对齐区域。Since the first compensated image obtained by performing motion compensation is relatively rough, there may be cases of incomplete alignment, such as misalignment in textured regions. Therefore, the first difference image can be determined based on the first compensation image to obtain the output result of the first block matching stage. Exemplarily, differential processing may be performed according to the first compensation image and the target layer reference image to acquire the first differential image. The first difference image may be used to reflect the alignment effect of the first compensation image and the reference image of the target layer. Areas in the first differential image whose values are preset values represent aligned areas, and areas whose values are not preset values represent unaligned areas. The preset value may be, for example, 0, which is used to indicate that the type of the region is an alignment region.

需要说明的是，在进行第一次块匹配时，第一匹配块R1的大小可以为比较大的数值，例如可以为16×16。使用较大的第一匹配块进行第一次块匹配，能够加快匹配的速度，提高匹配效率。It should be noted that, when performing block matching for the first time, the size of the first matching block R1 can be a relatively large value, for example, it can be 16×16. Using a larger first matching block for the first block matching can speed up the matching speed and improve the matching efficiency.

在一些实施例中，以a3/a2/a1/a0为每层参考图像，b3/b2/b1/b0则为每层待对齐图像。可以匹配块的大小为R，匹配块之间的间隔也为R为例，对第一次块匹配的过程进行详细介绍，具体可参考图7中所示：In some embodiments, a3/a2/a1/a0 are used as reference images for each layer, and b3/b2/b1/b0 are images to be aligned for each layer. The size of the matching block can be R, and the interval between matching blocks is also R as an example. The process of the first block matching is described in detail, as shown in Figure 7:

步骤S701，在第三层中以第三层空域图像a3为参考图像，基于第三层的待对齐图像b3做运动矢量估计，具体过程可以包括：对第三层参考图像a3中每个大小为R1×R1的第一匹配块blockA在第三层的待对齐图像b3中，大小为W0×W0的搜索窗口内搜索与第一搜索块最相似的相似块，假设找到的相似块blockB的坐标为(b_x,b_y)，第一匹配块blockA的坐标为(a_x，a_y)，则第一匹配块blockA在第三层的待对齐图像b3中的运动矢量为(b_x-a_x,b_y-a_y)，所有的第一匹配块都遍历完成后得到了第三层参考图像中每个第一匹配块的运动矢量组成的图，即第三层参考图像a3的运动矢量图MVmap3。Step S701, in the third layer, use the third layer spatial domain image a3 as a reference image, and perform motion vector estimation based on the third layer image b3 to be aligned. The specific process may include: for each size of the third layer reference image a3 The first matching block blockA of R1×R1 searches for the similar block most similar to the first search block in the search window of size W0×W0 in the image b3 to be aligned on the third layer, assuming that the coordinates of the found similar block blockB are (b_x, b_y), the coordinates of the first matching block blockA are (a_x, a_y), then the motion vector of the first matching block blockA in the image b3 to be aligned on the third layer is (b_x-a_x, b_y-a_y), After traversing all the first matching blocks, a map composed of motion vectors of each first matching block in the third layer reference image is obtained, that is, the motion vector map MVmap3 of the third layer reference image a3.

步骤S702，在估计第二层图像的运动矢量时，以第二层空域图像a2为参考图像进行运动估计，但是第二层图像的运动估计是在上一层(第三层)运动估计的基础上进行。例如第一匹配块blockA的坐标为(a_x，a_y)，根据第三层参考图像a3的运动矢量图MVmap3得到第一匹配块的运动矢量为(mv_x,mv_y)，则可以在第二层的待对齐图像b2中以(a_x+mv_x,a_y+mv_y)为中心，以W1×W1为大小的搜索窗口内进行第二层的运动估计。获取相似块以及每个第一匹配块的运动矢量的计算方式可以与第三层的运动矢量的计算方式相同，此处不再赘述。并且，第二层参考图像a2中的每个第一匹配块的运动矢量都计算完成后，可以得到第二层参考图像a2的运动矢量图MVmap2。Step S702, when estimating the motion vector of the second layer image, the motion estimation is performed with the second layer spatial domain image a2 as the reference image, but the motion estimation of the second layer image is based on the motion estimation of the upper layer (third layer) Carried on. For example, the coordinates of the first matching block blockA are (a_x, a_y), and the motion vector of the first matching block is (mv_x, mv_y) obtained according to the motion vector map MVmap3 of the third layer reference image a3, then it can be obtained in the second layer to be The motion estimation of the second layer is performed in the search window with the size of W1×W1 centered on (a_x+mv_x, a_y+mv_y) in the alignment image b2. The calculation method of acquiring similar blocks and the motion vector of each first matching block may be the same as the calculation method of the motion vector of the third layer, which will not be repeated here. And, after the motion vector of each first matching block in the second layer reference image a2 is calculated, the motion vector vector MVmap2 of the second layer reference image a2 can be obtained.

类似地，步骤S703，参考第二层参考图像a2的运动矢量图MVmap2计算出第一层参考图像a1的运动矢量图MVmap1。Similarly, in step S703, refer to the motion vector map MVmap2 of the second layer reference image a2 to calculate the motion vector map MVmap1 of the first layer reference image a1.

步骤S704，参考第一层参考图像a1的运动矢量图Mvmap1计算出第零层参考图像(底层参考图像)a0的运动矢量图Mvmap0，从而计算出底层参考图像a0每个第一匹配块在底层待对齐图像b0中的位移。Step S704, refer to the motion vector map Mvmap1 of the first layer reference image a1 to calculate the motion vector map Mvmap0 of the zeroth layer reference image (bottom reference image) a0, thereby calculating the motion vector map Mvmap0 of each first matching block in the bottom layer reference image a0 Align the displacement in image b0.

步骤S705，根据每个第一匹配块的运动矢量从底层待对齐图像b0中取出每个运动矢量处的图像块，逐块组成起来以进行第一次运动补偿，第一次运动补偿后得到的图像为第一补偿图像B01。Step S705, according to the motion vector of each first matching block, the image block at each motion vector is taken out from the underlying image b0 to be aligned, and assembled block by block to perform the first motion compensation, and the obtained after the first motion compensation The image is the first compensated image B01.

步骤S706，用第一补偿图像B01与底层参考图像a0做差分处理后得到第一差分图像Diffmask1，第一差分图像Diffmask1可以反映第一补偿图像B01与底层参考图像a0的对齐效果。具体地，差分处理可以为对像素进行相减，以削弱相似部分，突出不同部分。Diffmask1中值为0的区域表示该区域对齐完成，反之数值为非0说明该区域对齐失败。In step S706, the first difference image Diffmask1 is obtained after difference processing between the first compensation image B01 and the underlying reference image a0. The first difference image Diffmask1 can reflect the alignment effect between the first compensation image B01 and the bottom reference image a0. Specifically, the differential processing may be to subtract pixels to weaken similar parts and highlight different parts. An area with a value of 0 in Diffmask1 indicates that the alignment of the area is completed, otherwise a value other than 0 indicates that the alignment of the area fails.

通过步骤S701至步骤S706的第一次块匹配，可以对多层时域图像和多层空域图像的每一层进行图像匹配对齐，得到第一差分图像。通过分层进行匹配，增加了匹配范围且减小了计算量。并且，可以通过第一匹配块实现快速匹配。Through the first block matching from step S701 to step S706, image matching and alignment can be performed on each layer of the multi-layer temporal image and the multi-layer spatial image to obtain the first difference image. Matching is performed by layering, which increases the matching range and reduces the amount of calculation. Moreover, fast matching can be realized through the first matching block.

在步骤S520中，以所述第一差分图像为基础，依次对每层空域图像的第二匹配块在每层时域图像中通过第二次块匹配进行运动矢量估计，确定每层空域图像的运动矢量图，并根据所述运动矢量图进行运动补偿，以确定第二差分图像。In step S520, based on the first difference image, the motion vector estimation is performed on the second matching block of each layer of the spatial domain image in the time domain image of each layer through the second block matching to determine the a motion vector diagram, and perform motion compensation according to the motion vector diagram to determine a second difference image.

本公开实施例中，由于第一次块匹配已经对齐了部分区域，因此为了减少计算量，只需要对第一次块匹配中未对齐的区域进行处理。基于此，可以第一次块匹配得到的第一差分图像作为指导，只对第一差分图像对应的非对齐区域进行精细匹配。具体地，可根据第一差分图像确定每层时域图像的第一非对齐区域，第一非对齐区域指的是以第一差分图像为指导而确定的非对齐区域，具体可根据第一差分图像中数值为0的区域确定非对齐区域。In the embodiment of the present disclosure, since the first block matching has already aligned some regions, in order to reduce the amount of calculation, only the unaligned regions in the first block matching need to be processed. Based on this, the first difference image obtained by the first block matching can be used as a guide, and only the non-aligned area corresponding to the first difference image is finely matched. Specifically, the first non-alignment area of each layer of time-domain image can be determined according to the first difference image, the first non-alignment area refers to the non-alignment area determined with the guidance of the first difference image, specifically according to the first difference Areas with a value of 0 in the image determine non-alignment areas.

在第二次块匹配过程中，可根据第二匹配块进行运动矢量估计。第二匹配块R2的大小可以小于第一匹配块R1，R2×R2例如可以为8×8。通过较小尺寸的第二匹配块进行分层匹配，能够实现精细匹配。During the second block matching process, motion vector estimation can be performed according to the second matching block. The size of the second matching block R2 may be smaller than that of the first matching block R1, and R2×R2 may be 8×8, for example. Fine matching can be realized by performing hierarchical matching through the second matching block with a smaller size.

第二次块匹配的过程与第一次块匹配的过程基本相同，主要包括以下步骤：The process of the second block matching is basically the same as that of the first block matching, mainly including the following steps:

将每层空域图像作为每层参考图像，将每层时域图像作为每层待对齐图像。根据第一差分图像从每层待对齐图像中确定第一非对齐区域。依次对每层参考图像中的第二匹配块在所述第一非对齐区域进行第二次块匹配，根据每个第二匹配块在每层参考图像中的相似块确定每个第二匹配块的运动矢量，根据每个第二匹配块的运动矢量组成的图确定每层参考图像在每层时域图像中的运动矢量图。The spatial domain image of each layer is used as the reference image of each layer, and the time domain image of each layer is used as the image to be aligned of each layer. A first non-alignment area is determined from each layer of images to be aligned according to the first difference image. performing a second block matching on the second matching blocks in the reference image of each layer in turn in the first non-alignment area, and determining each second matching block according to the similar blocks of each second matching block in the reference image of each layer The motion vector of each layer of the reference image in the time domain image of each layer is determined according to the graph formed by the motion vector of each second matching block.

根据底层参考图像表示的目标层参考图像的运动矢量图，确定出每个第二匹配块的运动矢量，根据运动矢量以及第二匹配块的坐标之和确定出补偿坐标，并根据补偿坐标从目标层待对齐图像中获取图像块进行运动补偿，确定未完全对齐的第二补偿图像。According to the motion vector diagram of the target layer reference image represented by the bottom layer reference image, the motion vector of each second matching block is determined, the compensation coordinate is determined according to the sum of the motion vector and the coordinates of the second matching block, and the compensation coordinate is obtained from the target Obtain image blocks from the image to be aligned in the layer to perform motion compensation, and determine a second compensated image that is not completely aligned.

根据所述第二补偿图像以及所述目标层参考图像进行差分处理，获取所述第二差分图像。第二差分图像中可能包括对齐区域和非对齐区域，但是其中包含的非对齐区域小于第一差分图像中的非对齐区域，且非对齐区域依然可以为数值0所表示的区域。Perform difference processing according to the second compensation image and the target layer reference image to acquire the second difference image. The second differential image may include an aligned area and an unaligned area, but the unaligned area contained therein is smaller than the unaligned area in the first differential image, and the unaligned area may still be an area represented by a value of 0.

在第二次块匹配时，可基于第一差分图像进行匹配。由于第一差分图像是根据目标层参考图像得到，对于其它层而言，可以将第一差分图像按照每一层与目标层的分辨率关系进行缩放得到每层对应的差分图像，从而根据每层对应的差分图像对每层空域图像和每层时域图像进行对齐。其中，分辨率关系可以为底层参考图像与中间层以及顶层之间的分辨率的对应关系。During the second block matching, the matching may be performed based on the first difference image. Since the first difference image is obtained based on the reference image of the target layer, for other layers, the first difference image can be scaled according to the resolution relationship between each layer and the target layer to obtain the corresponding difference image of each layer, so that according to each layer The corresponding difference image aligns the spatial domain image of each layer with the temporal image of each layer. Wherein, the resolution relationship may be a corresponding relationship between the resolutions of the bottom layer reference image and the middle layer and the top layer.

在一些实施例中，在第二次块匹配对齐时则只需要对第一次块匹配对齐失败的区域再做精细对齐即可。参考图8中所示，第二次块匹配以第一差分图像Diffmask1为指导，只对第一差分图像Diffmask1中的非0区域做匹配。第二次块匹配的过程包括步骤S801至步骤S806，具体执行步骤与步骤S701至步骤S706的步骤相同，此处不再赘述。In some embodiments, during the second block matching and alignment, it is only necessary to perform fine alignment on the area where the block matching and alignment failed in the first time. Referring to FIG. 8 , the second block matching is guided by the first differential image Diffmask1, and only the non-zero regions in the first differential image Diffmask1 are matched. The second block matching process includes steps S801 to S806, and the specific execution steps are the same as those in steps S701 to S706, and will not be repeated here.

需要注意的是，第二次块匹配时，第二匹配块的尺寸R2×R2小于第一匹配块的尺寸，以根据小尺寸的匹配块在第一次块匹配的基础上对非对齐区域实现更为精细的匹配。完成运动补偿后得到第二次分层对齐的图像，即第二补偿图像B02。进一步通过第二补偿图像B02与目标层参考图像a0做差分处理后得到新的差分图像，即第二差分图像Diffmask2。It should be noted that during the second block matching, the size R2×R2 of the second matching block is smaller than the size of the first matching block, so that the non-aligned area can be realized on the basis of the first block matching based on the small-sized matching block. finer matching. After the motion compensation is completed, the second hierarchically aligned image is obtained, that is, the second compensated image B02. Further, a new differential image, namely, the second differential image Diffmask2 is obtained after differential processing is performed on the second compensation image B02 and the target layer reference image a0.

在步骤S530中，基于所述第二差分图像，依次对每层参考图像的第三匹配块在每层时域图像中进行第三次块匹配，确定每层参考图像的运动矢量图，并根据所述运动矢量图进行运动补偿，以确定所述对齐图像。In step S530, based on the second difference image, the third block matching is performed on the third matching block of each layer of reference image in the time domain image of each layer in order to determine the motion vector diagram of each layer of reference image, and according to Motion compensation is performed on the motion vector diagram to determine the aligned image.

本公开实施例中，可以第二差分图像为指导进行第三次块匹配，且第三次块匹配所使用的第三匹配块的尺寸最小。In the embodiment of the present disclosure, the third block matching can be performed based on the second difference image, and the size of the third matching block used in the third block matching is the smallest.

由于第一次块匹配和第二次块匹配已经对齐了部分区域，因此为了减少计算量，第三次块匹配只需要对第二次块匹配中未对齐的区域进行处理。具体地，可根据第二差分图像确定每层时域图像的第二非对齐区域，第二非对齐区域指的是以第二差分图像为指导而确定的非对齐区域，具体可根据第二差分图像中数值为0的区域确定，且可以与第一非对齐区域不同。Since the first block matching and the second block matching have already aligned some regions, in order to reduce the amount of computation, the third block matching only needs to process the unaligned regions in the second block matching. Specifically, the second non-alignment area of each layer of time-domain image can be determined according to the second difference image, the second non-alignment area refers to the non-alignment area determined with the guidance of the second difference image, specifically according to the second difference The region with a value of 0 in the image is determined and may be different from the first non-aligned region.

在第三次块匹配时，基于第二差分图像进行匹配。由于第二差分图像是根据目标层参考图像得到，对于其它层而言，可以将第二差分图像按照分辨率关系进行缩放得到其它层对应的差分图像，从而根据每层对应的差分图像对每层空域图像和每层时域图像进行对齐。其中，分辨率关系可以为底层参考图像与其它层参考图像之间的分辨率的对应关系。During the third block matching, the matching is performed based on the second difference image. Since the second difference image is obtained based on the reference image of the target layer, for other layers, the second difference image can be scaled according to the resolution relationship to obtain the difference images corresponding to other layers, so that each layer can be processed according to the difference image corresponding to each layer The spatial domain image is aligned with the temporal image of each layer. Wherein, the resolution relationship may be a corresponding relationship between resolutions of the bottom layer reference image and other layer reference images.

在第三次块匹配过程中，可根据第三匹配块进行运动矢量估计。第三匹配块R3的大小可以小于第二匹配块R2，R3×R3例如可以为5×5。通过较小尺寸的第三匹配块进行分层匹配，能够实现精细匹配。During the third block matching process, motion vector estimation can be performed according to the third matching block. The size of the third matching block R3 may be smaller than that of the second matching block R2, and R3×R3 may be 5×5, for example. Fine matching can be achieved by performing hierarchical matching through the third matching block with a smaller size.

第三次块匹配的过程与第二次块匹配的过程基本相同，主要包括以下步骤：The process of the third block matching is basically the same as that of the second block matching, mainly including the following steps:

首先确定第二非对齐区域，依次对每层参考图像中的第三匹配块在所述第二非对齐区域进行第三次块匹配，根据每个第三匹配块在每层参考图像中的相似块确定每个第三匹配块的运动矢量，根据每个第三匹配块的运动矢量组成的图确定每层参考图像在每层时域图像中的运动矢量图。First determine the second non-alignment area, sequentially perform block matching on the third matching block in each layer of reference image in the second non-alignment area, according to the similarity of each third matching block in each layer of reference image The block determines the motion vector of each third matching block, and determines the motion vector diagram of each layer of reference image in each layer of time domain image according to the graph composed of the motion vectors of each third matching block.

根据底层参考图像表示的目标层参考图像的运动矢量图，确定出每个第三匹配块的运动矢量，根据运动矢量以及第三匹配块的坐标之和确定出补偿坐标，并根据补偿坐标从目标层待对齐图像中获取图像块进行运动补偿，确定未完全对齐的第三补偿图像。需要说明的是，在进行第三次块匹配时，由于是最后一次进行匹配，因此需要将所有区域对齐，而不再获取差分图像，直接将运动补偿得到的第三补偿图像作为对齐图像。该对齐图像指的则是参考帧图像的时域降噪图像以及空域降噪图像最终对齐后的图像。According to the motion vector diagram of the target layer reference image represented by the bottom layer reference image, the motion vector of each third matching block is determined, the compensation coordinate is determined according to the sum of the motion vector and the coordinates of the third matching block, and the compensation coordinate is obtained from the target The image block is obtained from the image to be aligned in the layer for motion compensation, and the third compensated image that is not completely aligned is determined. It should be noted that when the third block matching is performed, since it is the last matching, all regions need to be aligned, and the difference image is no longer obtained, and the third compensation image obtained by motion compensation is directly used as the alignment image. The aligned image refers to a final aligned image of the temporal noise reduction image of the reference frame image and the spatial noise reduction image.

参考图9中所示，第三次块匹配以第二差分图像Diffmask2为指导，只对第二差分图像Diffmask2中的非0区域做匹配。第三次块匹配的过程包括步骤S901至步骤S905，具体执行步骤与步骤S801至步骤S805的步骤相同，此处不再赘述。需要注意的是，第三次块匹配时，第三匹配块的尺寸R3×R3小于第一匹配块的尺寸且小于第二匹配块的尺寸，以根据小尺寸的匹配块实现更为精细的匹配。完成运动补偿后得到第三次分层对齐的图像，即第三补偿图像B03，并且可以将第三补偿图像确定为最终的对齐图像B1。Referring to FIG. 9 , the third block matching is guided by the second differential image Diffmask2, and only the non-zero regions in the second differential image Diffmask2 are matched. The third block matching process includes steps S901 to S905, and the specific execution steps are the same as those in steps S801 to S805, and will not be repeated here. It should be noted that during the third block matching, the size R3×R3 of the third matching block is smaller than the size of the first matching block and smaller than the size of the second matching block, so as to achieve a finer matching based on the small size matching block . After the motion compensation is completed, the third layer-aligned image, that is, the third compensated image B03 is obtained, and the third compensated image can be determined as the final aligned image B1.

需要补充的是，在进行运动补偿时，首先可以确定每层空域图像中匹配块对应的补偿块。补偿块指的是用于进行运动补偿的图像块，可以包括匹配块中的部分区域或全部区域。对于不同的块匹配过程而言，其补偿块的大小也不同，因此可以根据块匹配的次数来选择不同的方式确定补偿块。示例性地，在第一次块匹配以及第二次块匹配时，将所述对应的匹配块的全部区域作为补偿块。具体地，在第一次块匹配时，将第一匹配块中的全部区域作为补偿块；在第二次块匹配时，将第二匹配块中的全部区域作为补偿块。举例而言，基于第一匹配块或第二匹配块的运动矢量，从目标层的待对齐图像中获取补偿坐标处对应的图像块，将补偿坐标处对应的图像块(像素块)放置于新的图像中的该位置处，即当前的第一匹配块或第二匹配块的位置处，对与第一匹配块或第二匹配块大小相同的补偿块进行运动补偿。例如，第一匹配块为16×16，则补偿时也对16×16的第一匹配块进行补偿。What needs to be added is that when performing motion compensation, the compensation block corresponding to the matching block in each layer of spatial domain image may be determined first. The compensation block refers to an image block used for motion compensation, and may include a part or all of the areas in the matching block. For different block matching processes, the size of the compensation block is also different, so different ways can be selected to determine the compensation block according to the times of block matching. Exemplarily, during the first block matching and the second block matching, the entire area of the corresponding matching block is used as the compensation block. Specifically, in the first block matching, all areas in the first matching block are used as compensation blocks; in the second block matching, all areas in the second matching block are used as compensation blocks. For example, based on the motion vector of the first matching block or the second matching block, the image block corresponding to the compensation coordinate is obtained from the image to be aligned of the target layer, and the image block (pixel block) corresponding to the compensation coordinate is placed in the new At this position in the image, that is, at the position of the current first matching block or the second matching block, motion compensation is performed on a compensation block having the same size as the first matching block or the second matching block. For example, if the first matching block is 16×16, the first matching block of 16×16 is also compensated during compensation.

除此之外，在第三次块匹配时，将第三次块匹配对应的匹配块的部分区域作为补偿块，并按照偏移量更新所述补偿块。具体地，可以将第三匹配块中的部分区域作为补偿块，例如第三匹配块为5×5，可以将第三匹配块中心的第四匹配块作为补偿块，第四匹配块的尺寸可以小于第三匹配块，例如可以为2×2。进一步地，由于补偿块小于第三匹配块，因此可以根据偏移量对补偿块进行更新。偏移量用于表示补偿块移动的步长，偏移量具体可以根据实际需求进行设定，例如可以为2个像素。基于此，可以在预设方向上按照偏移量得到新的补偿块的中心，将新的补偿块的中心向每个方向进行扩展得到新的第三匹配块，从而得到新的补偿块，即新的2×2的第四匹配块，直至遍历完整幅图像为止。预设方向可以根据实际需求确定，例如水平方向或垂直方向。进一步，可基于第三匹配块的运动矢量从目标层待对齐图像中获取补偿坐标处对应的图像块，对2×2的补偿块进行运动补偿，以实现精细补偿。In addition, during the third block matching, a part of the matching block corresponding to the third block matching is used as a compensation block, and the compensation block is updated according to the offset. Specifically, a part of the third matching block can be used as a compensation block, for example, the third matching block is 5×5, and the fourth matching block at the center of the third matching block can be used as a compensation block, and the size of the fourth matching block can be smaller than the third matching block, for example, it may be 2×2. Further, since the compensation block is smaller than the third matching block, the compensation block can be updated according to the offset. The offset is used to indicate the step size of the compensation block movement, and the offset can be set according to actual needs, for example, it can be 2 pixels. Based on this, the center of the new compensation block can be obtained according to the offset in the preset direction, and the center of the new compensation block can be extended to each direction to obtain a new third matching block, thereby obtaining a new compensation block, namely A new 2×2 fourth matching block until the entire image is traversed. The preset direction can be determined according to actual needs, such as horizontal direction or vertical direction. Further, based on the motion vector of the third matching block, the image block corresponding to the compensation coordinates can be obtained from the target layer image to be aligned, and motion compensation can be performed on the 2×2 compensation block to achieve fine compensation.

在第一次块匹配和第二次块匹配时，通过与匹配块相同尺寸的补偿块进行补偿，能够提高运动补偿的效率。第三次块匹配时，通过更小尺寸的补偿块进行补偿，能够实现精细补偿，提高补偿的准确性，实现快速精准补偿的功能。During the first block matching and the second block matching, the compensation block with the same size as the matching block is used for compensation, which can improve the efficiency of motion compensation. In the third block matching, compensation is performed through a smaller-sized compensation block, which can realize fine compensation, improve the accuracy of compensation, and realize the function of fast and accurate compensation.

本公开实施例中，通过对时域降噪图像和空域降噪图像进行金字塔分解，执行多次尺度不同的分层匹配，将时域降噪图像对齐至空域降噪图像，用多层匹配的方式增加了匹配时的搜索范围，减少了搜索的计算量。与此同时，通过不同尺度的多次分层匹配，能够提升每层匹配的准确性，通过逐渐减小的尺度进行分层匹配，提升了图像匹配的精确度。分层匹配通过多分辨率由粗到细的搜索思路，在不同分辨率表示的不同层上采用不同大小的匹配块，通过递归的方法采用不同尺度估计运动矢量。第一层由于用大块和大搜索窗，可以可靠地估计块运动的基本方向，对于下一层则在上层运动矢量的基础上做更精确的估值。由于上一层保证了运动矢量整体上的可靠性，因此，可克服小块匹配造成的误匹配问题，提高了匹配的精准性。除此之外，由于在进行降噪时，通过估计匹配块的运动矢量考虑到图像中运动区域的降噪，因此能够实现在所有区域的降噪。In the embodiment of the present disclosure, by performing pyramid decomposition on the time-domain noise-reduced image and the spatial-domain noise-reduced image, multiple layers of matching with different scales are performed, the temporal-domain noise-reduced image is aligned to the spatial-domain noise-reduced image, and the multi-layer matching The method increases the search scope during matching and reduces the calculation amount of search. At the same time, the accuracy of each layer of matching can be improved through multiple layered matching at different scales, and the accuracy of image matching can be improved through layered matching at gradually reduced scales. Hierarchical matching adopts a multi-resolution search idea from coarse to fine, uses matching blocks of different sizes on different layers represented by different resolutions, and uses different scales to estimate motion vectors through a recursive method. Because the first layer uses a large block and a large search window, it can reliably estimate the basic direction of block motion. For the next layer, a more accurate estimate is made on the basis of the motion vector of the upper layer. Since the upper layer guarantees the overall reliability of the motion vector, it can overcome the mismatching problem caused by small block matching and improve the matching accuracy. In addition, since the noise reduction of the moving area in the image is considered by estimating the motion vector of the matching block when performing the noise reduction, the noise reduction in all areas can be realized.

在步骤S240中，将所述空域降噪图像以及所述对齐图像进行融合，获取所述当前图像对应的降噪图像。In step S240, the spatial noise reduction image and the aligned image are fused to obtain a noise reduction image corresponding to the current image.

本公开实施例中，可以对空域降噪图像以及频域降噪图像的对齐图像进行融合，实现空时域融合，对运动区域和非运动区域进行时域降噪。In the embodiment of the present disclosure, the aligned images of the spatial noise reduction image and the frequency domain noise reduction image can be fused to realize space-time domain fusion, and time domain noise reduction can be performed on the moving area and the non-moving area.

图10中示意性示出了进行融合的流程图，参考图10中所示，主要包括以下步骤：Fig. 10 schematically shows a flow chart for fusion, referring to Fig. 10, it mainly includes the following steps:

在步骤S1001中，将所述空域降噪图像1010以及所述对齐图像1020进行运动检测，确定差异图1030；In step S1001, motion detection is performed on the spatialnoise reduction image 1010 and the alignedimage 1020, and adifference map 1030 is determined;

在步骤S1002中，根据所述差异图确定融合权重图1040；In step S1002, afusion weight map 1040 is determined according to the difference map;

在步骤S1003中，按照所述融合权重图对所述空域降噪图像以及所述对齐图像进行融合，获取所述降噪图像1050。In step S1003, the spatial noise reduction image and the aligned image are fused according to the fusion weight map to obtain thenoise reduction image 1050 .

其中，可以对所述空域降噪图像和所述对齐图像分别进行平滑处理；将平滑后的空域降噪图像以及平滑后的对齐图像相减得到差分图，并对所述差分图进行形态学处理，获取所述差异图。具体地，形态学处理可以为去除散点以及孔洞填充。基于此，例如把空域降噪图像1010以及对齐图像1020分别做平滑后相减得到二者的差分图，再对差分图做形态学处理去除孤立的散点并填充孔洞后就得到差异图。Wherein, the spatial noise reduction image and the alignment image may be respectively smoothed; the smoothed spatial noise reduction image and the smoothed alignment image are subtracted to obtain a difference map, and the difference map is subjected to morphological processing , to obtain the difference map. Specifically, the morphological processing can be removing scattered points and filling holes. Based on this, for example, the spatial domainnoise reduction image 1010 and the alignedimage 1020 are smoothed and then subtracted to obtain a difference map of the two, and then the difference map is morphologically processed to remove isolated scattered points and fill holes to obtain a difference map.

接下来以差异图Motion Mask为指导生成融合权重图Alpha Map，融合权重图用于表示空域降噪图像以及时域降噪图像的融合程度。融合原则是：差异图的值越小说明空域降噪图像以及时域降噪图像越相似，则给融合权重图的值越大；反之差异图的值越大说明空域降噪图像以及时域降噪图像的差异越大，则给融合权重图的值越小。Next, the fusion weight map Alpha Map is generated under the guidance of the difference map Motion Mask, and the fusion weight map is used to indicate the degree of fusion of the spatial domain noise reduction image and the temporal domain noise reduction image. The fusion principle is: the smaller the value of the difference map, the more similar the spatial noise reduction image and the temporal noise reduction image are, and the larger the value of the fusion weight map is; The greater the difference of the noisy image, the smaller the value given to the fusion weight map.

基于此，可以按照融合权重图进行融合。具体地，融合策略为降噪图像C0＝A1*(1.0-AlphaMap)+B1*AlphaMap，其中降噪图像C0即是当前图像的时域降噪后的图像。在此基础上，当前图像的时域降噪后的降噪图像在下一帧时域降噪时又会作为下一帧的待对齐图像，即下一帧的时域降噪图像，从而循环整个过程，对所有帧进行空时域结合降噪，得到每一帧图像的时域降噪结果。Based on this, fusion can be performed according to the fusion weight map. Specifically, the fusion strategy is noise-reduced image C0=A1*(1.0-AlphaMap)+B1*AlphaMap, where the noise-reduced image C0 is the temporal-domain noise-reduced image of the current image. On this basis, the noise-reduced image after the temporal noise reduction of the current image will be used as the image to be aligned in the next frame, that is, the temporal noise-reduced image of the next frame, thus looping through the entire In the process, all frames are combined with space-time domain noise reduction to obtain the temporal domain noise reduction result of each frame image.

图11中示意性示出了进行空时域结合降噪的流程图，参考图11中所示，主要包括以下步骤：Figure 11 schematically shows a flow chart of performing space-time domain combined noise reduction, referring to what is shown in Figure 11, it mainly includes the following steps:

在步骤S1101中，获取当前图像A0。In step S1101, the current image A0 is acquired.

在步骤S1102中，对当前图像进行空域降噪，得到空域降噪图像A1。In step S1102, spatial noise reduction is performed on the current image to obtain a spatial noise reduction image A1.

在步骤S1103中，获取空域降噪图像A1。In step S1103, the spatial noise reduction image A1 is acquired.

在步骤S1104中，对空域降噪图像A1进行金字塔分解得到多层空域图像a0/a1/a2/a3。In step S1104, perform pyramid decomposition on the spatial domain noise reduction image A1 to obtain multi-layer spatial domain images a0/a1/a2/a3.

在步骤S1105中，获取时域降噪图像B0。In step S1105, a time-domain noise-reduced image B0 is acquired.

在步骤S1106中，对时域降噪图像B0进行金字塔分解得到多层时域图像b0/b1/b2/b3。In step S1106, perform pyramid decomposition on the time-domain noise-reduced image B0 to obtain multi-layer time-domain images b0/b1/b2/b3.

在步骤S1107中，对多层时域图像和多层空域图像进行分层对齐。In step S1107, hierarchical alignment is performed on the multi-layer time domain image and the multi-layer spatial domain image.

在步骤S1108中，获取对齐图像B1。In step S1108, the aligned image B1 is acquired.

在步骤S1109中，将对齐图像B1和空域降噪图像A1进行运动检测。In step S1109, motion detection is performed on the aligned image B1 and the spatial noise reduction image A1.

在步骤S1110中，获取运动检测对应的差异图。In step S1110, a difference map corresponding to motion detection is obtained.

在步骤S1111中，基于差异图确定融合权重图，并基于融合权重图将对齐图像和空域降噪图像进行双帧融合。In step S1111, the fusion weight map is determined based on the difference map, and the aligned image and the spatial noise reduction image are subjected to double-frame fusion based on the fusion weight map.

在步骤S1112中，获取融合结果表示的降噪图像C0。该降噪图像可以为当前图像A0对应的时域降噪后的图像。In step S1112, the noise-reduced image C0 represented by the fusion result is acquired. The noise-reduced image may be a time-domain noise-reduced image corresponding to the current image A0.

本公开实施例中的技术方案，先通过对图像进行精细对齐再进行时域降噪的方案，避免了不进行对齐而直接进行时域融合而容易产生的伪影问题，也避免了纹理复杂区域不进行对齐或对齐效果粗糙导致的纹理区域降噪效果不佳的问题，提高了在非运动区域和运动区域的降噪效果，也提高了降噪的质量和可靠性、全面性。使用金字塔分层且进行多次分尺度匹配的方式进行图像对齐匹配，由于每层都存在对应的匹配范围，因此能够扩大了匹配的运动范围且相对于相关技术中需要转换至时域而言减小了计算量。通过多次不同尺度的匹配，基于上次的非对齐区域进行精细的块匹配，使两幅图像对齐的更为准确，提升了匹配的准确性。在融合时非对齐区域的占比较少，提升了时域降噪的效果，进而提高降噪图像的图像质量。The technical solution in the embodiments of the present disclosure, through fine alignment of images first and then time-domain noise reduction, avoids the artifact problem that is easily generated by directly performing time-domain fusion without alignment, and also avoids areas with complex textures The problem of poor noise reduction in texture areas caused by non-alignment or rough alignment results has improved the noise reduction effect in non-moving and moving areas, and also improved the quality, reliability, and comprehensiveness of noise reduction. Image alignment matching is performed by using pyramid layering and multiple sub-scale matching. Since each layer has a corresponding matching range, the matching range of motion can be expanded and compared with the need to convert to the time domain in related technologies. The amount of calculation is reduced. Through multiple matchings of different scales, fine block matching is performed based on the last unaligned area, which makes the alignment of the two images more accurate and improves the matching accuracy. The proportion of non-aligned areas is small during fusion, which improves the effect of temporal noise reduction, thereby improving the image quality of the noise-reduced image.

本公开实施例中提供了一种图像降噪装置，参考图12中所示，该图像降噪装置1200可以包括：An embodiment of the present disclosure provides an image noise reduction device. Referring to FIG. 12 , the image noise reduction device 1200 may include:

第一图像分解模块1201，用于获取当前图像以及所述当前图像的参考帧图像的时域降噪图像，并对所述时域降噪图像进行分解得到多层时域图像；The firstimage decomposition module 1201 is configured to acquire temporal noise reduction images of the current image and reference frame images of the current image, and decompose the temporal noise reduction images to obtain multi-layer temporal images;

第二图像分解模块1202，用于对所述当前图像进行空域降噪得到所述当前图像对应的空域降噪图像，并对所述空域降噪图像进行分解得到多层空域图像；The secondimage decomposition module 1202 is configured to perform spatial noise reduction on the current image to obtain a spatial noise reduction image corresponding to the current image, and decompose the spatial noise reduction image to obtain a multi-layer spatial image;

图像匹配模块1203，用于将每层时域图像和每层空域图像进行匹配，确定所述时域降噪图像的对齐图像；Animage matching module 1203, configured to match each layer of time domain images with each layer of space domain images, and determine an aligned image of the time domain noise-reduced image;

图像融合模块1204，用于将所述空域降噪图像以及所述对齐图像进行融合，获取所述当前图像对应的降噪图像。Theimage fusion module 1204 is configured to fuse the spatial noise reduction image and the alignment image to obtain a noise reduction image corresponding to the current image.

在本公开的一种示例性实施例中，图像匹配模块包括：多尺度对齐模块，用于对每层时域图像和每层空域图像进行多次不同尺度的图像对齐，确定所述对齐图像。In an exemplary embodiment of the present disclosure, the image matching module includes: a multi-scale alignment module, configured to perform multiple image alignments of different scales on each layer of temporal images and each layer of spatial images, and determine the aligned images.

在本公开的一种示例性实施例中，第一图像分解模块包括：下采样模块，用于对所述时域降噪图像进行下采样得到多层时域图像，所述多层时域图像的分辨率不同。In an exemplary embodiment of the present disclosure, the first image decomposition module includes: a down-sampling module, configured to down-sample the temporal-domain noise-reduced image to obtain a multi-layer time-domain image, the multi-layer time-domain image resolutions are different.

在本公开的一种示例性实施例中，图像融合模块包括：差异图获取模块，用于将所述空域降噪图像以及所述对齐图像进行运动检测，确定差异图；权重融合模块，用于根据所述差异图确定融合权重图，并按照所述融合权重图对所述空域降噪图像以及所述对齐图像进行融合，获取所述降噪图像。In an exemplary embodiment of the present disclosure, the image fusion module includes: a difference map acquisition module, configured to perform motion detection on the spatial noise reduction image and the aligned image, and determine a difference map; a weight fusion module, used to A fusion weight map is determined according to the difference map, and the spatial domain noise reduction image and the aligned image are fused according to the fusion weight map to obtain the noise reduction image.

在本公开的一种示例性实施例中，多尺度对齐模块包括：对齐控制模块，用于按照分辨率的排列顺序，依次对每层空域图像在每层时域图像中进行运动矢量估计，并根据运动矢量进行运动补偿，直至进行多次图像对齐为止，以确定所述对齐图像。In an exemplary embodiment of the present disclosure, the multi-scale alignment module includes: an alignment control module, configured to sequentially perform motion vector estimation on each layer of spatial domain images in each layer of time domain images according to the order of resolution, and Motion compensation is performed according to the motion vector until image alignment is performed multiple times, so as to determine the aligned image.

在本公开的一种示例性实施例中，对齐控制模块包括：块匹配模块，用于采用块匹配方式依次对每层空域图像的匹配块在每层时域图像中进行运动矢量估计，并根据运动矢量进行运动补偿，以确定所述对齐图像。In an exemplary embodiment of the present disclosure, the alignment control module includes: a block matching module, configured to sequentially perform motion vector estimation on the matching blocks of each layer of spatial domain images in each layer of time domain images in a block matching manner, and according to Motion vectors are used for motion compensation to determine the aligned images.

在本公开的一种示例性实施例中，块匹配模块包括：第一次块匹配模块，用于依次对每层空域图像中的第一匹配块，在每层时域图像中通过第一次块匹配进行运动矢量估计，确定每层空域图像的运动矢量图，并根据所述运动矢量图进行运动补偿，以确定第一差分图像；第二次块匹配模块，用于以所述第一差分图像为基础，依次对每层空域图像的第二匹配块在每层时域图像中通过第二次块匹配进行运动矢量估计，确定每层空域图像的运动矢量图，并根据所述运动矢量图进行运动补偿，以确定第二差分图像；第三次块匹配模块，用于基于所述第二差分图像，依次对每层空域图像的第三匹配块在每层时域图像中通过第三次块匹配进行运动矢量估计，确定每层空域图像的运动矢量图，并根据所述运动矢量图进行运动补偿，以确定所述对齐图像。In an exemplary embodiment of the present disclosure, the block matching module includes: a block matching module for the first time, configured to sequentially match the first matching block in each layer of spatial domain image, and pass the first matching block in each layer of time domain image Block matching performs motion vector estimation, determines the motion vector diagram of each layer of spatial domain image, and performs motion compensation according to the motion vector diagram to determine the first difference image; the second block matching module is used to use the first difference Based on the image, the motion vector estimation is performed on the second matching block of each layer of air domain image in each layer of time domain image through the second block matching, and the motion vector diagram of each layer of air domain image is determined, and according to the motion vector diagram Perform motion compensation to determine the second difference image; the third block matching module is configured to sequentially perform the third matching block of each layer of spatial domain image in each layer of time domain image based on the second difference image The block matching performs motion vector estimation, determines the motion vector diagram of each layer of spatial domain image, and performs motion compensation according to the motion vector diagram to determine the alignment image.

在本公开的一种示例性实施例中，第一次块匹配模块包括：图像配置模块，用于将当前层空域图像作为当前层参考图像，将当前层时域图像作为当前层的待对齐图像；运动矢量估计模块，用于对当前层参考图像中的每个第一匹配块在所述待对齐图像中进行运动矢量估计，确定当前层参考图像的运动矢量图，以确定每层参考图像的运动矢量图；运动补偿模块，用于根据目标层参考图像的运动矢量图进行运动补偿，确定第一补偿图像；差分图像获取模块，用于基于所述第一补偿图像确定所述第一差分图像。In an exemplary embodiment of the present disclosure, the block matching module for the first time includes: an image configuration module, configured to use the current layer spatial domain image as the current layer reference image, and use the current layer time domain image as the current layer image to be aligned ; The motion vector estimation module is used to perform motion vector estimation on each first matching block in the current layer reference image in the image to be aligned, and determine the motion vector diagram of the current layer reference image to determine the motion vector of each layer reference image A motion vector diagram; a motion compensation module, configured to perform motion compensation according to a motion vector diagram of a target layer reference image, and determine a first compensated image; a differential image acquisition module, configured to determine the first differential image based on the first compensated image .

在本公开的一种示例性实施例中，运动矢量估计模块包括：运动矢量确定模块，用于根据当前层参考图像中每个第一匹配块在所述待对齐图像中的相似块，确定每个第一匹配块的运动矢量；运动矢量图确定模块，用于根据所述当前层参考图像中每个第一匹配块的运动矢量组成的图，确定所述运动矢量图。In an exemplary embodiment of the present disclosure, the motion vector estimation module includes: a motion vector determination module, configured to determine each A motion vector of a first matching block; a motion vector diagram determining module, configured to determine the motion vector diagram according to a graph composed of motion vectors of each first matching block in the current layer reference image.

在本公开的一种示例性实施例中，运动矢量确定模块包括：相似块搜索模块，用于基于当前层的搜索窗口，在所述待对齐图像中搜索每个第一匹配块的相似块；运动矢量计算模块，用于根据所述每个第一匹配块的坐标与所述相似块的坐标之间的差值，确定每个第一匹配块在所述待对齐图像中的运动矢量。In an exemplary embodiment of the present disclosure, the motion vector determination module includes: a similar block search module, configured to search for a similar block of each first matching block in the image to be aligned based on a search window of the current layer; A motion vector calculation module, configured to determine the motion vector of each first matching block in the image to be aligned according to the difference between the coordinates of each first matching block and the coordinates of the similar block.

在本公开的一种示例性实施例中，所述装置还包括：第一搜索窗口确定模块，用于若当前层参考图像为顶层参考图像，根据第一预设窗口确定当前层的搜索窗口；第二搜索窗口确定模块，用于若当前层参考图像不为顶层参考图像，结合所述当前层参考图像的上一层参考图像的运动矢量图、第一匹配块的坐标以及第二预设窗口确定所述当前层的搜索窗口。In an exemplary embodiment of the present disclosure, the device further includes: a first search window determining module, configured to determine a search window of the current layer according to a first preset window if the reference image of the current layer is a reference image of the top layer; The second search window determination module is used to combine the motion vector diagram of the previous layer reference image of the current layer reference image, the coordinates of the first matching block and the second preset window if the current layer reference image is not the top layer reference image A search window for the current layer is determined.

在本公开的一种示例性实施例中，第二搜索窗口确定模块包括：窗口控制模块，用于根据上一层参考图像中第一匹配块的运动矢量以及第一匹配块的坐标确定中心点，并基于所述中心点以及第二预设窗口，确定所述当前层的搜索窗口。In an exemplary embodiment of the present disclosure, the second search window determination module includes: a window control module, configured to determine the center point according to the motion vector of the first matching block in the upper layer reference image and the coordinates of the first matching block , and based on the center point and the second preset window, determine the search window of the current layer.

在本公开的一种示例性实施例中，运动补偿模块包括：运动矢量获取模块，用于基于所述目标层参考图像的运动矢量图，确定目标层参考图像中每个第一匹配块的运动矢量；补偿控制模块，用于根据每个第一匹配块的运动矢量以及每个第一匹配块的坐标确定补偿坐标，从目标层的待对齐图像中获取与所述补偿坐标对应的图像块进行运动补偿，得到第一补偿图像。In an exemplary embodiment of the present disclosure, the motion compensation module includes: a motion vector acquisition module, configured to determine the motion of each first matching block in the target layer reference image based on the motion vector diagram of the target layer reference image Vector; a compensation control module, configured to determine the compensation coordinates according to the motion vector of each first matching block and the coordinates of each first matching block, and obtain an image block corresponding to the compensation coordinates from the image to be aligned of the target layer to perform motion compensation to obtain a first compensated image.

在本公开的一种示例性实施例中，第一差分图像获取模块包括：差分处理模块，用于根据所述第一补偿图像和所述目标层参考图像进行差分处理，获取所述第一差分图像。In an exemplary embodiment of the present disclosure, the first difference image acquisition module includes: a difference processing module, configured to perform difference processing according to the first compensation image and the target layer reference image, and obtain the first difference image.

在本公开的一种示例性实施例中，第二次块匹配模块包括：第一非对齐区域确定模块，用于根据所述第一差分图像确定所述每层时域图像的第一非对齐区域；块匹配模块，用于依次对每层空域图像中的第二匹配块在所述第一非对齐区域进行第二次块匹配，确定每层空域图像在每层时域图像中的运动矢量图；运动补偿模块，用于根据目标层空域图像的运动矢量图进行运动补偿，确定第二补偿图像；第二差分图像获取模块，用于根据所述第二补偿图像以及所述目标层空域图像进行差分处理，获取所述第二差分图像。In an exemplary embodiment of the present disclosure, the second block matching module includes: a first non-alignment region determination module, configured to determine the first non-alignment of the time-domain image of each layer according to the first difference image Area; block matching module, used to sequentially perform second block matching on the second matching block in the first non-alignment area in each layer of air domain image, and determine the motion vector of each layer of air domain image in each layer of time domain image Figure; motion compensation module, used to perform motion compensation according to the motion vector diagram of the target layer spatial domain image, and determine a second compensation image; a second differential image acquisition module, used to base the second compensation image and the target layer spatial domain image Perform difference processing to acquire the second difference image.

在本公开的一种示例性实施例中，第三次块匹配模块包括：第二非对齐区域确定模块，用于根据所述第二差分图像确定所述每层时域图像的第二非对齐区域；块匹配模块，用于依次对每层空域图像中的第三匹配块，在所述第二非对齐区域进行第三次块匹配，确定每层空域图像的运动矢量图；运动补偿模块，用于根据目标层空域图像的运动矢量图进行运动补偿，确定第三补偿图像；对齐图像确定模块，用于将所述第三补偿图像确定为所述对齐图像。In an exemplary embodiment of the present disclosure, the third block matching module includes: a second non-alignment region determination module, configured to determine the second non-alignment of the time-domain image of each layer according to the second difference image Region; block matching module, used to perform block matching for the third time in the second non-aligned region on the third matching block in each layer of airspace image sequentially, and determine the motion vector diagram of each layer of airspace image; motion compensation module, The method is configured to perform motion compensation according to the motion vector diagram of the spatial image of the target layer, and determine a third compensated image; an aligned image determining module is configured to determine the third compensated image as the aligned image.

在本公开的一种示例性实施例中，差异图获取模块包括：平滑处理模块，用于对所述空域降噪图像和所述对齐图像分别进行平滑处理；形态学处理模块，用于将平滑后的空域降噪图像以及平滑后的对齐图像相减得到差分图，并对所述差分图进行形态学处理，获取所述差异图。In an exemplary embodiment of the present disclosure, the difference map acquisition module includes: a smoothing processing module for smoothing the spatial noise reduction image and the alignment image respectively; a morphological processing module for smoothing Subtracting the final spatial noise reduction image and the smoothed aligned image to obtain a difference map, and performing morphological processing on the difference map to obtain the difference map.

在本公开的一种示例性实施例中，运动补偿模块包括：补偿块确定模块，用于根据每层空域图像中的匹配块确定用于运动补偿的补偿块；补偿控制模块，用于根据目标层空域图像中每个匹配块的运动矢量确定补偿坐标，并根据补偿坐标对应的图像块对每层空域图像的所述补偿块进行运动补偿。In an exemplary embodiment of the present disclosure, the motion compensation module includes: a compensation block determination module, configured to determine a compensation block for motion compensation according to a matching block in each layer of spatial domain image; a compensation control module, configured to determine a compensation block according to a target The motion vector of each matching block in the spatial domain image of the layer determines the compensation coordinates, and motion compensation is performed on the compensation block of the spatial domain image of each layer according to the image block corresponding to the compensation coordinates.

在本公开的一种示例性实施例中，补偿块确定模块包括：第一补偿块确定模块，用于在第一次块匹配以及第二次块匹配时，将所述匹配块的全部区域作为补偿块；第二补偿块确定模块，用于在第三次块匹配时，将所述匹配块的部分区域作为补偿块，并按照偏移量更新所述补偿块。In an exemplary embodiment of the present disclosure, the compensation block determination module includes: a first compensation block determination module, configured to use the entire area of the matching block as the first block matching and the second block matching Compensation block; a second compensation block determining module, configured to use a part of the matching block as a compensation block during the third block matching, and update the compensation block according to the offset.

需要说明的是，上述图像降噪装置中各部分的具体细节在图像降噪方法部分实施方式中已经详细说明，未披露的细节内容可以参见方法部分的实施方式内容，因而不再赘述。It should be noted that the specific details of each part of the above-mentioned image noise reduction device have been described in detail in some implementations of the image noise reduction method. For undisclosed details, please refer to the implementation content of the method part, so they will not be repeated here.

本公开的示例性实施方式还提供一种电子设备。该电子设备可以是上述终端101。一般的，该电子设备可以包括处理器与存储器，存储器用于存储处理器的可执行指令，处理器配置为经由执行可执行指令来执行上述图像降噪方法。Exemplary embodiments of the present disclosure also provide an electronic device. The electronic device may be the above-mentionedterminal 101 . Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above image noise reduction method by executing the executable instructions.

下面以图13中的移动终端1300为例，对该电子设备的构造进行示例性说明。本领域技术人员应当理解，除了特别用于移动目的的部件之外，图13中的构造也能够应用于固定类型的设备。The following takes the mobile terminal 1300 in FIG. 13 as an example to illustrate the structure of the electronic device. Those skilled in the art will appreciate that, in addition to components specifically intended for mobile purposes, the configuration in Fig. 13 can also be applied to equipment of a stationary type.

如图13所示，移动终端1300具体可以包括：处理器1301、存储器1302、总线1303、移动通信模块1304、天线1、无线通信模块1305、天线2、显示屏1306、摄像模块1307、音频模块1308、电源模块1309与传感器模块1310。As shown in Figure 13, the mobile terminal 1300 may specifically include: aprocessor 1301, amemory 1302, abus 1303, amobile communication module 1304, anantenna 1, awireless communication module 1305, anantenna 2, adisplay screen 1306, acamera module 1307, and anaudio module 1308 , apower module 1309 and asensor module 1310 .

处理器1301可以包括一个或多个处理单元，例如：处理器1301可以包括AP(Application Processor，应用处理器)、调制解调处理器、GPU(Graphics ProcessingUnit，图形处理器)、ISP(Image Signal Processor，图像信号处理器)、控制器、编码器、解码器、DSP(Digital Signal Processor，数字信号处理器)、基带处理器和/或NPU(Neural-Network Processing Unit，神经网络处理器)等。本示例性实施方式中的图像降噪方法可以由AP、GPU或DSP来执行，当方法涉及到神经网络相关的处理时，可以由NPU来执行，例如NPU可以加载神经网络参数并执行神经网络相关的算法指令。Theprocessor 1301 may include one or more processing units, for example: theprocessor 1301 may include an AP (Application Processor, an application processor), a modem processor, a GPU (Graphics Processing Unit, a graphics processor), an ISP (Image Signal Processor , image signal processor), controller, encoder, decoder, DSP (Digital Signal Processor, digital signal processor), baseband processor and/or NPU (Neural-Network Processing Unit, neural network processor), etc. The image noise reduction method in this exemplary embodiment can be performed by AP, GPU or DSP. When the method involves neural network related processing, it can be performed by NPU. For example, NPU can load neural network parameters and perform neural network related processing. algorithmic instructions.

编码器可以对图像或视频进行编码(即压缩)，以减小数据大小，便于存储或发送。解码器可以对图像或视频的编码数据进行解码(即解压缩)，以还原出图像或视频数据。移动终端1300可以支持一种或多种编码器和解码器，例如：JPEG(Joint PhotographicExperts Group，联合图像专家组)、PNG(Portable Network Graphics，便携式网络图形)、BMP(Bitmap，位图)等图像格式，MPEG(Moving Picture Experts Group，动态图像专家组)1、MPEG10、H.1063、H.1064、HEVC(High Efficiency Video Coding，高效率视频编码)等视频格式。An encoder encodes (i.e. compresses) an image or video to reduce the data size for storage or transmission. The decoder can decode (that is, decompress) the coded data of the image or video to restore the image or video data. Mobile terminal 1300 can support one or more encoders and decoders, for example: images such as JPEG (Joint Photographic Experts Group, Joint Photographic Experts Group), PNG (Portable Network Graphics, portable network graphics), BMP (Bitmap, bitmap) Format, video formats such as MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, HEVC (High Efficiency Video Coding, high-efficiency video coding).

处理器1301可以通过总线1303与存储器1302或其他部件形成连接。Theprocessor 1301 can form a connection with thememory 1302 or other components through thebus 1303 .

存储器1302可以用于存储计算机可执行程序代码，可执行程序代码包括指令。处理器1301通过运行存储在存储器1302的指令，执行移动终端1300的各种功能应用以及数据处理。存储器1302还可以存储应用数据，例如存储图像，视频等文件。Thememory 1302 may be used to store computer-executable program code, which includes instructions. Theprocessor 1301 executes various functional applications and data processing of the mobile terminal 1300 by executing instructions stored in thememory 1302 . Thememory 1302 can also store application data, such as storing images, videos and other files.

移动终端1300的通信功能可以通过移动通信模块1304、天线1、无线通信模块1305、天线2、调制解调处理器以及基带处理器等实现。天线1和天线2用于发射和接收电磁波信号。移动通信模块1304可以提供应用在移动终端1300上3G、4G、5G等移动通信解决方案。无线通信模块1305可以提供应用在移动终端1300上的无线局域网、蓝牙、近场通信等无线通信解决方案。The communication function of the mobile terminal 1300 can be realized by themobile communication module 1304, theantenna 1, thewireless communication module 1305, theantenna 2, the modem processor and the baseband processor, etc.Antenna 1 andAntenna 2 are used to transmit and receive electromagnetic wave signals. Themobile communication module 1304 can provide 3G, 4G, 5G and other mobile communication solutions applied on themobile terminal 1300 . Thewireless communication module 1305 can provide wireless communication solutions such as wireless local area network, bluetooth, and near field communication applied on themobile terminal 1300 .

显示屏1306用于实现显示功能，如显示用户界面、图像、视频等。摄像模块1307用于实现拍摄功能，如拍摄图像、视频等，且摄像模块中可以包含色温传感器阵列。音频模块1308用于实现音频功能，如播放音频，采集语音等。电源模块1309用于实现电源管理功能，如为电池充电、为设备供电、监测电池状态等。传感器模块1310可以包括一种或多种传感器，用于实现相应的感应检测功能。例如，传感器模块1310可以包括惯性传感器，其用于检测移动终端1300的运动位姿，输出惯性传感数据。Thedisplay screen 1306 is used to implement a display function, such as displaying a user interface, images, videos, and the like. Thecamera module 1307 is used to realize the shooting function, such as taking images, videos, etc., and the camera module may include a color temperature sensor array. Theaudio module 1308 is used to implement audio functions, such as playing audio, collecting voice, and so on. Thepower module 1309 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring the state of the battery. Thesensor module 1310 may include one or more sensors for realizing corresponding sensing and detection functions. For example, thesensor module 1310 may include an inertial sensor, which is used to detect the movement pose of themobile terminal 1300 and output inertial sensor data.

需要说明的是，本公开实施例中还提供了一种计算机可读存储介质，该计算机可读存储介质可以是上述实施例中描述的电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。It should be noted that the embodiments of the present disclosure also provide a computer-readable storage medium, which may be contained in the electronic device described in the above-mentioned embodiments, or may exist independently without assembled into the electronic device.

计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

计算机可读存储介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读存储介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。The computer-readable storage medium may send, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wires, optical cables, RF, etc., or any suitable combination of the foregoing.

计算机可读存储介质承载有一个或者多个程序，当上述一个或者多个程序被一个该电子设备执行时，使得该电子设备实现如下述实施例中所述的方法。The computer-readable storage medium bears one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device is made to implement the methods described in the following embodiments.

通过以上的实施方式的描述，本领域的技术人员易于理解，这里描述的示例实施方式可以通过软件实现，也可以通过软件结合必要的硬件的方式来实现。因此，根据本公开实施方式的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中或网络上，包括若干指令以使得一台计算设备(可以是个人计算机、服务器、终端装置、或者网络设备等)执行根据本公开实施方式的方法。Through the description of the above implementations, those skilled in the art can easily understand that the example implementations described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of software products, and the software products can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.) or on the network , including several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

此外，上述附图仅是根据本公开示例性实施例的方法所包括的处理的示意性说明，而不是限制目的。易于理解，上述附图所示的处理并不表明或限制这些处理的时间顺序。另外，也易于理解，这些处理可以是例如在多个模块中同步或异步执行的。In addition, the above-mentioned drawings are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not imply or limit the chronological order of these processes. In addition, it is also easy to understand that these processes may be executed synchronously or asynchronously in multiple modules, for example.

应当注意，尽管在上文详细描述中提及了用于动作执行的设备的若干模块或者单元，但是这种划分并非强制性的。实际上，根据本公开的实施方式，上文描述的两个或更多模块或者单元的特征和功能可以在一个模块或者单元中具体化。反之，上文描述的一个模块或者单元的特征和功能可以进一步划分为由多个模块或者单元来具体化。It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. Actually, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided to be embodied by a plurality of modules or units.

本领域技术人员在考虑说明书及实践这里公开的内容后，将容易想到本公开的其他实施例。本申请旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由权利要求指出。应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围仅由所附的权利要求来限。Other embodiments of the disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the disclosure herein. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered exemplary only, with the true scope and spirit of the disclosure indicated by the appended claims. It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (22)

1. An image noise reduction method, comprising:

acquiring a current image and a time domain noise reduction image of a reference frame image of the current image, and decomposing the time domain noise reduction image to obtain a multilayer time domain image;

performing spatial domain noise reduction on the current image to obtain a spatial domain noise reduction image corresponding to the current image, and decomposing the spatial domain noise reduction image to obtain a multilayer spatial domain image;

matching each layer of time domain image with each layer of space domain image to determine an alignment image of the time domain noise reduction image;

and fusing the airspace noise reduction image and the alignment image to obtain a noise reduction image corresponding to the current image.

2. The image denoising method according to claim 1, wherein the matching each layer of temporal image and each layer of spatial image to determine an aligned image of the temporally denoised image comprises:

and carrying out image alignment of different scales for multiple times on each layer of time domain image and each layer of space domain image, and determining the aligned images.

3. The method according to claim 2, wherein decomposing the temporally noise-reduced image to obtain a multi-layered temporally image comprises:

and downsampling the time domain noise reduction image to obtain a multilayer time domain image, wherein the multilayer time domain image has different resolutions.

4. The image denoising method according to claim 1, wherein the fusing the spatial domain denoising image and the alignment image to obtain the denoising image corresponding to the current image comprises:

performing motion detection on the spatial domain noise reduction image and the alignment image to determine a difference map;

and determining a fusion weight map according to the difference map, and fusing the spatial domain noise reduction image and the alignment image according to the fusion weight map to obtain the noise reduction image.

5. The method for reducing image noise according to claim 2, wherein the aligning the images of different scales for a plurality of times in each layer of the temporal image and each layer of the spatial image, and determining the aligned images comprises:

and according to the arrangement sequence of the resolution, sequentially carrying out motion vector estimation on each layer of spatial domain image in each layer of time domain image, and carrying out motion compensation according to the motion vector until image alignment is carried out for multiple times so as to determine the aligned image.

6. The image noise reduction method according to claim 5, wherein the sequentially performing motion vector estimation on each spatial domain image in each temporal domain image and performing motion compensation according to the motion vector comprises:

and sequentially carrying out motion vector estimation on the matching blocks of each layer of spatial domain image in each layer of time domain image by adopting a block matching mode, and carrying out motion compensation according to the motion vectors so as to determine the aligned image.

7. The method of claim 6, wherein the performing motion vector estimation on the matching blocks of each spatial domain image layer in each temporal domain image layer in sequence by using block matching and performing motion compensation according to the motion vector to determine the aligned image comprises:

sequentially carrying out motion vector estimation on first matching blocks in each layer of spatial domain image through first block matching in each layer of temporal domain image, determining a motion vector diagram of each layer of spatial domain image, and carrying out motion compensation according to the motion vector diagram to determine a first differential image;

on the basis of the first differential image, sequentially carrying out motion vector estimation on a second matching block of each layer of spatial domain image in each layer of temporal domain image through second block matching, determining a motion vector diagram of each layer of spatial domain image, and carrying out motion compensation according to the motion vector diagram to determine a second differential image;

and based on the second differential image, sequentially carrying out motion vector estimation on a third matching block of each layer of spatial domain image in each layer of temporal domain image through third block matching, determining a motion vector diagram of each layer of spatial domain image, and carrying out motion compensation according to the motion vector diagram so as to determine the aligned image.

8. The image denoising method according to claim 7, wherein the sequentially performing motion vector estimation on the first matching block in each spatial domain image layer by the first block matching in each spatial domain image layer, determining a motion vector map of each spatial domain image layer, and performing motion compensation according to the motion vector map to determine a first difference image comprises:

taking a current layer space domain image as a current layer reference image, and taking a current layer time domain image as an image to be aligned of a current layer;

performing motion vector estimation on each first matching block in the current layer reference image in the image to be aligned, and determining a motion vector diagram of the current layer reference image so as to determine the motion vector diagram of each layer of reference image;

performing motion compensation according to the motion vector diagram of the target layer reference image to determine a first compensation image;

determining the first differential image based on the first compensated image.

9. The image noise reduction method according to claim 8, wherein the performing motion vector estimation on each first matching block in the current layer reference image in the image to be aligned to determine the motion vector map of the current layer reference image comprises:

determining a motion vector of each first matching block in a current layer reference image according to a similar block of each first matching block in the image to be aligned;

and determining the motion vector diagram according to the diagram formed by the motion vectors of each first matching block in the reference image of the current layer.

10. The method of claim 9, wherein the determining the motion vector of each first matching block according to the similar block of each first matching block in the current layer reference image in the image to be aligned comprises:

searching similar blocks of each first matching block in the image to be aligned based on a search window of a current layer;

and determining a motion vector of each first matching block in the image to be aligned according to the difference between the coordinate of each first matching block and the coordinate of the similar block.

11. The method of image noise reduction according to claim 10, further comprising:

if the current layer reference image is the top layer reference image, determining a search window of the current layer according to a first preset window;

and if the current layer reference image is not the top layer reference image, determining a search window of the current layer by combining a motion vector diagram of a previous layer reference image of the current layer reference image, the coordinates of the first matching block and a second preset window.

12. The image denoising method of claim 11, wherein the determining a search window of the current layer reference image by combining a motion vector map of a previous layer reference image of the current layer reference image, coordinates of a first matching block, and a second preset window comprises:

and determining a central point according to the motion vector of the first matching block in the reference image of the previous layer and the coordinate of the first matching block, and determining a search window of the current layer based on the central point and a second preset window.

13. The image denoising method according to claim 8, wherein the determining the first compensation image by performing motion compensation according to a motion vector map of a target layer reference image comprises:

determining a motion vector of each first matching block in the target layer reference image based on the motion vector diagram of the target layer reference image;

and determining compensation coordinates according to the motion vector of each first matching block and the coordinates of each first matching block, and acquiring an image block corresponding to the compensation coordinates from the image to be aligned of the target layer for motion compensation to obtain a first compensation image.

14. The method of image noise reduction according to claim 8, wherein the determining the first difference image based on the first compensated image comprises:

and carrying out difference processing according to the first compensation image and the target layer reference image to obtain the first difference image.

15. The image denoising method according to claim 7, wherein the sequentially performing motion vector estimation on the second matching block of each spatial domain image in each temporal domain image by the second block matching based on the first difference image to determine a motion vector map of each spatial domain image, and performing motion compensation according to the motion vector map to determine a second difference image comprises:

determining a first non-aligned region of each layer of time domain image according to the first differential image;

sequentially carrying out second block matching on second matching blocks in each layer of spatial domain image in the first non-aligned region, and determining a motion vector diagram of each layer of spatial domain image in each layer of temporal domain image;

performing motion compensation according to the motion vector diagram of the spatial domain image of the target layer, and determining a second compensation image;

and carrying out difference processing according to the second compensation image and the target layer airspace image to obtain a second difference image.

16. The image denoising method according to claim 7, wherein the sequentially performing motion vector estimation on a third matching block of each spatial domain image in each temporal domain image through third block matching based on the second difference image, determining a motion vector map of each spatial domain image, and performing motion compensation according to the motion vector map to determine the aligned image comprises:

determining a second non-aligned region of each layer of time domain image according to the second differential image;

sequentially carrying out third block matching on third matching blocks in each layer of airspace image in the second non-aligned region to determine a motion vector diagram of each layer of airspace image;

performing motion compensation according to the motion vector diagram of the spatial domain image of the target layer, and determining a third compensation image;

determining the third compensated image as the aligned image.

17. The method according to claim 4, wherein the detecting the motion of the spatially denoised image and the aligned image to determine a disparity map comprises:

respectively smoothing the spatial domain noise reduction image and the alignment image;

and subtracting the smoothed airspace noise reduction image and the smoothed alignment image to obtain a difference image, and performing morphological processing on the difference image to obtain the difference image.

18. The method for reducing image noise according to claim 5, wherein the performing motion compensation according to the motion vector comprises:

determining a compensation block for motion compensation according to the matching block in each layer of spatial domain image;

and determining compensation coordinates according to the motion vector of each matching block in the target layer airspace image, and performing motion compensation on the compensation block of each layer airspace image according to the image block corresponding to the compensation coordinates.

19. The method for image noise reduction according to claim 18, wherein the determining a compensation block for motion compensation according to the matching block in each spatial domain image comprises:

when the first time block matching and the second time block matching are carried out, all areas of the matching blocks are used as compensation blocks;

and when the third block is matched, taking a partial area of the matched block as a compensation block, and updating the compensation block according to the offset.

20. An image noise reduction apparatus, comprising:

the first image decomposition module is used for acquiring a current image and a time domain noise reduction image of a reference frame image of the current image, and decomposing the time domain noise reduction image to obtain a multilayer time domain image;

the second image decomposition module is used for carrying out spatial domain noise reduction on the current image to obtain a spatial domain noise reduction image corresponding to the current image, and decomposing the spatial domain noise reduction image to obtain a multilayer spatial domain image;

the image matching module is used for matching each layer of time domain image with each layer of space domain image and determining an alignment image of the time domain noise reduction image;

and the image fusion module is used for fusing the airspace noise reduction image and the alignment image to obtain a noise reduction image corresponding to the current image.

21. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the image denoising method of any one of claims 1-19 via execution of the executable instructions.

22. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the image denoising method according to any one of claims 1 through 19.

Images