CN118587103A - Image processing method, device, electronic device and computer readable storage medium - Google Patents

Abstract

The invention discloses an image processing method, an image processing device, electronic equipment and a computer readable storage medium, wherein the method comprises the following steps: acquiring a plurality of frames of different exposure images; calibrating the exposure image to obtain a calibration image; calculating a first weight corresponding to the calibration image; determining a motion region of the calibration image; calculating a second weight of the motion region of the calibration image; and performing image synthesis on the multi-frame calibration image according to the first weight and the second weight to obtain a target synthesized image. According to the invention, the scene positions of the exposure images with different exposure time are matched through image calibration, so that the difference generated by the movement of the image sensor during shooting is eliminated, the blurring phenomenon generated during pixel level fusion is avoided, the image quality is improved, the calibration image is synthesized by recalculating the weights corresponding to the calibration image and the motion area of the calibration image, the generation of synthetic image ghosts is effectively eliminated, and the image quality is further improved.

Description

Translated fromChinese

图像处理方法、装置、电子设备及计算机可读存储介质Image processing method, device, electronic device and computer readable storage medium

技术领域Technical Field

本发明涉及图像处理技术领域，尤其是涉及一种图像处理方法、装置、电子设备及计算机可读存储介质。The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium.

背景技术Background Art

相关技术中，当场景为变化的动态场景时，则会因为不同帧曝光图像之间存在曝光时间间隔，导致不同帧曝光图像之间因为运动物体而产生位移偏差，从而出现模糊的鬼影现象，进而严重影响图像质量。In the related art, when the scene is a changing dynamic scene, there will be an exposure time interval between different frames of exposure images, which will cause displacement deviations between different frames of exposure images due to moving objects, resulting in blurred ghosting, which will seriously affect the image quality.

发明内容Summary of the invention

本发明旨在至少解决现有技术中存在的技术问题之一。The present invention aims to solve at least one of the technical problems existing in the prior art.

为此，本发明的一个目的在于提出一种图像处理方法，该方法通过图像校准，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量，通过重新计算校准图像和校准图像的运动区域对应的权重对校准图像进行图像合成，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。To this end, an object of the present invention is to propose an image processing method, which, through image calibration, matches the scene positions of exposure images with different exposure times, eliminates the differences caused by the movement of the image sensor during shooting, avoids the blurring phenomenon caused by pixel-level fusion, and thus is beneficial to improving image quality. The calibration images are synthesized by recalculating the weights corresponding to the calibration images and the moving areas of the calibration images, thereby effectively eliminating the generation of ghost images in the synthesized images, thereby further improving the image quality.

为此，本发明的第二个目的在于提出一种图像处理装置。Therefore, a second object of the present invention is to provide an image processing device.

为此，本发明的第三个目的在于提出一种电子设备。To this end, a third objective of the present invention is to provide an electronic device.

为此，本发明的第四个目的在于提出一种计算机可读存储介质。To this end, a fourth object of the present invention is to provide a computer-readable storage medium.

为了实现上述目的，本发明第一方面实施例提出了一种图像处理方法，包括以下步骤：获取多帧不同的曝光图像；对所述曝光图像进行校准，得到校准图像；计算所述校准图像对应的第一权重；确定所述校准图像的运动区域；计算所述校准图像的运动区域第二权重；根据所述第一权重和第二权重对多帧所述校准图像进行图像合成，得到目标合成图像。In order to achieve the above-mentioned objectives, an embodiment of the first aspect of the present invention proposes an image processing method, comprising the following steps: acquiring multiple frames of images with different exposures; calibrating the exposure images to obtain a calibration image; calculating a first weight corresponding to the calibration image; determining a motion area of the calibration image; calculating a second weight of the motion area of the calibration image; synthesizing multiple frames of the calibration images according to the first weight and the second weight to obtain a target synthesized image.

根据本发明实施例的图像处理方法，通过对曝光图像校准，得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和校准图像的运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像校准，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过重新计算校准图像和校准图像的运动区域对应的权重对校准图像，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。According to the image processing method of the embodiment of the present invention, the exposure image is calibrated to obtain a calibration image, the motion area of the calibration image is determined, the first weight corresponding to the calibration image and the second weight corresponding to the motion area of the calibration image are calculated, and then multiple frames of calibration images are synthesized according to the first weight and the second weight to obtain a target synthetic image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image calibration, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by recalculating the calibration image and the weight corresponding to the motion area of the calibration image, the generation of synthetic image ghosts is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

另外，本发明上述实施例的图像处理方法，还可以具有如下附加的技术特征：In addition, the image processing method of the above embodiment of the present invention may also have the following additional technical features:

在本发明的一个实施例中，对所述曝光图像进行校准，包括：对所述曝光图像进行图像配准，得到配准图像；对所述配准图像进行图像校正，得到所述校准图像。In one embodiment of the present invention, calibrating the exposure image includes: performing image registration on the exposure image to obtain a registered image; and performing image correction on the registered image to obtain the calibrated image.

在本发明的一个实施例中，对所述曝光图像进行图像配准，包括：按照预设的曝光曲线将所述曝光图像的曝光量置于预设曝光水平，得到调整后的曝光图像，所述调整后的曝光图像包括参考图像和预配准图像；获取所述预配准图像和参考图像之间的特征点对，以得到所述配准图像。In one embodiment of the present invention, image registration is performed on the exposure image, including: placing the exposure amount of the exposure image at a preset exposure level according to a preset exposure curve to obtain an adjusted exposure image, wherein the adjusted exposure image includes a reference image and a pre-registered image; and acquiring feature point pairs between the pre-registered image and the reference image to obtain the registered image.

在本发明的一个实施例中，获取所述预配准图像和参考图像之间的特征点对，包括：对所述预配准图像和所述参考图像进行特征点检测，对应得到所述预配准图像和所述参考图像的特征点信息；所述参考图像包括预设原点，从预设原点开始，以预设匹配范围对所述预配准图像和所述参考图像进行特征点匹配，得到所述特征点对。In one embodiment of the present invention, obtaining the feature point pair between the pre-registered image and the reference image includes: performing feature point detection on the pre-registered image and the reference image to obtain feature point information of the pre-registered image and the reference image accordingly; the reference image includes a preset origin, and starting from the preset origin, feature point matching is performed on the pre-registered image and the reference image within a preset matching range to obtain the feature point pair.

在本发明的一个实施例中，对所述配准图像进行校正，包括：计算所述特征点对的距离；统计计算得到的特征点对的距离，将满足预设条件的特征点对的距离作为图像偏移量；计算所述配准图像所需的偏移方向；根据所述图像偏移量和所述偏移方向对所述配准图像进行偏移操作，以对所述配准图像进行校正，得到所述校准图像。In one embodiment of the present invention, the registration image is corrected, including: calculating the distance of the feature point pairs; statistically calculating the distance of the feature point pairs, and using the distance of the feature point pairs that meets preset conditions as the image offset; calculating the offset direction required for the registration image; and performing an offset operation on the registration image according to the image offset and the offset direction to correct the registration image and obtain the calibrated image.

在本发明的一个实施例中，所述方法还包括：若在所述预设匹配范围进行内未匹配到特征点对，则按照预设步长扩大匹配范围，以重新进行特征点匹配，直至得到所述特征点对。In one embodiment of the present invention, the method further includes: if no feature point pair is matched within the preset matching range, expanding the matching range according to a preset step length to re-match the feature points until the feature point pair is obtained.

在本发明的一个实施例中，所述计算所述特征点对的距离，包括：以所述参考图像为基准，计算出所述参考图像与所述配准图像的相匹配的特征点之间的欧氏距离，将计算的所述欧氏距离作为所述特征点对的距离。In one embodiment of the present invention, the calculating the distance of the feature point pair includes: taking the reference image as a reference, calculating the Euclidean distance between the matching feature points of the reference image and the registration image, and using the calculated Euclidean distance as the distance of the feature point pair.

在本发明的一个实施例中，所述满足预设条件的特征点对的距离包括：出现次数最多的特征点对的距离。In one embodiment of the present invention, the distance between the feature point pairs that meet the preset conditions includes: the distance between the feature point pairs that appear the most times.

在本发明的一个实施例中，所述计算所述配准图像所需的偏移方向，包括：以所述参考图像的像素点坐标为原点计算所述配准图像中对应像素点坐标所在的象限位置；根据所述象限位置确定所述配准图像所需的所述偏移方向。In one embodiment of the present invention, the calculation of the offset direction required for the registered image includes: calculating the quadrant position of the corresponding pixel point coordinates in the registered image with the pixel point coordinates of the reference image as the origin; and determining the offset direction required for the registered image according to the quadrant position.

在本发明的一个实施例中，在根据所述图像偏移量和所述偏移方向对所述配准图像进行偏移操作之后，还包括：对偏移后的配准图像的边界进行补零处理，以使其与对应的曝光图像的尺寸相同，以得到所述校准图像。In one embodiment of the present invention, after performing an offset operation on the registered image according to the image offset amount and the offset direction, it also includes: performing zero padding processing on the boundary of the offset registered image to make it the same size as the corresponding exposure image to obtain the calibration image.

在本发明的一个实施例中，确定所述校准图像的运动区域，包括：获取所述校准图像和所述参考图像的二值图像；将所述校准图像和参考图像的二值图像相减求绝对值，得到所述校准图像的运动区域。In one embodiment of the present invention, determining the motion region of the calibration image includes: acquiring binary images of the calibration image and the reference image; and subtracting the binary images of the calibration image and the reference image to obtain an absolute value to obtain the motion region of the calibration image.

在本发明的一个实施例中，计算所述校准图像对应的第一权重之前，包括：确定所述校准图像的非运动区域。In one embodiment of the present invention, before calculating the first weight corresponding to the calibration image, the method includes: determining a non-moving area of the calibration image.

在本发明的一个实施例中，确定所述校准图像的非运动区域，包括：将所述校准图像的运动区域取反，得到所述非运动区域。In one embodiment of the present invention, determining the non-motion region of the calibration image includes: inverting the motion region of the calibration image to obtain the non-motion region.

在本发明的一个实施例中，所述非运动区域的图像包括长曝光图像和短曝光图像，所述确定所述校准图像的非运动区域之后，还包括：在RGB颜色空间下，将预设的长曝光图像的信噪比映射到所述非运动区域的短曝光图像上，以提高所述短曝光图像的信噪比。In one embodiment of the present invention, the image of the non-moving area includes a long-exposure image and a short-exposure image. After determining the non-moving area of the calibration image, the method further includes: mapping a preset signal-to-noise ratio of the long-exposure image to the short-exposure image of the non-moving area in an RGB color space to improve the signal-to-noise ratio of the short-exposure image.

在本发明的一个实施例中，计算所述校准图像对应的第一权重，包括：计算所述校准图像的对比度、曝光度和图像熵；通过所述对比度、曝光度、图像熵计算所述校准图像对应的权重，以得到所述第一权重。In one embodiment of the present invention, calculating the first weight corresponding to the calibration image includes: calculating the contrast, exposure and image entropy of the calibration image; and calculating the weight corresponding to the calibration image through the contrast, exposure and image entropy to obtain the first weight.

在本发明的一个实施例中，所述计算所述校准图像的对比度，包括：对所述校准图像对应的曝光图像进行拉普拉斯滤波后，对得到的滤波结果求取绝对值，以得到所述校准图像的对比度。In one embodiment of the present invention, the calculating the contrast of the calibration image includes: performing Laplace filtering on an exposure image corresponding to the calibration image, and then obtaining an absolute value of the filtering result to obtain the contrast of the calibration image.

在本发明的一个实施例中，所述计算所述校准图像的曝光度包括：采用三角权重函数计算所述校准图像的曝光度。In one embodiment of the present invention, the calculating the exposure of the calibration image includes: calculating the exposure of the calibration image using a triangular weight function.

在本发明的一个实施例中，所述计算所述校准图像的图像熵，包括：用预设矩阵作为熵值计算的范围，通过计算所述预设矩阵内所有像素点出现的概率来计算中间点的熵值，将该熵值作为所述校准图像的图像熵。In one embodiment of the present invention, calculating the image entropy of the calibration image includes: using a preset matrix as the range of entropy value calculation, calculating the entropy value of the middle point by calculating the probability of occurrence of all pixel points in the preset matrix, and using the entropy value as the image entropy of the calibration image.

在本发明的一个实施例中，通过所述对比度、曝光度、图像熵计算所述校准图像对应的权重，包括：将所述对比度、曝光度、图像熵相乘并进行归一化，得到所述校准图像的权重。In one embodiment of the present invention, the weight corresponding to the calibration image is calculated by using the contrast, exposure, and image entropy, including: multiplying the contrast, exposure, and image entropy and normalizing them to obtain the weight of the calibration image.

在本发明的一个实施例中，计算所述校准图像的运动区域对应的第二权重，包括：W′_x,＝(W₁*(I_1ex+I_2ex)+W₂)/I_1ex，其中，W′_x,为第二权重，I_1ex为曝光图像I₁对应的曝光量，I_2ex为曝光图像I₂对应的曝光量，W₁为曝光图像I₁的对应的第一权重，W₂为曝光图像I₂对应的第一权重，其中，I₁和I₂为相邻的两帧曝光图像，(x,y)表示参考图像的坐标点。In one embodiment of the present invention, the second weight corresponding to the motion area of the calibration image is calculated, including: W′_x, =(W₁ *(I_1ex +I_2ex )+W₂ )/I_1ex , wherein W′_x, is the second weight, I_1ex is the exposure amount corresponding to the exposure image I₁ , I_2ex is the exposure amount corresponding to the exposure image I₂ , W₁ is the corresponding first weight of the exposure image I₁ , and W₂ is the corresponding first weight of the exposure image I₂ , wherein I₁ and I₂ are two adjacent frames of exposure images, and (x, y) represents the coordinate point of the reference image.

在本发明的一个实施例中，根据所述第一权重和第二权重对多帧所述校准图像进行图像合成，包括：对所述第一权重和第二权重进行加权平均计算，得到所述校准图像的权重图；根据所述权重图得到所述待合成图像的每个像素点的像素值；根据所述待合成图像的每个像素点的像素值合成得到所述目标合成图像。In one embodiment of the present invention, image synthesis is performed on multiple frames of the calibration image according to the first weight and the second weight, including: performing weighted averaging calculation on the first weight and the second weight to obtain a weight map of the calibration image; obtaining a pixel value of each pixel point of the image to be synthesized according to the weight map; and synthesizing the target synthesized image according to the pixel value of each pixel point of the image to be synthesized.

在本发明的一个实施例中，对所述第一权重和第二权重进行加权平均计算，包括：其中，I_(x,)为所述目标合成图像，i为当前待合成图像，n和m分别为当前待合成图像的尺寸，Value_i(,)为当前待合成图像的像素值，B_i(,)为非运动区域的值，W_i(x,y)为校准图像对应的第一权重，W′_i(,)为运动区域对应的第二权重。In one embodiment of the present invention, performing weighted average calculation on the first weight and the second weight includes: Wherein, I_(x,) is the target synthesized image, i is the current image to be synthesized, n and m are the sizes of the current image to be synthesized, Value_i(,) is the pixel value of the current image to be synthesized, Bi_(,) is the value of the non-motion area, Wi_(x,y) is the first weight corresponding to the calibration image, and W′i_{(,) is} the second weight corresponding to the motion area.

为了实现上述目的，本发明第二方面实施例提出了一种图像处理装置，包括：获取模块，用于获取多帧不同的曝光图像；校准模块，用于对所述曝光图像进行校准，得到校准图像；第一计算模块，用于计算所述校准图像对应的第一权重；处理模块，用于确定所述校准图像的运动区域；第二计算模块，用于计算所述校准图像的运动区域第二权重；合成模块，用于根据所述第一权重和第二权重对多帧所述校准图像进行图像合成，得到目标合成图像。In order to achieve the above-mentioned objectives, the second aspect of the present invention proposes an image processing device, including: an acquisition module, used to acquire multiple frames of different exposure images; a calibration module, used to calibrate the exposure image to obtain a calibration image; a first calculation module, used to calculate a first weight corresponding to the calibration image; a processing module, used to determine the motion area of the calibration image; a second calculation module, used to calculate a second weight of the motion area of the calibration image; and a synthesis module, used to synthesize multiple frames of the calibration image according to the first weight and the second weight to obtain a target synthesized image.

根据本发明实施例的图像处理装置，通过对曝光图像校准，得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和校准图像的运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像校准，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过重新计算校准图像和校准图像的运动区域对应的权重对校准图像，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。According to the image processing device of the embodiment of the present invention, the exposure image is calibrated to obtain a calibration image, the motion area of the calibration image is determined, the first weight corresponding to the calibration image and the second weight corresponding to the motion area of the calibration image are calculated, and then multiple frames of calibration images are synthesized according to the first weight and the second weight to obtain a target synthetic image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image calibration, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by recalculating the calibration image and the weight corresponding to the motion area of the calibration image, the generation of synthetic image ghosts is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

为了实现上述目的，本发明第三方面实施例提出了一种电子设备，该电子设备包括如本发明上述第二方面实施例所述的图像处理装置；或者，该电子设备包括：处理器、存储器、存储在所述存储器上并可在所述处理器上运行的图像处理程序，所述图像处理程序被所述处理器执行时实现如本发明上述第一方面实施例所述的图像处理方法。In order to achieve the above-mentioned purpose, the third aspect embodiment of the present invention proposes an electronic device, which includes the image processing device as described in the second aspect embodiment of the present invention; or, the electronic device includes: a processor, a memory, and an image processing program stored in the memory and executable on the processor, and when the image processing program is executed by the processor, it implements the image processing method as described in the first aspect embodiment of the present invention.

根据本发明实施例的电子设备，通过对曝光图像校准，得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和校准图像的运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像校准，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过重新计算校准图像和校准图像的运动区域对应的权重对校准图像，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。According to the electronic device of the embodiment of the present invention, by calibrating the exposure image, a calibration image is obtained, the motion area of the calibration image is determined, the first weight corresponding to the calibration image and the second weight corresponding to the motion area of the calibration image are calculated, and then multiple frames of calibration images are synthesized according to the first weight and the second weight to obtain a target synthesized image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image calibration, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by recalculating the calibration image and the weight corresponding to the motion area of the calibration image, the generation of ghost images of the synthesized image is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

为了实现上述目的，本发明第四方面实施例提出了一种计算机可读存储介质，所述计算机可读存储介质上存储有图像处理程序，所述图像处理程序被处理器执行时实现如本发明上述第一方面实施例所述的图像处理方法。In order to achieve the above-mentioned objectives, the fourth aspect embodiment of the present invention proposes a computer-readable storage medium, on which an image processing program is stored. When the image processing program is executed by a processor, the image processing method as described in the first aspect embodiment of the present invention is implemented.

根据本发明实施例的计算机可读存储介质，其上存储的图像处理程序被处理器执行时，通过对曝光图像校准，得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和校准图像的运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像校准，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过重新计算校准图像和校准图像的运动区域对应的权重对校准图像，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。According to the computer-readable storage medium of the embodiment of the present invention, when the image processing program stored thereon is executed by the processor, the exposure image is calibrated to obtain a calibration image, the motion area of the calibration image is determined, the first weight corresponding to the calibration image and the second weight corresponding to the motion area of the calibration image are calculated, and then multiple frames of calibration images are synthesized according to the first weight and the second weight to obtain a target synthetic image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image calibration, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by recalculating the calibration image and the weight corresponding to the motion area of the calibration image for the calibration image, the generation of synthetic image ghosts is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

本发明的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本发明的实践了解到。Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention.

附图说明BRIEF DESCRIPTION OF THE DRAWINGS

本发明的上述和/或附加的方面和优点从结合下面附图对实施例的描述中将变得明显和容易理解，其中：The above and/or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

图1是根据本发明一个实施例的图像处理方法的流程图；FIG1 is a flow chart of an image processing method according to an embodiment of the present invention;

图2是根据本发明一个实施例的图像校正过程示意图；FIG2 is a schematic diagram of an image correction process according to an embodiment of the present invention;

图3是根据本发明一个实施例的确定校准图像的运动区域的过程示意图；FIG3 is a schematic diagram of a process of determining a motion region of a calibration image according to an embodiment of the present invention;

图4是根据本发明一个实施例的计算校准图像中非运动区域对应的第一权重的过程示意图；4 is a schematic diagram of a process of calculating a first weight corresponding to a non-motion area in a calibration image according to an embodiment of the present invention;

图5是根据本发明一个实施例的图像处理装置的结构示意图。FIG. 5 is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present invention.

具体实施方式DETAILED DESCRIPTION

为了能够更加详尽地了解本发明实施例的特点与技术内容，下面结合附图对本发明实施例的实现进行详细阐述，所附附图仅供参考说明之用，并非用来限定本发明实施例。在以下的技术描述中，为方便解释起见，通过多个细节以提供对所披露实施例的充分理解。然而，在没有这些细节的情况下，一个或多个实施例仍然可以实施。在其它情况下，为简化附图，熟知的结构和装置可以简化展示。In order to be able to understand the features and technical contents of the embodiments of the present invention in more detail, the implementation of the embodiments of the present invention is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present invention. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

目前的图像传感器未能够完全获取自然场景下的所有信息，不能有效还原自然场景下的真实图像信息，使得图像的成像质量不高。为了提高图像的成像质量，需要将这样的LDR(Low Dynamic Range，低动态范围)图像通过相应的图像处理技术转换成HDR(HighDynamic Range，高动态范围)图像。The current image sensors cannot fully acquire all the information in the natural scene, and cannot effectively restore the real image information in the natural scene, resulting in low image quality. In order to improve the image quality, such LDR (Low Dynamic Range) images need to be converted into HDR (High Dynamic Range) images through corresponding image processing technology.

相关技术中，高动态范围图像的合成方式主要是以图像的信息熵或对比度或Dense SIFT(Scale-invariant feature transform，尺度不变特征变换)等描述算子进行权重的计算，进而对多曝光图像进行融合得到高动态范围的图像。然而，上述这种高动态范围图像的合成方式，在场景为静态时尚可保证合成图像的质量，但在场景为变化动态场景时，无法保证图像质量。In the related art, the synthesis method of high dynamic range images is mainly to calculate the weights based on the image information entropy or contrast or Dense SIFT (Scale-invariant feature transform) and other descriptive operators, and then fuse multiple exposure images to obtain high dynamic range images. However, the above-mentioned synthesis method of high dynamic range images can guarantee the quality of the synthesized image when the scene is static, but cannot guarantee the image quality when the scene is a dynamic scene.

由此，通过图像校准，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量，通过重新计算校准图像中运动区域和非运动区域对应的权重对校准图像进行图像合成，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。Therefore, through image calibration, the scene positions of exposure images with different exposure times are matched, the differences caused by the movement of the image sensor during shooting are eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is beneficial to improving image quality. The calibration images are synthesized by recalculating the weights corresponding to the moving areas and non-moving areas in the calibration images, which effectively eliminates the generation of ghost images in the synthesized images, thereby further improving image quality.

下面参考图1-图5描述根据本发明实施例的图像处理方法、装置、电子设备及计算机可读存储介质。The following describes an image processing method, an apparatus, an electronic device, and a computer-readable storage medium according to embodiments of the present invention with reference to FIGS. 1 to 5 .

图1是根据本发明一个实施例的图像处理方法的流程图。如图1所示，该图像处理方法，包括步骤S1～步骤S6。Fig. 1 is a flow chart of an image processing method according to an embodiment of the present invention. As shown in Fig. 1, the image processing method includes steps S1 to S6.

步骤S1：获取多帧不同的曝光图像。Step S1: Acquire multiple frames of images with different exposures.

在具体实施例中，可获取图像传感器输入的多帧不同的曝光图像，例如记作I_n。多帧不同的曝光图像指多张曝光图像的曝光度，也即曝光水平不同，例如，在拍摄时，图像传感器可能会发生移动或抖动等，进而导致不同帧曝光图像之间存在曝光时间间隔，使得不同帧曝光图像的场景位置不匹配，或者拍摄角度有差异，导致不同帧曝光图像之间产生位移偏差，进而产生模糊现象。图像传感器例如为但不限于为CMOS图像传感器。In a specific embodiment, multiple frames of different exposure images input by an image sensor may be obtained, for example, recorded as_In . Multiple frames of different exposure images refer to multiple exposure images with different exposure degrees, that is, different exposure levels. For example, during shooting, the image sensor may move or shake, etc., which may result in an exposure time interval between different frames of exposure images, making the scene positions of different frames of exposure images mismatched, or the shooting angles different, resulting in displacement deviations between different frames of exposure images, and thus causing blurring. The image sensor is, for example, but not limited to, a CMOS image sensor.

步骤S2，对曝光图像进行校准，得到校准图像。Step S2, calibrating the exposure image to obtain a calibrated image.

其中，校准图像为经过配准和校准后的图像，获取多帧不同的曝光图像后，对图像进行校准处理，得到校准图像。The calibration image is an image that has been registered and calibrated. After acquiring multiple frames of images with different exposures, the images are calibrated to obtain the calibration image.

在实施例中，如上所述，获取多帧不同的曝光图像因为拍摄时图像传感器发生移动或抖动导致其相互之间存在曝光时间间隔，使得多帧曝光图像之间的曝光水平不同，进而影响图像质量，因此，对曝光图像进行校准，以将不同曝光图像校准到同一拍摄角度下，从而降低多个曝光图像的因拍摄角度会存在差异而对图像质量的影响，进而提升图像质量。由此，经过校准后得到的多个校准图像，其曝光水平相同，且拍摄角度相同，进而利于在图像合成时因不同像素级融合产生的模糊现象，消除鬼影，提高图像质量。In the embodiment, as described above, when acquiring multiple frames of different exposure images, there is an exposure time interval between them because the image sensor moves or shakes during shooting, so that the exposure levels between the multiple frames of exposure images are different, which in turn affects the image quality. Therefore, the exposure images are calibrated to calibrate the different exposure images to the same shooting angle, thereby reducing the impact of the multiple exposure images on the image quality due to the differences in shooting angles, thereby improving the image quality. Therefore, the multiple calibration images obtained after calibration have the same exposure level and the same shooting angle, which is beneficial to the blur phenomenon caused by the fusion of different pixel levels during image synthesis, eliminating ghost images, and improving image quality.

步骤S3，计算校准图像对应的第一权重。Step S3, calculating a first weight corresponding to the calibration image.

在实施例中，确定校准图像后，通过对比度、曝光度及图像熵计算校准图像的第一权重，可以理解的是，通过计算校准图像的第一权重，便于根据第一权重实现对校准图像的合成。In an embodiment, after the calibration image is determined, the first weight of the calibration image is calculated by contrast, exposure and image entropy. It can be understood that by calculating the first weight of the calibration image, it is convenient to synthesize the calibration image according to the first weight.

步骤S4：确定校准图像的运动区域。Step S4: Determine the motion area of the calibration image.

具体而言，即在得到校准图像后，确定校准图像的运动区域。图像传感器拍摄动态场景时，动态场景下存在移动物体，运动区域即动态场景下图像中移动物体所在的部分，反之，运动区域之外的部分即为非运动区域，也称为背景区域。具体的，前述图像校准过程解决了图像传感器移动为HDR图像带来的影响，但还未能消除图像中移动物体产生的鬼影问题。因此，本发明实施例通过获取校准图像的运动图，确定其运动区域，并在确定校准图像的运动区域后，对校准图像的运动区域取反，得到非运动区域，在得到非运动区域后，便于提高非运动区域短曝光图像的信噪比，并针对运动区域，重新赋予其对应的权重，最终得到合成后的去鬼影图像，从而提高图像质量。Specifically, after obtaining the calibration image, the motion area of the calibration image is determined. When the image sensor shoots a dynamic scene, there are moving objects in the dynamic scene. The motion area is the part of the image in the dynamic scene where the moving object is located. Conversely, the part outside the motion area is the non-motion area, also called the background area. Specifically, the aforementioned image calibration process solves the impact of image sensor movement on the HDR image, but has not yet eliminated the ghost problem caused by moving objects in the image. Therefore, the embodiment of the present invention obtains a motion map of the calibration image to determine its motion area, and after determining the motion area of the calibration image, the motion area of the calibration image is inverted to obtain the non-motion area. After obtaining the non-motion area, it is easy to improve the signal-to-noise ratio of the short exposure image of the non-motion area, and for the motion area, re-assign its corresponding weight, and finally obtain a synthesized ghost-free image, thereby improving the image quality.

步骤S5：计算校准图像的运动区域第二权重。Step S5: Calculate the second weight of the motion area of the calibration image.

具体的，计算运动区域的权重，即为第二权重，进而在计算出第一权重和第二权重之后，可根据第一权重和第二权重来进行多帧校准图像的合成。Specifically, the weight of the motion region is calculated, that is, the second weight. After the first weight and the second weight are calculated, the synthesis of multiple calibration images can be performed according to the first weight and the second weight.

步骤S6：根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。Step S6: synthesize the multiple frames of calibration images according to the first weight and the second weight to obtain a target synthesized image.

具体的，在进行图像合成时，根据第一权重和第二权重对相邻的两帧校准图像进行合成，进而据此实现对多帧不同的曝光图像对应的多帧校准图像进行图像合成，得到最终的消除鬼影，图像质量高的目标合成图像。Specifically, when performing image synthesis, two adjacent frames of calibration images are synthesized according to the first weight and the second weight, and then image synthesis of multiple frames of calibration images corresponding to multiple frames of different exposure images is achieved, so as to obtain the final target synthesized image with ghost elimination and high image quality.

从而，上述的图像处理方法，通过对曝光图像校准，得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和校准图像的运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像校准，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过重新计算校准图像和校准图像的运动区域对应的权重对校准图像，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。Thus, the above-mentioned image processing method obtains a calibration image by calibrating the exposure image, determines the motion area of the calibration image, calculates the first weight corresponding to the calibration image and the second weight corresponding to the motion area of the calibration image, and then synthesizes multiple frames of calibration images according to the first weight and the second weight to obtain a target synthetic image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image calibration, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by recalculating the calibration image and the weight corresponding to the motion area of the calibration image, the generation of synthetic image ghosts is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

在一些实施例中，对曝光图像进行校准，包括：对曝光图像进行图像配准，得到配准图像；对配准图像进行图像校正，得到校准图像。In some embodiments, calibrating the exposure image includes: performing image registration on the exposure image to obtain a registered image; and performing image correction on the registered image to obtain a calibrated image.

具体而言，在本发明实施例中，在不同曝光图像合成时对各曝光图像进行图像配准，使各曝光图像的曝光量处于同一水平，进而使具有不同曝光时间的各曝光图像的场景位置相匹配，从而避免像素级融合时产生的模糊现象，进而利于提升合成图像的图像质量。Specifically, in an embodiment of the present invention, image registration is performed on each exposure image when different exposure images are synthesized so that the exposure amount of each exposure image is at the same level, and then the scene position of each exposure image with different exposure time is matched, thereby avoiding blurring caused by pixel-level fusion, and thus helping to improve the image quality of the synthesized image.

如前，图像传感器在拍摄时可能会发生移动或抖动，导致曝光图像相互之间存在曝光时间间隔，进而使具有不同曝光时间的各曝光图像的场景位置相匹配，多个曝光图像的拍摄角度会存在差异，进而影响图像质量。因此，在本发明的实施例中，在对图像进行配准之后，得到配准图像，并对配准图像进一步进行校正，以将不同曝光图像校正到同一拍摄角度下，从而降低多个曝光图像因拍摄角度存在差异而对图像质量的影响，进而提升图像质量。由此，经过配准和校正后得到的多个校准图像，其曝光水平相同，且拍摄角度相同，进而利于在图像合成时因不同像素级融合产生的模糊现象，消除鬼影，提高图像质量。As mentioned above, the image sensor may move or shake during shooting, resulting in an exposure time interval between the exposure images, thereby matching the scene positions of the exposure images with different exposure times, and the shooting angles of the multiple exposure images will be different, thereby affecting the image quality. Therefore, in an embodiment of the present invention, after the images are registered, a registered image is obtained, and the registered image is further corrected to correct the different exposure images to the same shooting angle, thereby reducing the impact of the differences in shooting angles of the multiple exposure images on the image quality, thereby improving the image quality. As a result, the multiple calibrated images obtained after registration and correction have the same exposure level and the same shooting angle, which is beneficial to eliminate the blurring phenomenon caused by the fusion of different pixel levels during image synthesis, eliminate ghosting, and improve image quality.

在本发明的一个实施例中，对曝光图像进行图像配准，包括：按照预设的曝光曲线将曝光图像的曝光量置于预设曝光水平，得到调整后的曝光图像，调整后的曝光图像包括参考图像和预配准图像；获取预配准图像和参考图像之间的特征点对，以得到配准图像。In one embodiment of the present invention, image registration is performed on an exposure image, including: placing the exposure amount of the exposure image at a preset exposure level according to a preset exposure curve to obtain an adjusted exposure image, the adjusted exposure image including a reference image and a pre-registered image; acquiring feature point pairs between the pre-registered image and the reference image to obtain a registered image.

具体而言，在图像传感器拍摄时，动态场景中不光有移动的物体，还有图像传感器自身移动的可能性，所以在不同曝光图像合成时需要进行图像配准，使具有不同曝光时间的曝光图像的场景位置相匹配，从而避免了像素级融合时产生的模糊现象，利于提高图像质量。具体的图像配准过程包括：按照预设的曝光曲线将曝光图像的曝光量置于预设曝光水平，得到调整后的曝光图像，调整后的曝光图像包括参考图像和预配准图像，通过获取预配准图像与参考图像之间的特征点对，以根据特征点对的匹配结果，得到配准图像，由此，可将多帧不同的曝光图像均置于预设曝光水平，使其曝光量相同，具有相同的曝光水平。换言之，即针对输入的多帧不同曝光图像I_n，按照预设的曝光曲线将多帧不同曝光图像置于同一曝光水平，得到调整后的曝光图形，以便根据预配准图像及参考图像之间的特征点对得到配准图像，其中，不同曝光图像置于同一曝光水平例如I^′_n＝f(x,y)，其中f(x,y)为预设的曝光曲线，I^′_n为预配准图像，(x,y)为参考图像的像素点位置坐标。Specifically, when the image sensor is shooting, there are not only moving objects in the dynamic scene, but also the possibility of the image sensor itself moving. Therefore, image registration is required when synthesizing different exposure images to match the scene positions of the exposure images with different exposure times, thereby avoiding the blurring phenomenon caused by pixel-level fusion and improving image quality. The specific image registration process includes: placing the exposure amount of the exposure image at a preset exposure level according to a preset exposure curve to obtain an adjusted exposure image, the adjusted exposure image includes a reference image and a pre-registered image, and obtaining a registered image based on the matching results of the feature point pairs by obtaining feature point pairs between the pre-registered image and the reference image. Thus, multiple frames of different exposure images can be placed at a preset exposure level so that their exposure amounts are the same and they have the same exposure level. In other words, for the input multiple frames of differently exposed images I_n , the multiple frames of differently exposed images are placed at the same exposure level according to the preset exposure curve to obtain an adjusted exposure pattern, so as to obtain a registered image based on the feature point pairs between the pre-registered image and the reference image, wherein the differently exposed images are placed at the same exposure level, for example, I^′_n =f(x, y), wherein f(x, y) is the preset exposure curve, I^′_n is the pre-registered image, and (x, y) is the pixel position coordinate of the reference image.

在本发明的一个实施例中，获取预配准图像和参考图像之间的特征点对，包括：对预配准图像和参考图像进行特征点检测，对应得到预配准图像和参考图像的特征点信息；参考图像可以包括预设原点，从预设原点开始，以预设匹配范围对预配准图像和参考图像进行特征点匹配，得到特征点对。In one embodiment of the present invention, obtaining feature point pairs between a pre-registered image and a reference image includes: performing feature point detection on the pre-registered image and the reference image to obtain feature point information of the pre-registered image and the reference image respectively; the reference image may include a preset origin, starting from the preset origin, performing feature point matching on the pre-registered image and the reference image within a preset matching range to obtain feature point pairs.

具体而言，即对预配准图像I^′_n和参考图像分别进行特征点检测，对应得到预配准图像和参考图像的特征点信息。在本发明的实施例中，可采用SURF(Speeded Up RobustFeatures，快速特征点检测)算法来对预配准图像I^′_n和参考图像分别进行特征点检测，从而提高特征点检测的效率和准确性。进一步地，得到的特征点信息例如包括：图像的特征点位置和特征矩阵的特征点信息。Specifically, feature point detection is performed on the pre-registered image I^′_n and the reference image, respectively, and feature point information of the pre-registered image and the reference image is obtained accordingly. In an embodiment of the present invention, a SURF (Speeded Up Robust Features, fast feature point detection) algorithm can be used to perform feature point detection on the pre-registered image I^′_n and the reference image, respectively, so as to improve the efficiency and accuracy of feature point detection. Furthermore, the feature point information obtained includes, for example, the feature point position of the image and the feature point information of the feature matrix.

在得到特征点信息后，从参考图像的预设原点开始，以预设匹配范围对预配准图像和参考图像进行特征点匹配，得到特征点对。具体的，预设原点为在匹配图像中预先选定的点，以该点为起始点或初始点进行特征点匹配，初始匹配范围为预设匹配范围，当匹配到特征点后，记录相应的特征点对。在具体实施例中，例如以参考图像上例如I(0,0)作为预设原定，从I(0,0)开始进行特征点匹配，初始匹配范围R为10。相较于通过选取一个初始点后进行全局搜索匹配的方式，本发明实施例通过选取初始点，并设定匹配范围，在匹配范围内进行特征点匹配，无需进行全局匹配，利于在节省资源的情况下尽可能快的进行特征点匹配，因此，匹配效率更高。After obtaining the feature point information, starting from the preset origin of the reference image, feature point matching is performed on the pre-registered image and the reference image within a preset matching range to obtain a feature point pair. Specifically, the preset origin is a point pre-selected in the matching image, and feature point matching is performed with the point as the starting point or initial point. The initial matching range is the preset matching range. When the feature point is matched, the corresponding feature point pair is recorded. In a specific embodiment, for example, I(0,0) on the reference image is used as the preset origin, and feature point matching is performed starting from I(0,0), and the initial matching range R is 10. Compared to the method of performing global search and matching after selecting an initial point, the embodiment of the present invention selects an initial point and sets a matching range to perform feature point matching within the matching range. There is no need to perform global matching, which is conducive to performing feature point matching as quickly as possible while saving resources. Therefore, the matching efficiency is higher.

在本发明的一个实施例中，结合图2所示，对配准图像进行校正，包括：步骤S32：计算特征点对的距离；步骤S33：统计计算得到的特征点对的距离，将满足预设条件的特征点对的距离作为图像偏移量；步骤S34：计算配准图像所需的偏移方向；步骤S35：根据图像偏移量和偏移方向对配准图像进行偏移操作，以对配准图像进行校正，得到校准图像。进而，通过图像校正，对于移动的图像传感器合成高动态图像有很好的成像效果，有效的避免了图像传感器在移动过程中对待合成图像产生的差异，利于提高合成图像质量。In one embodiment of the present invention, as shown in FIG2, the registration image is corrected, including: step S32: calculating the distance of the feature point pair; step S33: statistically calculating the distance of the feature point pair, and taking the distance of the feature point pair that meets the preset conditions as the image offset; step S34: calculating the offset direction required for the registration image; step S35: performing an offset operation on the registration image according to the image offset and the offset direction to correct the registration image and obtain a calibrated image. Furthermore, through image correction, the mobile image sensor has a good imaging effect for synthesizing high-dynamic images, effectively avoiding the difference in the image sensor to be synthesized during the movement process, which is conducive to improving the quality of the synthesized image.

具体而言，即在图像配准之后，对配准图像进一步进行图像校正，得到校准图像。具体包括：计算参考图像与预配准图像进行特征点匹配，得到特征点对；计算得到的特征点对中两个匹配的特征点之间距离。由于得到的特征点对一般为多个，因此，计算得到的特征点对的距离也对应为多个，统计多个特征点对的距离，将满足预设条件的特征点对的距离作为图像偏移量，也即配准图像当前的所需的偏移量；进一步地，计算配准图像所需的偏移方向，进而，在确定好图像偏移量和偏移方向后，即可根据图像偏移量和偏移方向来对配准图像进行偏移操作，得到校准图像，可以理解的是，经过偏移操作后得到的校准图像，消除了图像偏差，从而准确性更好，利于提高合成图像的质量，其中，参考图像可以为多帧曝图像中满足要求的标准图像。Specifically, after the image registration, the registered image is further corrected to obtain a calibrated image. Specifically, it includes: calculating the feature point matching between the reference image and the pre-registered image to obtain a feature point pair; calculating the distance between two matched feature points in the obtained feature point pair. Since the obtained feature point pairs are generally multiple, the distance of the feature point pairs calculated is also multiple, and the distances of multiple feature point pairs are counted, and the distances of the feature point pairs that meet the preset conditions are used as the image offset, that is, the current required offset of the registered image; further, the offset direction required for the registered image is calculated, and then, after determining the image offset and offset direction, the registered image can be offset according to the image offset and offset direction to obtain a calibrated image. It can be understood that the calibrated image obtained after the offset operation eliminates the image deviation, so that the accuracy is better, which is conducive to improving the quality of the composite image, wherein the reference image can be a standard image that meets the requirements in multiple frames of exposure images.

在本发明的一个实施例中，还包括：若在预设匹配范围进行内未匹配到特征点对，则按照预设步长扩大匹配范围，以重新进行特征点匹配，直至得到特征点对。In one embodiment of the present invention, the method further includes: if no feature point pair is matched within the preset matching range, the matching range is expanded according to a preset step length to re-match the feature points until the feature point pair is obtained.

具体而言，即在初始匹配范围内如果未能匹配到特征点，则需进一步扩大匹配范围，以确保能够得到特征点对，提高特征点匹配的可靠性。在本发明实施例中，以预设步长来逐步扩大匹配范围。在具体实施例中，初始匹配范围例如为10，预设步长例如为5，即未匹配到特征点时，每次按照步长为5来扩大匹配范围，从而在节省资源的情况下尽可能快的进行特征点匹配，同时提高特征点匹配的可靠性。Specifically, if the feature point cannot be matched within the initial matching range, the matching range needs to be further expanded to ensure that the feature point pair can be obtained and to improve the reliability of the feature point matching. In an embodiment of the present invention, the matching range is gradually expanded with a preset step size. In a specific embodiment, the initial matching range is, for example, 10, and the preset step size is, for example, 5, that is, when the feature point is not matched, the matching range is expanded with a step size of 5 each time, so that the feature point matching is performed as quickly as possible while saving resources, and the reliability of the feature point matching is improved.

在本发明的一个实施例中，上述步骤S32中，计算特征点对的距离，包括：以参考图像为基准，计算出参考图像与配准图像的相匹配的特征点之间的欧氏距离，将计算的欧氏距离作为特征点对的距离。In one embodiment of the present invention, in the above step S32, calculating the distance of the feature point pair includes: taking the reference image as a reference, calculating the Euclidean distance between the matching feature points of the reference image and the registered image, and using the calculated Euclidean distance as the distance of the feature point pair.

具体而言，即以预设的参考图像为基准，计算出参考图像与配准图像之间相匹配的特征点之间的欧式距离，以作为参考图像和配准图像之间的特征点对的距离。在具体实施例中，可通过如下计算式来计算特征点对的距离：Dis(x,y)＝|I(x₁,y₁)-I′(x₂,y₂)|，其中，Dis(x,y)为计算的特征点对的距离，I(x₁,y₁)为参考图像，I′(x₂,y₂)为配准图像。Specifically, the Euclidean distance between the feature points that match the reference image and the registered image is calculated based on the preset reference image as the reference image, and is used as the distance between the feature point pairs between the reference image and the registered image. In a specific embodiment, the distance between the feature point pairs can be calculated by the following calculation formula: Dis(x,y)=|I(x₁ ,y₁ )-I′(x₂ ,y₂ )|, where Dis(x,y) is the calculated distance between the feature point pairs, I(x₁ ,y₁ ) is the reference image, and I′(x₂ ,y₂ ) is the registered image.

在本发明的一个实施例中，步骤S33中，满足预设条件的特征点对的距离包括：出现次数最多的特征点对的距离。In one embodiment of the present invention, in step S33, the distance of the feature point pair that meets the preset condition includes: the distance of the feature point pair that appears the most times.

具体而言，即在计算得到特征点对的距离后，对得到的特征点对的距离进行统计，特征点对的距离的大小成正态分布，将出现次数最多的距离值作为配准图像和参考图像之间的偏移量，也即图像偏移量。Specifically, after calculating the distance of the feature point pairs, the distance of the feature point pairs is statistically analyzed. The distance of the feature point pairs is normally distributed, and the distance value with the most occurrences is used as the offset between the registered image and the reference image, that is, the image offset.

在本发明的一个实施例中，步骤S34中，计算配准图像所需的偏移方向，包括：以参考图像的像素点坐标为原点计算配准图像中对应像素点坐标所在的象限位置；根据象限位置确定配准图像所需的偏移方向。In one embodiment of the present invention, in step S34, calculating the offset direction required for registering the image includes: calculating the quadrant position of the corresponding pixel point coordinates in the registering image with the pixel point coordinates of the reference image as the origin; and determining the offset direction required for registering the image according to the quadrant position.

具体的，设参考图像为I(x₁,y₁)为参考图像，(x₁,y₁)为其像素点坐标，I′(x₂,y₂)为配准图像，(x₂,y₂)为其像素点坐标。以(x₁,y₁)为原点，计算(x₂,y₂)所在的象限位置，从而计算出配准图像所需的偏移方向，然后进行偏移操作。在具体实施例中，可通过如下计算式来计算配准图像所需的偏移方向：其中，Dir(x,y)为偏移方向。Specifically, let the reference image I(x₁ ,y₁ ) be the reference image, (x₁ ,y₁ ) be its pixel coordinates, I′(x₂ ,y₂ ) be the registration image, (x₂ ,y₂ ) be its pixel coordinates. With (x₁ ,y₁ ) as the origin, calculate the quadrant position of (x₂ ,y₂ ) to calculate the offset direction required for the registration image, and then perform the offset operation. In a specific embodiment, the offset direction required for the registration image can be calculated by the following calculation formula: Among them, Dir(x,y) is the offset direction.

在本发明的一个实施例中，在根据图像偏移量和偏移方向对配准图像进行偏移操作之后，还包括：对偏移后的配准图像的边界进行补零处理，以使其与对应的曝光图像的尺寸相同，以得到校准图像。换言之，即针对偏移操作后的配准图像，将其边界补零凑成与初始获取的对应曝光图像同样大小的尺寸，以得到校准图像，从而便于后续的图像合成。In one embodiment of the present invention, after performing an offset operation on the registered image according to the image offset amount and the offset direction, the method further includes: padding the border of the offset registered image with zeros to make it the same size as the corresponding exposure image, so as to obtain a calibration image. In other words, for the offset registered image, padding the border with zeros to make it the same size as the initially acquired corresponding exposure image, so as to obtain a calibration image, thereby facilitating subsequent image synthesis.

在本发明的一个实施例中，如图3所示，确定校准图像的运动区域，包括：步骤S41：获取校准图像和参考图像的二值图像；步骤S42：将校准图像和参考图像的二值图像相减求绝对值，得到校准图像的运动区域。In one embodiment of the present invention, as shown in FIG3 , determining the motion region of the calibration image includes: step S41 : acquiring binary images of the calibration image and the reference image; step S42 : subtracting the binary images of the calibration image and the reference image to obtain an absolute value, thereby obtaining the motion region of the calibration image.

具体而言，即先确定校准图像和参考图像各自对应的二值图像，然后将两个二值图像相减求取绝对值，此时得到的差值图即是校准图像和参考图像相对运动部分的标记图，也即校准图像的运动区域。Specifically, first determine the binary images corresponding to the calibration image and the reference image, then subtract the two binary images to obtain the absolute value. The difference image obtained at this time is the marking image of the relative motion part of the calibration image and the reference image, that is, the motion area of the calibration image.

在本发明的一个实施例中，步骤S41中，获取校准图像的二值图像，包括：对校准图像进行灰度化处理，得到第一灰度图；确定用于划分第一灰度图的第一阈值；根据第一阈值对第一灰度图进行划分，得到校准图像的二值图像。In one embodiment of the present invention, in step S41, a binary image of the calibration image is obtained, including: grayscale processing of the calibration image to obtain a first grayscale image; determining a first threshold for dividing the first grayscale image; and dividing the first grayscale image according to the first threshold to obtain a binary image of the calibration image.

具体而言，对校准图像进行灰度化处理，包括：Gray₁＝0.3R+0.59G+0.11B，其中，Gray₁为第一灰度图的灰度值，R为红色分量，G为绿色分量，B为蓝色分量。对校准图像进行灰度化处理后，进一步确定第一阈值，记作Threshold1，Threshold1用于划分第一灰度图，可据此对第一灰度图划分得到校准图像的二值图像。Specifically, the calibration image is grayed, including: Gray₁ = 0.3R + 0.59G + 0.11B, where Gray₁ is the gray value of the first gray image, R is the red component, G is the green component, and B is the blue component. After the calibration image is grayed, a first threshold is further determined, recorded as Threshold1, and Threshold1 is used to divide the first gray image, and the first gray image can be divided to obtain a binary image of the calibration image.

在本发明的一个实施例中，确定用于划分第一灰度图的第一阈值，包括：统计第一灰度图的直方图，累加每个像素值出现的次数，得到第一总次数；将统计的直方图上每个像素值的出现的次数从小到大依次累加，直到累加次数大于或等于第一总次数的一半时，确定当前所对应的像素值为第一阈值。In one embodiment of the present invention, determining a first threshold for dividing a first grayscale image includes: counting a histogram of the first grayscale image, accumulating the number of occurrences of each pixel value, and obtaining a first total number of occurrences; accumulating the number of occurrences of each pixel value on the counted histogram from small to large until the accumulated number of occurrences is greater than or equal to half of the first total number of occurrences, and determining that the current corresponding pixel value is the first threshold.

具体而言，即统计第一灰度图的直方图，累加每个像素级上出现的次数，计第一总次数，记作SumAll1，计算SumHalf1＝SumAll1/2，将统计的直方图上每个像素级上的次数Value1从小到大依次累加，直到Value1>＝SumHalf1，此时所在的像素级即是划分第一灰度图的第一阈值Threshold1。Specifically, the histogram of the first grayscale image is counted, the number of occurrences at each pixel level is accumulated, the first total number is calculated, recorded as SumAll1, SumHalf1=SumAll1/2 is calculated, and the number of times Value1 at each pixel level on the statistical histogram is accumulated from small to large, until Value1>=SumHalf1, and the pixel level at this time is the first threshold Threshold1 for dividing the first grayscale image.

在本发明的一个实施例中，根据第一阈值对第一灰度图进行划分，包括：判断第一灰度图中各像素点的像素值是否大于或等于第一阈值；将像素值大于或等于第一阈值的像素点置为1，将像素值小于第一阈值的像素点置为0，以得到校准图像的二值图像。In one embodiment of the present invention, the first grayscale image is divided according to a first threshold, including: determining whether the pixel value of each pixel in the first grayscale image is greater than or equal to the first threshold; setting the pixel values greater than or equal to the first threshold to 1, and setting the pixel values less than the first threshold to 0, so as to obtain a binary image of the calibration image.

具体而言，即判断第一灰度图中各像素点的像素值是否大于或等于第一阈值Threshold1，如果某个像素点的像素值大于或等于Threshold1，则将该像素点用1表示，如果某个像素点的像素值小于Threshold1，则Threshold1用0表示，由此，可得到由Threshold1划分的二值图像，即得到校准图像对应的二值图像。Specifically, it is to determine whether the pixel value of each pixel in the first grayscale image is greater than or equal to the first threshold Threshold1. If the pixel value of a pixel is greater than or equal to Threshold1, the pixel is represented by 1. If the pixel value of a pixel is less than Threshold1, Threshold1 is represented by 0. Thus, a binary image divided by Threshold1 can be obtained, that is, a binary image corresponding to the calibration image can be obtained.

在本发明的一个实施例中，步骤S41中，获取参考图像的二值图像，包括：对参考图像进行灰度化处理，得到第二灰度图；确定用于划分第二灰度图的第二阈值；根据第二阈值对第二灰度图进行划分，得到参考图像的二值图像。In one embodiment of the present invention, in step S41, a binary image of a reference image is obtained, including: grayscale processing of the reference image to obtain a second grayscale image; determining a second threshold for dividing the second grayscale image; and dividing the second grayscale image according to the second threshold to obtain a binary image of the reference image.

具体而言，对参考图像进行灰度化处理，包括：Gray₂＝0.3R+0.59G+0.11B，其中，Gray₂为第二灰度图的灰度值，R为红色分量，G为绿色分量，B为蓝色分量。对参考图像进行灰度化处理后，进一步确定第二阈值，记作Threshold2，Threshold2用于划分第二灰度图，可据此对第二灰度图划分得到参考图像的二值图像。Specifically, the reference image is grayed, including: Gray₂ = 0.3R + 0.59G + 0.11B, where Gray₂ is the gray value of the second gray image, R is the red component, G is the green component, and B is the blue component. After the reference image is grayed, a second threshold is further determined, denoted as Threshold2. Threshold2 is used to divide the second gray image, and the second gray image can be divided to obtain a binary image of the reference image.

在本发明的一个实施例中，确定用于划分第二灰度图的第二阈值，包括：统计第二灰度图的直方图，累加每个像素值出现的次数，得到第二总次数；将统计的直方图上每个像素值的出现的次数从小到大依次累加，直到累加次数大于或等于第二总次数的一半时，确定当前所对应的像素值为第二阈值。In one embodiment of the present invention, determining a second threshold for dividing the second grayscale image includes: counting the histogram of the second grayscale image, accumulating the number of occurrences of each pixel value to obtain a second total number of times; accumulating the number of occurrences of each pixel value on the counted histogram from small to large until the accumulated number of times is greater than or equal to half of the second total number of times, and determining that the current corresponding pixel value is the second threshold.

具体而言，即统计第二灰度图的直方图，累加每个像素级上出现的次数，计第二总次数，记作SumAll2，计算SumHalf2＝SumAll2/2，将统计的直方图上每个像素级上的次数Value2从小到大依次累加，直到Value2>＝SumHalf2，此时所在的像素级即是划分第二灰度图的第二阈值Threshold2。Specifically, the histogram of the second grayscale image is counted, the number of occurrences at each pixel level is accumulated, the second total number is calculated, recorded as SumAll2, SumHalf2=SumAll2/2 is calculated, and the number Value2 at each pixel level on the statistical histogram is accumulated from small to large, until Value2>=SumHalf2, and the pixel level at this time is the second threshold Threshold2 for dividing the second grayscale image.

在本发明的一个实施例中，根据第二阈值对第二灰度图进行划分，包括：判断第二灰度图中各像素点的像素值是否大于第二阈值；将像素值大于或等于第二阈值的像素点置为1，将像素值小于第二阈值的像素点置为0，以得到参考图像的二值图像。In one embodiment of the present invention, the second grayscale image is divided according to a second threshold, including: determining whether the pixel value of each pixel in the second grayscale image is greater than the second threshold; setting the pixel values greater than or equal to the second threshold to 1, and setting the pixel values less than the second threshold to 0, so as to obtain a binary image of the reference image.

具体而言，即判断第二灰度图中各像素点的像素值是否大于或等于第二阈值Threshold2，如果某个像素点的像素值大于或等于Threshold2，则将该像素点用1表示，如果某个像素点的像素值小于Threshold2，则Threshold2用0表示，由此，可得到由Threshold2划分的二值图像，即得到参考图像对应的二值图像。Specifically, it is to determine whether the pixel value of each pixel in the second grayscale image is greater than or equal to the second threshold Threshold2. If the pixel value of a pixel is greater than or equal to Threshold2, the pixel is represented by 1. If the pixel value of a pixel is less than Threshold2, Threshold2 is represented by 0. In this way, a binary image divided by Threshold2 can be obtained, that is, a binary image corresponding to the reference image can be obtained.

进一步地，在得到校准图像和参考图像各自对应的二者图像后，将校准图像与参考图像的二值图像相减求绝对值，此时得到的差值图即是校准图像与参考图像的相对运动部分的标记图，即校准图像的运动区域。Furthermore, after obtaining the two images corresponding to the calibration image and the reference image respectively, the binary images of the calibration image and the reference image are subtracted to obtain the absolute value. The difference image obtained at this time is the marking image of the relative motion part between the calibration image and the reference image, that is, the motion area of the calibration image.

在本发明的一个实施例中，计算校准图像对应的第一权重之前，包括：确定所述校准图像的非运动区域。In one embodiment of the present invention, before calculating the first weight corresponding to the calibration image, the method includes: determining a non-motion area of the calibration image.

具体而言，在计算校准图像的第一权重之前，确定校准图像的非运动区域，也即背景区域。Specifically, before calculating the first weight of the calibration image, a non-motion area of the calibration image, that is, a background area, is determined.

在本发明的一个实施例中，确定校准图像的非运动区域，包括：将所述校准图像的运动区域取反，得到所述非运动区域。In one embodiment of the present invention, determining the non-motion region of the calibration image includes: inverting the motion region of the calibration image to obtain the non-motion region.

具体而言，前述已经得到校准图像的运动区域，其为一个二值图，那么将运动区域取反，即可得到非运动区域，也即背景区域。可以理解的是，由于背景区域是经运动区域取反得到的，因此其也为二值图，与运动区域的二值图相反，也即，运动区域的二值图中取1时，在背景区域中对应取0，运动区域的二值图中取0时，在背景区域中对应取1。Specifically, the motion region of the calibration image has been obtained as a binary image, so the motion region is inverted to obtain the non-motion region, that is, the background region. It can be understood that since the background region is obtained by inverting the motion region, it is also a binary image, which is opposite to the binary image of the motion region, that is, when the binary image of the motion region is 1, the corresponding image in the background region is 0, and when the binary image of the motion region is 0, the corresponding image in the background region is 1.

在本发明的一个实施例中，非运动区域的图像包括长曝光图像和短曝光图像，确定校准图像的非运动区域之后，还包括：在RGB颜色空间下，将预设的长曝光图像的信噪比映射到短曝光图像上，以提高短曝光图像的信噪比。In one embodiment of the present invention, the image of the non-moving area includes a long-exposure image and a short-exposure image. After determining the non-moving area of the calibration image, the method further includes: mapping a preset signal-to-noise ratio of the long-exposure image to the short-exposure image in an RGB color space to improve the signal-to-noise ratio of the short-exposure image.

具体而言，即在对运动区域取反得到非运动区域，即背景区域，按照背景区域取值为1的部分进行短曝光图像的信噪比增强，具体包括：在RGB颜色空间下，以预设的长曝光图像的信噪比为标准，将其映射到背景区域的短曝光图像上，以提高短曝光图像的信噪比，由此，通过长曝光来提高短曝光的信噪比，以减少短曝光图像中的噪声水平，有利于最终合成结果的细节信息显示，利于提高图像质量。Specifically, the motion area is inverted to obtain a non-motion area, that is, the background area, and the signal-to-noise ratio of the short exposure image is enhanced according to the part of the background area with a value of 1, specifically including: in the RGB color space, taking the preset signal-to-noise ratio of the long exposure image as the standard, mapping it to the short exposure image of the background area to improve the signal-to-noise ratio of the short exposure image, thereby improving the signal-to-noise ratio of the short exposure through long exposure to reduce the noise level in the short exposure image, which is beneficial to the display of detail information of the final synthesis result and to improve the image quality.

具体计算和处理过程包括：先计算长短曝光中G分量的比值其中，I₁G_(,)和I₂G_(,)分别对应为曝光图像I₁和曝光图像I₂在某一点的G分量大小，I₁和I₂为相邻的两帧曝光图像，(x,y)表示坐标点。进而，在得到RateG之后，可通过RateG分别来计算曝光图像I₁和曝光图像I₂的R分量和B分量的大小，具体地，可通过来计算I₁R_(,)和I₂R_(,)，可通过来计算I₁B_(,)和I₂B_(,)，其中，I₁R_(,)和I₂R_(,)分别对应为曝光图像I₁和曝光图像I₂在某一点的R分量大小，I₁B_(,)和I₂B_(,)分别对应为曝光图像I₁和曝光图像I₂在某一点的B分量大小。由此，可分别得到曝光图像在各通道的分量，进而实现对背景区域取值为1的部分的短曝光图像的信噪比增强。The specific calculation and processing process includes: first calculate the ratio of the G component in the long and short exposures Among them, I₁ G_(,) and I₂ G_(,) correspond to the G component size of the exposure image I₁ and the exposure image I₂ at a certain point, I₁ and I₂ are two adjacent frames of exposure images, and (x, y) represents the coordinate point. Then, after obtaining RateG, the size of the R component and the B component of the exposure image I₁ and the exposure image I₂ can be calculated by RateG. Specifically, To calculate I₁ R_(,) and I₂ R_(,) , we can use To calculate I₁ B_(,) and I₂ B_(,) , where I₁ R_(,) and I₂ R_(,) correspond to the R component size of the exposure image I₁ and the exposure image I₂ at a certain point, and I₁ B_(,) and I₂ B_(,) correspond to the B component size of the exposure image I₁ and the exposure image I₂ at a certain point. In this way, the components of the exposure image in each channel can be obtained respectively, and then the signal-to-noise ratio of the short exposure image with a background area value of 1 is enhanced.

在本发明的一个实施例中，如图4所示，计算校准图像对应的第一权重，包括：计算校准图像的对比度、曝光度和图像熵；通过对比度、曝光度、图像熵计算校准图像对应的权重，以得到第一权重。In one embodiment of the present invention, as shown in FIG. 4 , calculating the first weight corresponding to the calibration image includes: calculating the contrast, exposure and image entropy of the calibration image; and calculating the weight corresponding to the calibration image by the contrast, exposure and image entropy to obtain the first weight.

在本发明的一个实施例中，如图4所示，通过对比度、曝光度、图像熵计算校准图像对应的权重：将对比度、曝光度、图像熵相乘并进行归一化，得到校准图像对应的权重。In one embodiment of the present invention, as shown in FIG. 4 , the weight corresponding to the calibration image is calculated by contrast, exposure, and image entropy: the contrast, exposure, and image entropy are multiplied and normalized to obtain the weight corresponding to the calibration image.

具体而言，即将前述计算的到的相关权重，如对比度、曝光度、图像熵相乘并进行归一化，即可得到校准图像对应的权重。具体的计算式例如为：Specifically, the weights calculated above, such as contrast, exposure, and image entropy, are multiplied and normalized to obtain the weight corresponding to the calibration image. The specific calculation formula is, for example:

其中，W_x,y为第一权重，W_CON为对比度，W_EX为曝光度，W_EN为图像熵，MaxWeight为W_CON的最大值、W_EX的最大值和W_EN的最大值的乘积。Among them, Wx_,y is the first weight,_WCON is the contrast,_WEX is the exposure,_WEN is the image entropy, and MaxWeight is the product of the maximum value of_WCON , the maximum value of_WEX and the maximum value of_WEN .

在本发明的一个实施例中，如图4所示，计算校准图像对应的权重，包括：计算校准图像的对比度、曝光度和图像熵；通过对比度、曝光度、图像熵计算校准图像对应的权重。In one embodiment of the present invention, as shown in FIG. 4 , calculating the weight corresponding to the calibration image includes: calculating the contrast, exposure and image entropy of the calibration image; and calculating the weight corresponding to the calibration image through the contrast, exposure and image entropy.

具体而言，在确定出校准图像后，计算校准图像对应的第一权重，并在确定校准图像的运动区域后即可进行下一步的合成操作，即计算运动区域的权重，对计算得到的两个权重进行合并和加权均值计算后，得到目标合成图像。本发明实施例中，通过对比度、曝光度和图像熵进行校准图像的权重计算，能够有效提高合成图像的动态范围，利于提高图像合成的准确性，进而提升图像质量。Specifically, after determining the calibration image, the first weight corresponding to the calibration image is calculated, and after determining the motion area of the calibration image, the next synthesis operation can be performed, that is, the weight of the motion area is calculated, and the two calculated weights are combined and weighted mean is calculated to obtain the target synthetic image. In the embodiment of the present invention, the weight calculation of the calibration image is performed through contrast, exposure and image entropy, which can effectively improve the dynamic range of the synthetic image, help improve the accuracy of image synthesis, and thus improve the image quality.

在本发明的一个实施例中，计算校准图像的对比度的过程包括：对校准图像对应的曝光图像进行拉普拉斯滤波后，对得到的滤波结果求取绝对值，以得到校准图像的对比度。In one embodiment of the present invention, the process of calculating the contrast of the calibration image includes: performing Laplace filtering on the exposure image corresponding to the calibration image, and then obtaining an absolute value of the obtained filtering result to obtain the contrast of the calibration image.

具体而言，即对曝光图像I_n采用拉普拉斯滤波器进行滤波，对滤波结果求取绝对值，得到对比度，记作W_CON。其中，拉普拉斯滤波器的拉普拉斯核用[0,1,0；1,-4,1；0,1,0]表示。Specifically, the exposure image I_n is filtered using a Laplace filter, and the absolute value of the filtering result is obtained to obtain the contrast, which is recorded as W_CON . The Laplace kernel of the Laplace filter is represented by [0, 1, 0; 1, -4, 1; 0, 1, 0].

在本发明的一个实施例中，计算校准图像的曝光度包括：采用三角权重函数计算校准图像的曝光度，可以有效降低计算量，提高计算效率。In one embodiment of the present invention, calculating the exposure of the calibration image includes: using a triangular weight function to calculate the exposure of the calibration image, which can effectively reduce the amount of calculation and improve the calculation efficiency.

具体而言，采用三角权重函数计算校准图像的曝光度的原理为图像像素值越靠近像素值最大值的二分之一则权重越大，具体计算式例如为：Specifically, the principle of using the triangular weight function to calculate the exposure of the calibration image is that the closer the image pixel value is to half of the maximum pixel value, the greater the weight. The specific calculation formula is, for example:

其中，为计算得到的曝光度，Value为像素点的像素值，MaxValue为像素值中的最大值。Among them, is the calculated exposure, Value is the pixel value of the pixel point, and MaxValue is the maximum value of the pixel values.

在本发明的一个实施例中，计算校准图像的图像熵的过程包括：用预设矩阵作为熵值计算的范围，通过计算预设矩阵内所有像素点出现的概率来计算中间点的熵值，将该熵值作为校准图像的图像熵。In one embodiment of the present invention, the process of calculating the image entropy of the calibration image includes: using a preset matrix as the range of entropy value calculation, calculating the entropy value of the middle point by calculating the probability of occurrence of all pixel points in the preset matrix, and using the entropy value as the image entropy of the calibration image.

在具体实施例中，预设矩阵例如取3×3矩阵。即，采用3×3矩阵作为熵值计算的范围，通过计算3×3矩阵内九个像素出现的概率来计算中间点的熵值，记作W_EN，将该熵值W_EN作为校准图像的图像熵。In a specific embodiment, the preset matrix is, for example, a 3×3 matrix. That is, the 3×3 matrix is used as the entropy value calculation range, and the entropy value of the middle point is calculated by calculating the probability of occurrence of nine pixels in the 3×3 matrix, which is recorded as W_EN , and the entropy value W_{EN is} used as the image entropy of the calibration image.

在本发明的一个实施例中，可通过如下计算式计算运动区域对应的第二权重：In one embodiment of the present invention, the second weight corresponding to the motion area may be calculated by the following calculation formula:

W′_x,＝(W₁*(I_1ex+I_2ex)+W₂)/I_1ex，W′_x, =(W₁ *(I_1ex +I_2ex )+W₂ )/I_1ex ,

其中，W′_x,为第二权重，I_1ex为曝光图像I₁对应的曝光量，I_2ex为曝光图像I₂对应的曝光量，W₁为曝光图像I₁的对应的第一权重，也即曝光图像I₁对应的校准图像的第一权重，W₂为曝光图像I₂对应的第一权重，也即曝光图像I₂对应的校准图像的第一权重，其中，I₁和I₂为相邻的两帧曝光图像，(x,y)表示参考图像的坐标点。Among them, W′_x, is the second weight, I_1ex is the exposure amount corresponding to the exposure image I₁ , I_2ex is the exposure amount corresponding to the exposure image I₂ , W₁ is the corresponding first weight of the exposure image I₁ , that is, the first weight of the calibration image corresponding to the exposure image I₁ , W₂ is the first weight corresponding to the exposure image I₂ , that is, the first weight of the calibration image corresponding to the exposure image I₂ , wherein I₁ and I₂ are two adjacent frames of exposure images, and (x, y) represents the coordinate point of the reference image.

在本发明的一个实施例中，根据第一权重和第二权重对多帧校准图像进行图像合成，包括：对第一权重和第二权重进行加权平均计算，得到校准图像的权重图；根据权重图得到待合成图像的每个像素点的像素值；根据待合成图像的每个像素点的像素值合成得到目标合成图像。In one embodiment of the present invention, multiple frames of calibration images are synthesized according to a first weight and a second weight, including: performing a weighted average calculation on the first weight and the second weight to obtain a weight map of the calibration image; obtaining a pixel value of each pixel point of the image to be synthesized according to the weight map; and synthesizing a target synthesized image according to the pixel value of each pixel point of the image to be synthesized.

具体而言，即将计算得到的第一权重和第二权重合并到一起，构成最终的权重图，通过加权平均，计算出每个像素点的像素值，得到最终的去鬼影合成图像，即目标合成图像。Specifically, the calculated first weight and second weight are combined to form a final weight map, and the pixel value of each pixel is calculated by weighted averaging to obtain a final de-ghosting synthetic image, that is, a target synthetic image.

在本发明的一个实施例中，对第一权重和第二权重进行加权平均计算，包括：In one embodiment of the present invention, performing weighted average calculation on the first weight and the second weight includes:

其中，I_(x,)为目标合成图像，i为当前待合成图像，n和m分别为当前待合成的尺寸，Value_i(,)为当前待合成图像的像素值，B_i(,)为非运动的值，其取值为0或1，W_i(x,y)为校准图像对应的第一权重，W′_i(,)为运动区域对应的第二权重，(x,y)表示参考图像的像素点坐标。Wherein, I_(x,) is the target synthesized image, i is the current image to be synthesized, n and m are the sizes of the current image to be synthesized, Value_i(,) is the pixel value of the current image to be synthesized, Bi_(,) is the non-motion value, which is 0 or 1,_Wi(x,y) is the first weight corresponding to the calibration image, W′i_(,) is the second weight corresponding to the motion area, and (x,y) represents the pixel coordinates of the reference image.

综上，本发明实施例的图像处理方法，从动态HDR图像出现鬼影的最根本原因入手，针对动态场景中不仅存在移动物体，而且拍摄时图像传感器也会存在移动或抖动的情况，先将不同的曝光图像进行配准和校正到同一曝光水平和同一拍摄角度，通过图像配准和校正，消除了因拍摄时图像传感器移动产生的差异；然后再进行运动物体检测，重新确定运动物体的权重并据此进行图像合成，得到去鬼影的高质量合成图像。其中，利用SURF算法进行快速图像配准，通过计算匹配到的特征点对来进行欧氏距离的计算，据此得到图像偏移量，并算出图像偏移方向，根据图像偏移量和偏移方向对配准图像进行偏移操作，实现图像校正。进一步地，确定校准图像中的运动区域和背景区域，通过重新定义的权重计算方式来计算背景区域权重与运动区域权重，将其融合成为权重图，最后进行去鬼影的高动态图像合成，提高图像质量。In summary, the image processing method of the embodiment of the present invention starts from the most fundamental reason for the appearance of ghost images in dynamic HDR images. In view of the fact that there are not only moving objects in dynamic scenes, but also the image sensor may move or shake during shooting, the different exposure images are first aligned and corrected to the same exposure level and the same shooting angle. Through image alignment and correction, the differences caused by the movement of the image sensor during shooting are eliminated; then the moving object detection is performed, the weight of the moving object is re-determined and the image synthesis is performed accordingly to obtain a high-quality synthetic image without ghost images. Among them, the SURF algorithm is used for fast image alignment, and the Euclidean distance is calculated by calculating the matched feature point pairs, and the image offset is obtained accordingly, and the image offset direction is calculated. The aligned image is offset according to the image offset and the offset direction to achieve image correction. Further, the motion area and background area in the calibration image are determined, and the background area weight and the motion area weight are calculated by a redefined weight calculation method, which are merged into a weight map, and finally the high-dynamic image synthesis without ghost images is performed to improve the image quality.

相较于现有技术，本发明实施例中权重计算方式简单、占用资源较少、计算效率高、具有很好的移植性和适用性；通过长曝光来提高短曝光的信噪比，以减少短曝光图像中的噪声水平，有利于最终合成结果的细节信息显示；对于移动的图像传感器合成高动态图像有很好的成像效果，有效的避免了图像传感器在移动过程中对待合成图像产生的差异，从而提高图像质量；使用对比度、曝光度、图像熵来确定最终融合图像的权重，有效的提高了合成图像的动态范围；通过检测移动物体、确定运动区域、重新计算权重图、依据参考图像等方式，有效的消除了合成图像鬼影的产生，从而提高合成图像的质量。Compared with the prior art, the weight calculation method in the embodiment of the present invention is simple, occupies less resources, has high calculation efficiency, and has good portability and applicability; the signal-to-noise ratio of short exposure is improved by long exposure to reduce the noise level in the short exposure image, which is beneficial to the display of detail information of the final synthesis result; it has a good imaging effect for synthesizing high-dynamic images with a moving image sensor, effectively avoiding the difference in the image sensor's treatment of the synthesized image during movement, thereby improving the image quality; contrast, exposure, and image entropy are used to determine the weight of the final fused image, effectively improving the dynamic range of the synthesized image; by detecting moving objects, determining moving areas, recalculating weight maps, and based on reference images, the generation of ghost images of the synthesized image is effectively eliminated, thereby improving the quality of the synthesized image.

综上，根据本发明实施例的图像处理方法，获取曝光图像，对曝光图像进行图像配准得到配准图像，对配准图像进行图像校正得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像配准和校正，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过检测移动物体、确定运动区域、重新计算权重图来进行图像合成，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。In summary, according to the image processing method of the embodiment of the present invention, an exposure image is acquired, image registration is performed on the exposure image to obtain a registered image, image correction is performed on the registered image to obtain a calibrated image, the motion area of the calibrated image is determined, the first weight corresponding to the calibrated image and the second weight corresponding to the motion area are calculated, and then multiple frames of calibrated images are synthesized according to the first weight and the second weight to obtain a target synthesized image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image registration and correction, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by detecting moving objects, determining motion areas, and recalculating weight maps to perform image synthesis, the generation of ghost images of the synthesized image is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

本发明的进一步实施例还公开了一种图像处理装置。A further embodiment of the present invention also discloses an image processing device.

图5是根据本发明一个实施例的图像处理装置的结构示意图，如图5所示，该图像处理装置100包括：获取模块110、校准模块120、第一计算130、处理模块140、第二计算模块150和合成模块160。Figure 5 is a structural diagram of an image processing device according to an embodiment of the present invention. As shown in Figure 5, the image processing device 100 includes: an acquisition module 110, a calibration module 120, a first calculation module 130, a processing module 140, a second calculation module 150 and a synthesis module 160.

具体的，获取模块110用于获取多帧不同的曝光图像。Specifically, the acquisition module 110 is used to acquire multiple frames of images with different exposures.

校准模块120用于对所述曝光图像进行校准，得到校准图像。The calibration module 120 is used to calibrate the exposure image to obtain a calibrated image.

第一计算模块130用于计算所述校准图像对应的第一权重；The first calculation module 130 is used to calculate the first weight corresponding to the calibration image;

处理模块140用于确定校准图像的运动区域。The processing module 140 is used to determine the motion region of the calibration image.

第二计算模块150用于计算所述校准图像的运动区域第二权重。The second calculation module 150 is used to calculate the second weight of the motion area of the calibration image.

合成模块160用于根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。The synthesis module 160 is used to perform image synthesis on multiple frames of calibration images according to the first weight and the second weight to obtain a target synthesized image.

在本发明的一个实施例中，校准模块120用于对曝光图像进行图像配准，得到配准图像；对配准图像进行图像校正，得到校准图像。In one embodiment of the present invention, the calibration module 120 is used to perform image registration on the exposure image to obtain a registered image; and perform image correction on the registered image to obtain a calibrated image.

在本发明的一个实施例中，校准模块120用于按照预设的曝光曲线将曝光图像的曝光量置于预设曝光水平，得到调整后的曝光图像，调整后的曝光图像包括参考图像和预配准图像；获取预配准图像和参考图像之间的特征点对，以得到配准图像。In one embodiment of the present invention, the calibration module 120 is used to place the exposure amount of the exposure image at a preset exposure level according to a preset exposure curve to obtain an adjusted exposure image, wherein the adjusted exposure image includes a reference image and a pre-registered image; and obtain feature point pairs between the pre-registered image and the reference image to obtain a registered image.

在本发明的一个实施例中，校准模块120用于对预配准图像和参考图像进行特征点检测，对应得到预配准图像和参考图像的特征点信息；参考图像包括预设原点，从预设原点开始，以预设匹配范围对预配准图像和参考图像进行特征点匹配，得到特征点对。In one embodiment of the present invention, the calibration module 120 is used to perform feature point detection on the pre-registered image and the reference image to obtain feature point information of the pre-registered image and the reference image accordingly; the reference image includes a preset origin, and starting from the preset origin, feature point matching is performed on the pre-registered image and the reference image within a preset matching range to obtain a feature point pair.

在本发明的一个实施例中，校准模块120用于计算特征点对的距离；统计计算得到的特征点对的距离，将满足预设条件的特征点对的距离作为图像偏移量；计算配准图像所需的偏移方向；根据图像偏移量和偏移方向对配准图像进行偏移操作，以对配准图像进行校正，得到校准图像。In one embodiment of the present invention, the calibration module 120 is used to calculate the distance between feature point pairs; statistically calculate the distance between feature point pairs, and use the distance between feature point pairs that meets preset conditions as image offsets; calculate the offset direction required for registering the image; and perform an offset operation on the registered image according to the image offset and the offset direction to correct the registered image and obtain a calibrated image.

在本发明的一个实施例中，校准模块120还用于若在预设匹配范围进行内未匹配到特征点对，则按照预设步长扩大匹配范围，以重新进行特征点匹配，直至得到特征点对。In one embodiment of the present invention, the calibration module 120 is further configured to expand the matching range according to a preset step length to re-match the feature points if no feature point pair is matched within the preset matching range, until a feature point pair is obtained.

在本发明的一个实施例中，校准模块120用于以参考图像为基准，计算出参考图像与配准图像的相匹配的特征点之间的欧氏距离，将计算的欧氏距离作为特征点对的距离。In one embodiment of the present invention, the calibration module 120 is used to calculate the Euclidean distance between the matching feature points of the reference image and the registration image based on the reference image, and use the calculated Euclidean distance as the distance of the feature point pair.

在本发明的一个实施例中，校准模块120用于出现次数最多的特征点对的距离。In one embodiment of the present invention, the calibration module 120 is used to calibrate the distance of the feature point pair that appears most frequently.

在本发明的一个实施例中，校准模块120用于以参考图像的像素点坐标为原点计算配准图像中对应像素点坐标所在的象限位置；根据象限位置确定配准图像所需的偏移方向。In one embodiment of the present invention, the calibration module 120 is used to calculate the quadrant position of the corresponding pixel coordinates in the registration image using the pixel coordinates of the reference image as the origin; and determine the offset direction required for the registration image according to the quadrant position.

在本发明的一个实施例中，校准模块120还用于对偏移后的配准图像的边界进行补零处理，以使其与对应的曝光图像的尺寸相同，以得到校准图像。In one embodiment of the present invention, the calibration module 120 is further configured to perform zero padding on the boundary of the offset registration image so as to make the boundary have the same size as the corresponding exposure image, so as to obtain a calibration image.

在本发明的一个实施例中，处理模块140用于获取校准图像和参考图像的二值图像；将校准图像和参考图像的二值图像相减求绝对值，得到校准图像的运动区域。In one embodiment of the present invention, the processing module 140 is used to obtain binary images of the calibration image and the reference image; and to subtract the binary images of the calibration image and the reference image to obtain an absolute value, thereby obtaining a motion region of the calibration image.

在本发明的一个实施例中，第一计算模块130用于确定校准图像的非运动区域。In one embodiment of the present invention, the first calculation module 130 is used to determine a non-motion area of the calibration image.

在本发明的一个实施例中，处理模块140用于将校准图像的运动区域取反，得到非运动区域。In one embodiment of the present invention, the processing module 140 is used to invert the motion region of the calibration image to obtain the non-motion region.

在本发明的一个实施例中，处理模块140还用于在RGB颜色空间下，将预设的长曝光图像的信噪比映射到短曝光图像上，以提高短曝光图像的信噪比。In one embodiment of the present invention, the processing module 140 is further configured to map the preset signal-to-noise ratio of the long exposure image to the short exposure image in the RGB color space to improve the signal-to-noise ratio of the short exposure image.

在本发明的一个实施例中，第一计算模块130用于计算校准图像的对比度、曝光度和图像熵；通过对比度、曝光度、图像熵计算校准图像对应的权重，以得到第一权重。In one embodiment of the present invention, the first calculation module 130 is used to calculate the contrast, exposure and image entropy of the calibration image; and calculate the weight corresponding to the calibration image through the contrast, exposure and image entropy to obtain the first weight.

在本发明的一个实施例中，第一计算模块130用于对校准图像对应的曝光图像进行拉普拉斯滤波后，对得到的滤波结果求取绝对值，以得到校准图像的对比度。In one embodiment of the present invention, the first calculation module 130 is used to perform Laplace filtering on the exposure image corresponding to the calibration image, and then obtain an absolute value of the obtained filtering result to obtain the contrast of the calibration image.

在本发明的一个实施例中，第一计算模块130用于采用三角权重函数计算校准图像的曝光度。In one embodiment of the present invention, the first calculation module 130 is used to calculate the exposure of the calibration image using a triangular weight function.

在本发明的一个实施例中，第一计算模块130用于用预设矩阵作为熵值计算的范围，通过计算预设矩阵内所有像素点出现的概率来计算中间点的熵值，将该熵值作为校准图像的图像熵。In one embodiment of the present invention, the first calculation module 130 is used to use a preset matrix as the range of entropy value calculation, calculate the entropy value of the middle point by calculating the probability of occurrence of all pixels in the preset matrix, and use the entropy value as the image entropy of the calibration image.

在本发明的一个实施例中，第一计算模块130用于将对比度、曝光度、图像熵相乘并进行归一化，得到校准图像的权重。In one embodiment of the present invention, the first calculation module 130 is used to multiply the contrast, exposure, and image entropy and normalize them to obtain the weight of the calibration image.

在本发明的一个实施例中，第二计算模块150用于W′_x,＝(W₁*(I_1ex+I_2ex)+W₂)/I_1ex，其中，W′_x,为第二权重，I_1ex为曝光图像I₁对应的曝光量，I_2ex为曝光图像I₂对应的曝光量，W₁为曝光图像I₁的对应的第一权重，W₂为曝光图像I₂对应的第一权重，其中，I₁和I₂为相邻的两帧曝光图像，(x,y)表示参考图像的坐标点。In one embodiment of the present invention, the second calculation module 150 is used for W′_x, =(W₁ *(I_1ex +I_2ex )+W₂ )/I_1ex , wherein W′_x, is the second weight, I_1ex is the exposure amount corresponding to the exposure image I₁ , I_2ex is the exposure amount corresponding to the exposure image I₂ , W₁ is the corresponding first weight of the exposure image I₁ , W₂ is the corresponding first weight of the exposure image I₂ , wherein I₁ and I₂ are two adjacent frames of exposure images, and (x, y) represents the coordinate point of the reference image.

在本发明的一个实施例中，合成模块160用于对第一权重和第二权重进行加权平均计算，得到校准图像的权重图；根据权重图得到待合成图像的每个像素点的像素值；根据待合成图像的每个像素点的像素值合成得到目标合成图像。In one embodiment of the present invention, the synthesis module 160 is used to perform a weighted average calculation on the first weight and the second weight to obtain a weight map of the calibration image; obtain the pixel value of each pixel point of the image to be synthesized according to the weight map; and synthesize the target synthesized image according to the pixel value of each pixel point of the image to be synthesized.

在本发明的一个实施例中，合成模块160用于其中，I_(x,)为目标合成图像，i为当前待合成图像，n和m分别为当前待合成图像的尺寸，Value_i(,)为当前待合成图像的像素值，B_i(,)为非运动区域的值，W_i(x,y)为校准图像对应的第一权重，W′_i(,)为运动区域对应的第二权重In one embodiment of the present invention, the synthesis module 160 is used to Where I_(x,) is the target synthesized image, i is the current image to be synthesized, n and m are the sizes of the current image to be synthesized, Value_i(,) is the pixel value of the current image to be synthesized, Bi_(,) is the value of the non-motion area, Wi_(x,y) is the first weight corresponding to the calibration image, and W′i_(,) is the second weight corresponding to the motion area.

需要说明的是，在进行图像处理时，该图像处理装置100的具体实现方式与本发明上述任意一个实施例的图像处理方法的具体实现方式类似，因而关于该图像处理装置100的详细示例性描述，可参见前述关于图像处理方法的相关描述部分，为减少冗余，此处不再重复赘述。It should be noted that, when performing image processing, the specific implementation method of the image processing device 100 is similar to the specific implementation method of the image processing method of any one of the above-mentioned embodiments of the present invention. Therefore, for a detailed exemplary description of the image processing device 100, please refer to the above-mentioned relevant description part of the image processing method. In order to reduce redundancy, it will not be repeated here.

由此，根据本发明实施例的图像处理装置100，获取曝光图像，对曝光图像进行图像配准得到配准图像，对配准图像进行图像校正得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像配准和校正，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过检测移动物体、确定运动区域、重新计算权重图来进行图像合成，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。Therefore, according to the image processing device 100 of the embodiment of the present invention, an exposure image is acquired, image registration is performed on the exposure image to obtain a registered image, image correction is performed on the registered image to obtain a calibration image, the motion area of the calibration image is determined, the first weight corresponding to the calibration image and the second weight corresponding to the motion area are calculated, and then multiple frames of calibration images are synthesized according to the first weight and the second weight to obtain a target synthesized image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image registration and correction, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by detecting moving objects, determining motion areas, and recalculating weight maps to perform image synthesis, the generation of ghost images of the synthesized image is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

本发明的进一步实施例还公开了一种电子设备。A further embodiment of the present invention also discloses an electronic device.

在本发明的一个实施例中，该电子设备包括上述任意一个实施例所描述的图像处理装置100。In one embodiment of the present invention, the electronic device includes the image processing device 100 described in any one of the above embodiments.

在本发明的另一个实施例中，该电子设备包括：处理器、存储器、存储在所述存储器上并可在所述处理器上运行的图像处理程序，所述图像处理程序被所述处理器执行时实现如本发明上述任意一个实施例所描述的图像处理方法。In another embodiment of the present invention, the electronic device includes: a processor, a memory, and an image processing program stored in the memory and executable on the processor, wherein when the image processing program is executed by the processor, the image processing method described in any one of the above embodiments of the present invention is implemented.

在具体实施例中，该电子设备例如为图像传感设备，图像传感设备例如包括CMOS图像传感器。In a specific embodiment, the electronic device is, for example, an image sensing device, and the image sensing device includes, for example, a CMOS image sensor.

由此，在进行图像处理时，该电子设备的具体实现方式可参见前述关于图像处理方法或图像处理装置100的相关描述部分，为减少冗余，此处不再重复赘述。Therefore, when performing image processing, the specific implementation of the electronic device can refer to the aforementioned description of the image processing method or the image processing device 100, and will not be repeated here to reduce redundancy.

根据本发明实施例的电子设备，获取曝光图像，对曝光图像进行图像配准得到配准图像，对配准图像进行图像校正得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像配准和校正，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过检测移动物体、确定运动区域、重新计算权重图来进行图像合成，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。According to the electronic device of the embodiment of the present invention, an exposure image is acquired, image registration is performed on the exposure image to obtain a registered image, image correction is performed on the registered image to obtain a calibration image, the motion area of the calibration image is determined, a first weight corresponding to the calibration image and a second weight corresponding to the motion area are calculated, and then multiple frames of calibration images are synthesized according to the first weight and the second weight to obtain a target synthesized image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image registration and correction, the scene positions of exposure images with different exposure times are matched, the difference caused by the movement of the image sensor during shooting is eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, which is conducive to improving the image quality; by detecting moving objects, determining motion areas, and recalculating weight maps to perform image synthesis, the generation of ghost images of the synthesized image is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

本发明的进一步实施例还公开了一种计算机可读存储介质，该计算机可读存储介质上存储有图像处理程序，所述图像处理程序被处理器执行时实现如本发明上述任意一个实施例所描述的图像处理方法，具体关于图像处理方法的执行过程的详细描述，请参见前文相关部分，此处不再冗余赘述。A further embodiment of the present invention also discloses a computer-readable storage medium, on which an image processing program is stored. When the image processing program is executed by a processor, the image processing method described in any one of the above embodiments of the present invention is implemented. For a detailed description of the execution process of the image processing method, please refer to the relevant part of the previous text, which will not be repeated here.

由此，根据本发明实施例的计算机可读存储介质，其上存储的图像处理程序被处理器执行时，可获取曝光图像，对曝光图像进行图像配准得到配准图像，对配准图像进行图像校正得到校准图像，确定校准图像的运动区域，计算校准图像对应的第一权重和运动区域对应的第二权重，进而根据第一权重和第二权重对多帧校准图像进行图像合成，得到目标合成图像。由此，本发明考虑到拍摄时图像传感器会发生移动的情况，通过图像配准和校正，使不同曝光时间的曝光图像的场景位置相匹配，消除了因拍摄时图像传感器移动产生的差异，避免了像素级融合时产生的模糊现象，从而利于提高图像质量；通过检测移动物体、确定运动区域、重新计算权重图来进行图像合成，有效消除了合成图像鬼影的产生，从而进一步提高了图像质量。且本发明实现过程中算法简单，计算量小，易于实现，适用性强。Therefore, according to the computer-readable storage medium of the embodiment of the present invention, when the image processing program stored thereon is executed by the processor, an exposure image can be obtained, image registration can be performed on the exposure image to obtain a registered image, image correction can be performed on the registered image to obtain a calibration image, the motion area of the calibration image can be determined, the first weight corresponding to the calibration image and the second weight corresponding to the motion area can be calculated, and then multiple frames of calibration images can be synthesized according to the first weight and the second weight to obtain a target synthetic image. Therefore, the present invention takes into account the situation that the image sensor may move during shooting, and through image registration and correction, the scene positions of exposure images with different exposure times are matched, the differences caused by the movement of the image sensor during shooting are eliminated, and the blurring phenomenon caused by pixel-level fusion is avoided, thereby helping to improve the image quality; by detecting moving objects, determining motion areas, and recalculating weight maps to perform image synthesis, the generation of synthetic image ghosts is effectively eliminated, thereby further improving the image quality. In addition, the algorithm is simple in the implementation process of the present invention, the amount of calculation is small, it is easy to implement, and it has strong applicability.

在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示意性实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或示例。In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example.

尽管已经示出和描述了本发明的实施例，本领域的普通技术人员可以理解：在不脱离本发明的原理和宗旨的情况下可以对这些实施例进行多种变化、修改、替换和变型，本发明的范围由权利要求及其等同物限定。Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims (25)

Translated fromChinese

1.一种图像处理方法，其特征在于，包括以下步骤：1. An image processing method, characterized in that it comprises the following steps:获取多帧不同的曝光图像；Acquire multiple frames of images with different exposures;对所述曝光图像进行校准，得到校准图像；Calibrate the exposure image to obtain a calibrated image;计算所述校准图像对应的第一权重；Calculating a first weight corresponding to the calibration image;确定所述校准图像的运动区域；determining a motion region of the calibration image;计算所述校准图像的运动区域的第二权重；calculating a second weight of a motion region of the calibration image;根据所述第一权重和所述第二权重对多帧所述校准图像进行图像合成，得到目标合成图像。Image synthesis is performed on multiple frames of the calibration image according to the first weight and the second weight to obtain a target synthesized image.2.根据权利要求1所述的图像处理方法，其特征在于，对所述曝光图像进行校准，包括：2. The image processing method according to claim 1, characterized in that calibrating the exposure image comprises:对所述曝光图像进行图像配准，得到配准图像；Performing image registration on the exposure image to obtain a registered image;对所述配准图像进行图像校正，得到所述校准图像。Performing image correction on the registered image to obtain the calibrated image.3.根据权利要求2所述的图像处理方法，其特征在于，对所述曝光图像进行图像配准，包括：3. The image processing method according to claim 2, characterized in that performing image registration on the exposure image comprises:按照预设的曝光曲线将所述曝光图像的曝光量置于预设曝光水平，得到调整后的曝光图像，所述调整后的曝光图像包括参考图像和预配准图像；Setting the exposure amount of the exposure image to a preset exposure level according to a preset exposure curve to obtain an adjusted exposure image, wherein the adjusted exposure image includes a reference image and a pre-registered image;获取所述预配准图像和参考图像之间的特征点对，以得到所述配准图像。Acquire feature point pairs between the pre-registered image and a reference image to obtain the registered image.4.根据权利要求3所述的图像处理方法，其特征在于，获取所述预配准图像和参考图像之间的特征点对，包括：4. The image processing method according to claim 3, characterized in that obtaining feature point pairs between the pre-registered image and the reference image comprises:对所述预配准图像和所述参考图像进行特征点检测，对应得到所述预配准图像和所述参考图像的特征点信息；Performing feature point detection on the pre-registered image and the reference image to obtain feature point information of the pre-registered image and the reference image accordingly;所述参考图像包括预设原点，从预设原点开始，以预设匹配范围对所述预配准图像和所述参考图像进行特征点匹配，得到所述特征点对。The reference image includes a preset origin. Starting from the preset origin, feature point matching is performed on the pre-registered image and the reference image within a preset matching range to obtain the feature point pair.5.根据权利要求3所述的图像处理方法，其特征在于，对所述配准图像进行校正，包括：5. The image processing method according to claim 3, characterized in that the correction of the registered image comprises:计算所述特征点对的距离；Calculating the distance between the feature point pairs;统计计算得到的特征点对的距离，将满足预设条件的特征点对的距离作为图像偏移量；The distances of the feature point pairs obtained by statistical calculation are used as the image offsets.计算所述配准图像所需的偏移方向；Calculating the offset direction required for the registered image;根据所述图像偏移量和所述偏移方向对所述配准图像进行偏移操作，以对所述配准图像进行校正，得到所述校准图像。An offset operation is performed on the registered image according to the image offset amount and the offset direction to correct the registered image to obtain the calibrated image.6.根据权利要求4所述的图像处理方法，其特征在于，还包括：6. The image processing method according to claim 4, further comprising:若在所述预设匹配范围进行内未匹配到所述特征点对，则按照预设步长扩大匹配范围，以重新进行特征点匹配，直至得到所述特征点对。If the feature point pair is not matched within the preset matching range, the matching range is expanded according to a preset step size to re-match the feature points until the feature point pair is obtained.7.根据权利要求5所述的图像处理方法，其特征在于，所述计算所述特征点对的距离，包括：7. The image processing method according to claim 5, wherein calculating the distance between the feature point pairs comprises:以所述参考图像为基准，计算出所述参考图像与所述配准图像的相匹配的特征点之间的欧氏距离，将计算的所述欧氏距离作为所述特征点对的距离。Taking the reference image as a reference, the Euclidean distance between the matching feature points of the reference image and the registration image is calculated, and the calculated Euclidean distance is used as the distance of the feature point pair.8.根据权利要求5所述的图像处理方法，其特征在于，所述满足预设条件的特征点对的距离包括：出现次数最多的特征点对的距离。8 . The image processing method according to claim 5 , wherein the distance of the feature point pair that meets the preset condition comprises: the distance of the feature point pair that appears the most times.9.根据权利要求5所述的图像处理方法，其特征在于，所述计算所述配准图像所需的偏移方向，包括：9. The image processing method according to claim 5, characterized in that the calculation of the offset direction required for the registration image comprises:以所述参考图像的像素点坐标为原点计算所述配准图像中对应像素点坐标所在的象限位置；Calculate the quadrant position of the corresponding pixel point coordinates in the registration image using the pixel point coordinates of the reference image as the origin;根据所述象限位置确定所述配准图像所需的所述偏移方向。The shift direction required for registering the images is determined according to the quadrant position.10.根据权利要求5所述的图像处理方法，其特征在于，在根据所述图像偏移量和所述偏移方向对所述配准图像进行偏移操作之后，还包括：10. The image processing method according to claim 5, characterized in that after performing an offset operation on the registered image according to the image offset amount and the offset direction, it further comprises:对偏移后的配准图像的边界进行补零处理，以使其与对应的曝光图像的尺寸相同，以得到所述校准图像。The boundary of the offset registration image is zero-filled to make it the same size as the corresponding exposure image, so as to obtain the calibration image.11.根据权利要求4所述的图像处理方法，其特征在于，确定所述校准图像的运动区域，包括：11. The image processing method according to claim 4, wherein determining the motion area of the calibration image comprises:获取所述校准图像和所述参考图像的二值图像；Acquire binary images of the calibration image and the reference image;将所述校准图像和参考图像的二值图像相减求绝对值，得到所述校准图像的运动区域。The binary images of the calibration image and the reference image are subtracted to obtain an absolute value, thereby obtaining a motion region of the calibration image.12.根据权利要求1所述的图像处理方法，其特征在于，计算所述校准图像对应的第一权重之前，包括：12. The image processing method according to claim 1, characterized in that before calculating the first weight corresponding to the calibration image, it comprises:确定所述校准图像的非运动区域。A non-motion region of the calibration image is determined.13.根据权利要求12所述的图像处理方法，其特征在于，确定所述校准图像的非运动区域，包括：13. The image processing method according to claim 12, wherein determining the non-moving area of the calibration image comprises:将所述校准图像的运动区域取反，得到所述非运动区域。The motion region of the calibration image is inverted to obtain the non-motion region.14.根据权利要求12所述的图像处理方法，其特征在于，所述非运动区域的图像包括长曝光图像和短曝光图像，所述确定所述校准图像的非运动区域之后，还包括：14. The image processing method according to claim 12, wherein the image of the non-motion area comprises a long-exposure image and a short-exposure image, and after determining the non-motion area of the calibration image, further comprising:在RGB颜色空间下，将预设的长曝光图像的信噪比映射到所述短曝光图像上，以提高所述短曝光图像的信噪比。In the RGB color space, the preset signal-to-noise ratio of the long exposure image is mapped to the short exposure image to improve the signal-to-noise ratio of the short exposure image.15.根据权利要求1所述的图像处理方法，其特征在于，计算所述校准图像对应的第一权重，包括：15. The image processing method according to claim 1, wherein calculating the first weight corresponding to the calibration image comprises:计算所述校准图像的对比度、曝光度和图像熵；Calculating contrast, exposure and image entropy of the calibration image;通过所述对比度、曝光度、图像熵计算所述校准图像对应的权重，以得到所述第一权重。The weight corresponding to the calibration image is calculated by using the contrast, exposure, and image entropy to obtain the first weight.16.根据权利要求15所述的图像处理方法，其特征在于，所述计算所述校准图像的对比度，包括：16. The image processing method according to claim 15, wherein calculating the contrast of the calibration image comprises:对所述校准图像对应的曝光图像进行拉普拉斯滤波后，对得到的滤波结果求取绝对值，以得到所述校准图像的对比度。After Laplace filtering is performed on the exposure image corresponding to the calibration image, an absolute value of the obtained filtering result is obtained to obtain the contrast of the calibration image.17.根据权利要求16所述的图像处理方法，其特征在于，所述计算所述校准图像的曝光度包括：采用三角权重函数计算所述校准图像的曝光度。17 . The image processing method according to claim 16 , wherein calculating the exposure of the calibration image comprises: calculating the exposure of the calibration image using a triangular weight function.18.根据权利要求17所述的图像处理方法，其特征在于，所述计算所述校准图像的图像熵，包括：18. The image processing method according to claim 17, wherein calculating the image entropy of the calibration image comprises:用预设矩阵作为熵值计算的范围，通过计算所述预设矩阵内所有像素点出现的概率来计算中间点的熵值，将该熵值作为所述校准图像的图像熵。A preset matrix is used as the range for entropy value calculation, and the entropy value of the middle point is calculated by calculating the probability of occurrence of all pixel points in the preset matrix, and the entropy value is used as the image entropy of the calibration image.19.根据权利要求18所述的图像处理方法，其特征在于，通过所述对比度、曝光度、图像熵计算所述校准图像对应的权重，包括：19. The image processing method according to claim 18, characterized in that the weight corresponding to the calibration image is calculated by using the contrast, exposure, and image entropy, comprising:将所述对比度、曝光度、图像熵相乘并进行归一化，得到所述校准图像的权重。The contrast, exposure, and image entropy are multiplied and normalized to obtain the weight of the calibration image.20.根据权利要求3所述的图像处理方法，其特征在于，计算所述校准图像的运动区域对应的第二权重，包括：20. The image processing method according to claim 3, wherein calculating the second weight corresponding to the motion area of the calibration image comprises:w'_x,y＝(W₁*(I_lex+I_2ex)+W₂)/I_lex,w'_x,y =(W₁ *(I_lex +I_2ex )+W₂ )/I_lex ,其中，W′_x,y为第二权重，I_1ex为曝光图像I₁对应的曝光量，I_2ex为曝光图像I₂对应的曝光量，W₁为曝光图像I₁的对应的第一权重，W₂为曝光图像I₂对应的第一权重，其中，I₁和I₂为相邻的两帧曝光图像，(x,y)表示所述参考图像的坐标点。Among them, W′_x,y is the second weight, I_1ex is the exposure amount corresponding to the exposure image I₁ , I_2ex is the exposure amount corresponding to the exposure image I₂ , W₁ is the first weight corresponding to the exposure image I₁ , W₂ is the first weight corresponding to the exposure image I₂ , wherein I₁ and I₂ are two adjacent frames of exposure images, and (x, y) represents the coordinate point of the reference image.21.根据权利要求20所述的图像处理方法，其特征在于，根据所述第一权重和第二权重对多帧所述校准图像进行图像合成，包括：21. The image processing method according to claim 20, characterized in that the step of synthesizing a plurality of frames of the calibration images according to the first weight and the second weight comprises:对所述第一权重和所述第二权重进行加权平均计算，得到所述校准图像的权重图；Performing weighted average calculation on the first weight and the second weight to obtain a weight map of the calibration image;根据所述权重图得到待合成图像的每个像素点的像素值；Obtaining a pixel value of each pixel of the image to be synthesized according to the weight map;根据所述待合成图像的每个像素点的像素值合成得到所述目标合成图像。The target synthesized image is obtained by synthesizing according to the pixel value of each pixel point of the image to be synthesized.22.根据权利要求21所述的图像处理方法，其特征在于，对所述第一权重和第二权重进行加权平均计算，包括：22. The image processing method according to claim 21, characterized in that performing weighted average calculation on the first weight and the second weight comprises:其中，I_(x,)为所述目标合成图像，i为当前待合成图像，n和m分别为当前待合成图像的尺寸，Value_i(,)为当前待合成图像的像素值，B_i(,)为非运动区域的值，W_i(x,y)为校准图像对应的第一权重，W′_i(,)为运动区域对应的第二权重。Wherein, I_(x,) is the target synthesized image, i is the current image to be synthesized, n and m are the sizes of the current image to be synthesized, Value_i(,) is the pixel value of the current image to be synthesized, Bi_(,) is the value of the non-motion area, Wi_(x,y) is the first weight corresponding to the calibration image, and W′i_{(,) is} the second weight corresponding to the motion area.23.一种图像处理装置，其特征在于，包括：23. An image processing device, comprising:获取模块，用于获取多帧不同的曝光图像；An acquisition module, used for acquiring multiple frames of images with different exposures;校准模块，用于对所述曝光图像进行校准，得到校准图像；A calibration module, used to calibrate the exposure image to obtain a calibration image;第一计算模块，用于计算所述校准图像对应的第一权重；A first calculation module, used to calculate a first weight corresponding to the calibration image;处理模块，用于确定所述校准图像的运动区域；a processing module, configured to determine a motion region of the calibration image;第二计算模块，用于计算所述校准图像的运动区域的第二权重；A second calculation module, used for calculating a second weight of the motion area of the calibration image;合成模块，用于根据所述第一权重和第二权重对多帧所述校准图像进行图像合成，得到目标合成图像。A synthesis module is used to perform image synthesis on multiple frames of the calibration image according to the first weight and the second weight to obtain a target synthesized image.24.一种电子设备，其特征在于，包括：24. An electronic device, comprising:如权利要求23所述的图像处理装置，或者，The image processing apparatus according to claim 23, or处理器、存储器，以及存储在所述存储器上并可在所述处理器上运行的图像处理程序，所述图像处理程序被所述处理器执行时实现如权利要求1-22任一项所述的图像处理方法。A processor, a memory, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the image processing method according to any one of claims 1 to 22.25.一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有图像处理程序，所述图像处理程序被处理器执行时实现如权利要求1-22任一项所述的图像处理方法。25. A computer-readable storage medium, characterized in that an image processing program is stored on the computer-readable storage medium, and when the image processing program is executed by a processor, the image processing method according to any one of claims 1 to 22 is implemented.