技术领域Technical Field
本发明涉及计算机技术领域，具体涉及一种图像处理方法、电子设备和计算机可读存储介质。The present invention relates to the field of computer technology, and in particular to an image processing method, an electronic device and a computer-readable storage medium.
背景技术Background technique
神经辐射场(neural radiance fields，NeRF)作为计算机视觉方向快速发展的一种深度学习模型，在新视角合成任务(novel view synthesis)中得到了广泛应用。例如，可以向训练好的NeRF模型输入拍摄图像的各个像素点的空间位置坐标、用于表征目标视角的方向参数。进而，NeRF模型预测各个空间位置坐标在目标视角下对应的像素点的像素颜色，进而生成目标视角下的拍摄图像。Neural radiance fields (NeRF), a fast-growing deep learning model in computer vision, has been widely used in novel view synthesis tasks. For example, the spatial position coordinates of each pixel of a captured image and the directional parameters used to characterize the target view can be input into a trained NeRF model. Then, the NeRF model predicts the pixel color of the pixel corresponding to each spatial position coordinate under the target view, and then generates the captured image under the target view.
可以理解，NeRF模型基于空间位置坐标及方向参数预测该空间位置坐标在目标视角下的拍摄图像中的像素颜色，即，NeRF模型对于各个像素点的像素颜色的预测是相互独立的，并未考虑一个像素点与其相邻像素点之间的空间关系。可以理解，该空间关系可以包括像素点之间的边缘细节、纹理细节、高光细节等。因此，可能因NeRF模型未考虑像素点之间的空间关系而导致生成的目标视角下的拍摄图像质感较差。例如，目标视角下的拍摄图像可能缺乏高光细节等。It can be understood that the NeRF model predicts the pixel color of the spatial position coordinates in the captured image at the target perspective based on the spatial position coordinates and direction parameters, that is, the NeRF model predicts the pixel color of each pixel independently, and does not consider the spatial relationship between a pixel and its adjacent pixels. It can be understood that the spatial relationship may include edge details, texture details, highlight details, etc. between pixels. Therefore, the generated captured image at the target perspective may have poor texture because the NeRF model does not consider the spatial relationship between pixels. For example, the captured image at the target perspective may lack highlight details, etc.
具体地，参见图1示出的手机100基于NeRF模型生成目标视角下的拍摄图像的效果示意图，手机100基于用户的拍摄操作生成了拍摄图像100a，手机100根据用户输入的目标视角基于NeRF模型生成了拍摄图像100a在目标视角下的拍摄图像100b。由于NeRF模型未考虑像素点之间的空间关系，导致目标视角下的拍摄图像100b存在缺乏高光细节等缺陷。Specifically, referring to FIG1, a schematic diagram of the effect of the mobile phone 100 generating a captured image under a target viewing angle based on the NeRF model, the mobile phone 100 generates a captured image 100a based on the user's shooting operation, and the mobile phone 100 generates a captured image 100b under the target viewing angle based on the NeRF model according to the target viewing angle input by the user. Since the NeRF model does not consider the spatial relationship between pixels, the captured image 100b under the target viewing angle has defects such as lack of highlight details.
发明内容Summary of the invention
本申请提供了一种图像处理方法、电子设备和计算机可读存储介质。可以提高NeRF模型生成的目标视角下的拍摄图像的高光细节，提升图像质感。The present application provides an image processing method, an electronic device, and a computer-readable storage medium, which can improve the highlight details of a captured image at a target viewing angle generated by a NeRF model and enhance the image texture.
第一方面，本申请提供了一种图像处理方法，应用于电子设备，该方法包括：获取第一图像；向第一图像处理模型输入第一图像及第一偏振参数，得到第二图像，其中，第二图像包括对应第一偏振参数的第一高光信息。In a first aspect, the present application provides an image processing method for application to an electronic device, the method comprising: acquiring a first image; inputting the first image and a first polarization parameter into a first image processing model to obtain a second image, wherein the second image includes first highlight information corresponding to the first polarization parameter.
在此，第一图像可以是下文中的拍摄图像，第一偏振参数可以是下文中的目标偏振参数，第二图像可以是下文中的目标视角下的拍摄图像、实施例1中的具有高光细节的目标视角下的低分辨率拍摄图像、实施例2中的具有高光细节的原始视角下的低分辨率拍摄图像。Here, the first image may be the captured image described below, the first polarization parameter may be the target polarization parameter described below, and the second image may be the captured image at the target perspective described below, the low-resolution captured image at the target perspective with highlight details in Example 1, or the low-resolution captured image at the original perspective with highlight details in Example 2.
第一高光信息可以是下文中目标偏振参数对应的高光效果。基于该方式，可以为目标视角下的拍摄图像增加高光细节，提升图像质感。The first highlight information may be a highlight effect corresponding to the target polarization parameter described below. Based on this approach, highlight details may be added to the image captured under the target viewing angle to improve the image texture.
在上述第一方面的一种可能实现中，第一图像处理模型包括NeRF模型。In a possible implementation of the first aspect above, the first image processing model includes a NeRF model.
在此，NeRF模型的输入数据包括第一图像以及第一偏振参数，进而，本申请提供的图像处理方法可以通过NeRF模型为第一图像增加该第一偏振参数对应的高光细节，以提升图像质感。Here, the input data of the NeRF model includes a first image and a first polarization parameter. Furthermore, the image processing method provided in the present application can add highlight details corresponding to the first polarization parameter to the first image through the NeRF model to enhance the image texture.
在上述第一方面的一种可能实现中，第一图像的方向参数包括：表征第一图像的拍摄视角的第一方向参数，或者，第二方向参数，其中，第二方向参数表征的拍摄视角与第一图像的拍摄视角不同。In a possible implementation of the first aspect described above, the directional parameter of the first image includes: a first directional parameter representing a shooting angle of view of the first image, or a second directional parameter, wherein the shooting angle represented by the second directional parameter is different from the shooting angle of view of the first image.
在此，NeRF模型的输入数据还可以包括方向参数。例如，当输入的方向参数为表征第一图像的拍摄视角的第一方向参数时，例如下文实施例2所示的应用场景，基于本申请提供的NeRF模型生成的第二图像可以是与第一图像的拍摄视角相同的具有高光细节的图像。例如，实施例2中的具有高光细节的原始视角下的低分辨率拍摄图像。Here, the input data of the NeRF model may also include a direction parameter. For example, when the input direction parameter is a first direction parameter representing the shooting angle of the first image, such as the application scenario shown in Example 2 below, the second image generated based on the NeRF model provided by the present application may be an image with highlight details at the same shooting angle as the first image. For example, a low-resolution image captured at the original angle of view with highlight details in Example 2.
同样地，当输入的方向参数表征的拍摄视角与第一图像的拍摄视角不同时，例如下文实施例1所示的应用场景，即，向NeRF模型输入的方向参数为表征目标视角的前述第二方向参数（例如，下文中的目标方向参数），基于本申请提供的NeRF模型生成的第二图像可以是与第一图像的拍摄视角不同的具有高光细节的图像。例如，基于下述图2所述过程生成的具有高光细节的目标视角下的拍摄图像。Similarly, when the shooting angle represented by the input directional parameter is different from the shooting angle of the first image, such as the application scenario shown in Example 1 below, that is, the directional parameter input to the NeRF model is the aforementioned second directional parameter representing the target angle (for example, the target directional parameter hereinafter), the second image generated based on the NeRF model provided by the present application can be an image with highlight details at a shooting angle different from that of the first image. For example, an image shot at a target angle with highlight details generated based on the process described in FIG. 2 below.
此外，第一图像模型还可以是距离符号场表达网络（signed distance functionnetworks,SDF-Net）。具体地，对于SDF-Net模型，同样可以将偏振参数拼接至SDF-Net模型的输入数据中，使得SDF-Net模型同样可基于输入的第一偏振参数为生成的第二图像增加该第一偏振参数对应的高光效果。在此，SDF-Net模型的训练及应用过程与下文接收的NeRF模型的训练及应用过程实质相同，所能实现的效果相同，在此，不赘述SDF-Net模型的训练及应用过程。In addition, the first image model can also be a signed distance function network (SDF-Net). Specifically, for the SDF-Net model, the polarization parameter can also be spliced into the input data of the SDF-Net model, so that the SDF-Net model can also add a highlight effect corresponding to the first polarization parameter to the generated second image based on the input first polarization parameter. Here, the training and application process of the SDF-Net model is essentially the same as the training and application process of the NeRF model received below, and the effects that can be achieved are the same. Here, the training and application process of the SDF-Net model will not be repeated.
在上述第一方面的一种可能实现中，第一图像的图像参数至少包括第一图像的各个像素点的位置参数、第一图像的方向参数，并且，向第一图像处理模型输入第一图像及第一偏振参数，得到第二图像，包括：向第一图像处理模型输入第一图像的图像参数、以及第一偏振参数，得到各个像素点的像素颜色参数；根据各个像素点的像素颜色参数，生成第二图像。In a possible implementation of the first aspect above, the image parameters of the first image include at least position parameters of each pixel point of the first image and direction parameters of the first image, and the first image and the first polarization parameters are input into the first image processing model to obtain the second image, including: inputting the image parameters of the first image and the first polarization parameters into the first image processing model to obtain pixel color parameters of each pixel point; generating the second image according to the pixel color parameters of each pixel point.
在此，第一图像的各个像素点的位置参数可以是下文中的位置参数(x，y，z)，第一图像的方向参数可以是下文中的目标方向参数(θ，Φ)，向第一图像处理模型输入第一图像的图像参数以及第一偏振参数可以是下文中的六维数据(x，y，z，θ，Φ，p)。第一图像处理模型输出的各个像素点的像素颜色参数可以是下文中基于NeRF模型得到的像素颜色。Here, the position parameters of each pixel point of the first image may be the position parameters (x, y, z) mentioned below, the direction parameters of the first image may be the target direction parameters (θ, Φ) mentioned below, and the image parameters of the first image and the first polarization parameters input to the first image processing model may be the six-dimensional data (x, y, z, θ, Φ, p) mentioned below. The pixel color parameters of each pixel point output by the first image processing model may be the pixel color obtained based on the NeRF model mentioned below.
在上述第一方面的一种可能实现中，得到第二图像之后，方法还包括：向第二图像处理模型输入第二图像、第一偏振参数、第一参照图像及第二参照图像，得到第三图像，其中，第一参照图像包括与第一图像的拍摄对象和/或拍摄场景相同的高分辨率偏振图像，第一参照图像的图像参数至少包括第一参照图像的各个像素点的位置参数、第一参照图像的方向参数及偏振参数；第二参照图像为第一参照图像的图像参数输入第一图像处理模型，得到的低分辨率偏振图像。In a possible implementation of the first aspect above, after obtaining the second image, the method also includes: inputting the second image, the first polarization parameters, the first reference image and the second reference image into a second image processing model to obtain a third image, wherein the first reference image includes a high-resolution polarization image that is the same as the photographed object and/or the photographed scene of the first image, and the image parameters of the first reference image include at least position parameters of each pixel point of the first reference image, direction parameters and polarization parameters of the first reference image; and the second reference image is a low-resolution polarization image obtained by inputting the image parameters of the first reference image into the first image processing model.
在此，第二图像可以是实施例1中，基于NeRF模型生成的具有高光细节的目标视角下的低分辨率拍摄图像。可以是实施例2中，基于NeRF模型生成的具有高光细节的原始视角下的低分辨率拍摄图像。Here, the second image may be a low-resolution image captured at a target viewing angle with highlight details generated based on the NeRF model in Example 1. It may be a low-resolution image captured at an original viewing angle with highlight details generated based on the NeRF model in Example 2.
第一参照图像可以是下文中拍摄图像对应匹配的高分辨率参照图像。第二参照图像可以是下文中将高分辨率参照图像输入NeRF模型中，得到的基于NeRF模型被退化的低分辨率参照图像。The first reference image may be a high-resolution reference image corresponding to the captured image described below. The second reference image may be a low-resolution reference image degraded based on the NeRF model obtained by inputting the high-resolution reference image into the NeRF model described below.
第三图像可以是实施例1中的具有高光细节的目标视角下的超分辨率拍摄图像。可以是实施例2中的具有高光细节的原始视角下的超分辨率拍摄图像。The third image may be the super-resolution image captured at the target viewing angle with highlight details in embodiment 1. It may be the super-resolution image captured at the original viewing angle with highlight details in embodiment 2.
具体地，基于高分辨率参照图像可向第二图像处理模型生成的图像引入边缘细节、纹理细节等高频信息、基于低分辨率参照图像可向第二图像处理模型生成的图像引入残差特征以进一步修复因退化过程而丢失或扭曲的高频信息。Specifically, based on the high-resolution reference image, high-frequency information such as edge details and texture details can be introduced into the image generated by the second image processing model, and based on the low-resolution reference image, residual features can be introduced into the image generated by the second image processing model to further repair the high-frequency information lost or distorted due to the degradation process.
基于前述内容可提高NeRF模型生成的具有高光细节的目标视角或原始视角下的低分辨率拍摄图像的清晰度，生成具有高光细节的目标视角或原始视角下的超分辨率拍摄图像。可以理解，第二图像处理模型可以是任何能基于高分辨率参照图像和/或低分辨率参照图像向第二图像引入高频信息的图像处理模型。前述高频信息或称高频细节，包括但不限于边缘信息、纹理信息、高光信息等。Based on the above content, the clarity of the low-resolution image captured at the target perspective or original perspective with highlight details generated by the NeRF model can be improved, and a super-resolution image captured at the target perspective or original perspective with highlight details can be generated. It can be understood that the second image processing model can be any image processing model that can introduce high-frequency information into the second image based on the high-resolution reference image and/or the low-resolution reference image. The above-mentioned high-frequency information or high-frequency details includes but is not limited to edge information, texture information, highlight information, etc.
在上述第一方面的一种可能实现中，第二图像处理模型可以包括参考帧的超分辨率(reference-based super resolution，RefSR)模型。In a possible implementation of the first aspect above, the second image processing model may include a reference-based super resolution (RefSR) model.
在此，本申请提供的图像处理方法可基于RefSR模型提高ReRF模型生成的图像的分辨率及图像质感。Here, the image processing method provided in the present application can improve the resolution and image texture of the image generated by the ReRF model based on the RefSR model.
在上述第一方面的一种可能实现中，第二图像处理模型包括第一模型、第二模型、第三模型，并且，向第二图像处理模型输入第二图像、第一偏振参数、第一参照图像及第二参照图像，得到第三图像，包括：向第一模型输入第一参照图像、第二图像以及第一偏振参数，得到第一特征向量；向第二模型输入第二参照图像以及第二图像，得到第二特征向量；向第三模型输入第一特征向量及第二特征向量，得到第三图像；其中，第三图像为超分辨率图像，第三图像包括第一偏振参数对应的第一高光信息、以及第一高频信息，其中，第一高频信息对应于第一特征向量和第二特征向量。In a possible implementation of the first aspect above, the second image processing model includes a first model, a second model, and a third model, and the second image, the first polarization parameter, the first reference image and the second reference image are input into the second image processing model to obtain the third image, including: inputting the first reference image, the second image and the first polarization parameter into the first model to obtain the first eigenvector; inputting the second reference image and the second image into the second model to obtain the second eigenvector; inputting the first eigenvector and the second eigenvector into the third model to obtain the third image; wherein the third image is a super-resolution image, and the third image includes the first highlight information corresponding to the first polarization parameter, and the first high-frequency information, wherein the first high-frequency information corresponds to the first eigenvector and the second eigenvector.
在此，第一模型可以是下文中的RefSR模型通过高频建模得到的高频模型，第二模型可以是下文中的RefSR模型通过退化建模得到的退化模型，第三模型可以是下文中RefSR模型的融合模块。Here, the first model may be a high-frequency model obtained by high-frequency modeling of the RefSR model described below, the second model may be a degradation model obtained by degradation modeling of the RefSR model described below, and the third model may be a fusion module of the RefSR model described below.
具体地，第一特征向量可以是下文中，将高分辨率参照图像及低分辨率拍摄图像结合目标偏振参数通过RefSR模型中的高频模型输出的高频特征。Specifically, the first feature vector may be a high-frequency feature output by a high-frequency model in a RefSR model by combining a high-resolution reference image and a low-resolution captured image with target polarization parameters.
第二特征向量可以是下文中，将具有高光细节的目标视角下的低分辨率拍摄图像、低分辨率参照图像、高分辨率参照图像、目标偏振参数输入RefSR模型中的退化模型中，输出的残差特征。The second feature vector may be the residual feature outputted by inputting a low-resolution captured image with highlight details at a target perspective, a low-resolution reference image, a high-resolution reference image, and target polarization parameters into a degradation model in the RefSR model.
第三图像可以是基于RefSR模型中的融合模块将残差特征及高频特征进行融合，得到的具有高光细节的目标视角或原始视角下的超分辨率拍摄图像。在此，第一高频信息可以是前述高频特征及残差特征在第三图像中对应的高频信息，例如，边缘细节、纹理细节等。The third image may be a super-resolution image captured at a target perspective or an original perspective with highlight details obtained by fusing the residual features and the high-frequency features based on the fusion module in the RefSR model. Here, the first high-frequency information may be the high-frequency information corresponding to the aforementioned high-frequency features and the residual features in the third image, such as edge details, texture details, etc.
可以理解，基于上述方式可将拍摄图像调整为具有高光细节的目标视角或原始视角下的超分辨率拍摄图像。It can be understood that based on the above method, the captured image can be adjusted to a super-resolution captured image at a target viewing angle or an original viewing angle with highlight details.
在上述第一方面的一种可能实现中，向第一模型输入第一参照图像、第二图像以及第一偏振参数，得到第一特征向量，包括：对第二图像的上采样结果和第一参照图像进行空间至深度的重排操作，得到第三特征向量；将第三特征向量输入第一模型中的编码器，得到第四特征向量；基于第一模型将第一偏振参数拼接至第四特征向量中，得到第五特征向量；将第五特征向量输入第一模型中的解码器，得到第一特征向量。In a possible implementation of the first aspect above, a first reference image, a second image and a first polarization parameter are input into a first model to obtain a first eigenvector, including: performing a space-to-depth rearrangement operation on the upsampling result of the second image and the first reference image to obtain a third eigenvector; inputting the third eigenvector into an encoder in the first model to obtain a fourth eigenvector; splicing the first polarization parameter into the fourth eigenvector based on the first model to obtain a fifth eigenvector; and inputting the fifth eigenvector into a decoder in the first model to obtain the first eigenvector.
在此，基于对第二图像的上采样，可以初步提升NeRF模型生成的具有高光细节的目标视角或原始视角下的低分辨率拍摄图像的图像质量。Here, based on upsampling of the second image, the image quality of the low-resolution captured image at the target perspective or the original perspective with high light details generated by the NeRF model can be preliminarily improved.
前述空间至深度的重排操作可以是下文中的空间至深度（space to depth，S2D)操作。通过该S2D操作，可实现第二图像的上采样结果和第一参照图像之间的特征融合，其融合结果可为第三特征向量。可以理解，第三特征向量可为融合了第二图像的上采样结果和第一参照图像的特征图。The aforementioned space-to-depth rearrangement operation may be a space-to-depth (S2D) operation hereinafter. Through the S2D operation, feature fusion between the up-sampled result of the second image and the first reference image may be achieved, and the fusion result may be a third feature vector. It is understood that the third feature vector may be a feature map that fuses the up-sampled result of the second image and the first reference image.
进而，将第三特征向量输入第一模型中的编码器得到第四特征向量，可以包括，将融合了低分辨率拍摄图像的上采样结果和高分辨率参照图像的特征图输入高频模型的编码器中，通过该编码器输出高频特征向量（作为第四特征向量的一种示例）。Furthermore, inputting the third feature vector into the encoder in the first model to obtain the fourth feature vector can include inputting a feature map that fuses the upsampling result of the low-resolution captured image and the high-resolution reference image into the encoder of the high-frequency model, and outputting the high-frequency feature vector (as an example of the fourth feature vector) through the encoder.
基于第一模型将第一偏振参数拼接至第四特征向量中，得到第五特征向量，可以是基于高频模型将目标偏振参数拼接到编码器输出的高频特征向量中。对应地，第五特征向量可以是包含了目标偏振参数的高频特征向量。在此，可以基于concat函数将目标偏振参数拼接至高频特征向量中。本申请不对具体的拼接方式做限制性说明。Based on the first model, the first polarization parameter is spliced into the fourth eigenvector to obtain the fifth eigenvector, and the target polarization parameter can be spliced into the high-frequency eigenvector output by the encoder based on the high-frequency model. Correspondingly, the fifth eigenvector can be a high-frequency eigenvector containing the target polarization parameter. Here, the target polarization parameter can be spliced into the high-frequency eigenvector based on the concat function. This application does not make a restrictive description of the specific splicing method.
进而，将第五特征向量输入第一模型中的解码器，得到第一特征向量，可以是将包含了目标偏振参数的高频特征向量输入高频模型的解码器中，得到高频模型输出的高频特征。Furthermore, the fifth eigenvector is input into the decoder in the first model to obtain the first eigenvector, which may be inputting the high-frequency eigenvector including the target polarization parameter into the decoder of the high-frequency model to obtain the high-frequency features output by the high-frequency model.
可以理解，高频模型输出的高频特征可用于建构前述第一高频信息。It can be understood that the high-frequency features output by the high-frequency model can be used to construct the aforementioned first high-frequency information.
在上述第一方面的一种可能实现中，向第二模型输入第二参照图像以及第二图像，得到第二特征向量，包括：向第二模型输入第二参照图像以及第二图像，得到第六特征向量；对第六特征向量进行深度至空间的重排操作，得到第二特征向量。In a possible implementation of the first aspect above, inputting the second reference image and the second image into the second model to obtain the second eigenvector includes: inputting the second reference image and the second image into the second model to obtain the sixth eigenvector; and performing a depth-to-space rearrangement operation on the sixth eigenvector to obtain the second eigenvector.
在此，向第二模型输入第二参照图像以及第二图像，得到第六特征向量，可以包括向退化模型输入低分辨率参照图像以及低分辨率拍摄图像，得到退化模型输出的残差特征向量（作为第六特征向量的一种示例）。Here, inputting the second reference image and the second image into the second model to obtain the sixth eigenvector may include inputting a low-resolution reference image and a low-resolution captured image into the degradation model to obtain a residual eigenvector output by the degradation model (as an example of the sixth eigenvector).
在此，深度至空间的重排操作可以是下文中的深度至空间（depth to space，D2S)操作。进而，对第六特征向量进行D2S操作，可得到特征重排后的残差特征（作为第二特征向量的一种示例），以便于与高频特征进行特征融合。Here, the depth-to-space rearrangement operation may be a depth-to-space (D2S) operation described below. Further, the D2S operation is performed on the sixth eigenvector to obtain a residual feature after feature rearrangement (as an example of a second eigenvector) to facilitate feature fusion with the high-frequency feature.
第二方面，本申请提供了一种模型训练方法，应用于电子设备，方法包括：获取若干个第四图像，其中，第四图像包括对同一拍摄对象基于不同拍摄视角拍摄得到的高分辨率偏振图像，第四图像的图像参数至少包括第四图像的各个像素点的位置参数、第四图像的方向参数以及偏振参数；向第一图像处理模型输入第四图像的图像参数，得到第四图像的各个像素点的训练像素颜色参数；根据第四图像的各个像素点的训练像素颜色参数以及实际颜色参数通过损失函数计算损失值；调节第一图像处理模型的参数使得损失值位于预设区间。In a second aspect, the present application provides a model training method for application to electronic devices, the method comprising: acquiring a plurality of fourth images, wherein the fourth images comprise high-resolution polarization images obtained by photographing the same object based on different photographing angles, and the image parameters of the fourth images comprise at least position parameters of each pixel point of the fourth image, direction parameters of the fourth image, and polarization parameters; inputting the image parameters of the fourth image into a first image processing model to obtain training pixel color parameters of each pixel point of the fourth image; calculating a loss value through a loss function based on the training pixel color parameters and actual color parameters of each pixel point of the fourth image; and adjusting the parameters of the first image processing model so that the loss value is within a preset range.
在此，第四图像可以是下文中的高分辨率参照图像，若干个第四图像可以是下文中的多组高分辨率参照图像。第四图像的各个像素点的位置参数可以是下文中的高分辨率参照图像的位置参数(x，y，z)。第四图像的方向参数可以是下文中表征该高分辨率参照图像的拍摄视角的方向参数(θ，Φ)。第四图像的偏振参数可以是下文中表征拍摄该高分辨率参照图像的偏振滤镜或偏振镜头的偏振方向的偏振参数p。Here, the fourth image may be a high-resolution reference image as described below, and the plurality of fourth images may be a plurality of groups of high-resolution reference images as described below. The position parameters of each pixel of the fourth image may be position parameters (x, y, z) of the high-resolution reference image as described below. The direction parameters of the fourth image may be direction parameters (θ, Φ) of the shooting angle of view of the high-resolution reference image as described below. The polarization parameters of the fourth image may be polarization parameters p of the polarization direction of the polarization filter or polarization lens used to shoot the high-resolution reference image as described below.
第四图像的各个像素点的训练像素颜色参数可以是NeRF模型输出的，该像素点对应的空间点在高分辨率参照图像的拍摄视角下对应的像素点的像素颜色。在此，第四图像的各个像素点的实际像素颜色参数可以是高分辨率参照图像中各个像素点的原始的像素颜色。The training pixel color parameters of each pixel of the fourth image may be the pixel color of the corresponding spatial point of the pixel under the shooting angle of the high-resolution reference image output by the NeRF model. Here, the actual pixel color parameters of each pixel of the fourth image may be the original pixel color of each pixel in the high-resolution reference image.
在上述第二方面的一种可能实现中，第一图像处理模型包括NeRF模型。In a possible implementation of the second aspect above, the first image processing model includes a NeRF model.
在此，NeRF模型输入的训练数据包括为具有偏振参数对应的高光效果的第四图像，因此，基于第四图像以及第四图像的偏振参数训练NeRF模型，可使该NeRF模型学习到偏振参数与高光效果之间的对应关系。进而，可在应用该NeRF模型的过程中，基于输入的目标偏振参数为输入的拍摄图像增加该目标偏振参数对应的高光效果。Here, the training data input to the NeRF model includes a fourth image having a highlight effect corresponding to the polarization parameter. Therefore, by training the NeRF model based on the fourth image and the polarization parameter of the fourth image, the NeRF model can learn the corresponding relationship between the polarization parameter and the highlight effect. Furthermore, in the process of applying the NeRF model, based on the input target polarization parameter, the highlight effect corresponding to the target polarization parameter can be added to the input captured image.
在上述第二方面的一种可能实现中，该图像处理方法还包括：基于第四图像的各个像素点的训练像素颜色参数得到第五图像，其中，第五图像为第四图像对应的低分辨率偏振图像。In a possible implementation of the second aspect above, the image processing method further includes: obtaining a fifth image based on training pixel color parameters of each pixel point of the fourth image, wherein the fifth image is a low-resolution polarization image corresponding to the fourth image.
在此，第五图像可以是高分辨率参照图像经过该NeRF模型被退化为的低分辨率参照图像。可以理解，基于前述方式，NeRF模型输出的像素颜色受到了高分辨率参照图像的偏振参数的影响，因此，在经过多个训练迭代过程，使得损失值位于预设区间后，NeRF模型基于各个像素点的像素颜色输出的低分辨率偏振图像同样可呈现出该偏振参数对应的高光效果。在此，本申请不对损失值应位于的预设区间做限制性说明。Here, the fifth image can be a low-resolution reference image degraded by the NeRF model to a high-resolution reference image. It can be understood that, based on the aforementioned method, the pixel color output by the NeRF model is affected by the polarization parameters of the high-resolution reference image. Therefore, after multiple training iterations, the loss value is located in a preset interval. The low-resolution polarization image output by the NeRF model based on the pixel color of each pixel point can also present a highlight effect corresponding to the polarization parameter. Here, the present application does not make a restrictive description of the preset interval in which the loss value should be located.
在上述第二方面的一种可能实现中，该图像处理方法还包括：向第二图像处理模型输入第四图像、第五图像、以及第四图像的偏振参数，得到第六图像；根据第六图像以及第四图像通过损失函数计算损失值；调节第二图像处理模型的参数使得损失值位于预设区间。In a possible implementation of the second aspect above, the image processing method also includes: inputting the fourth image, the fifth image, and the polarization parameters of the fourth image into the second image processing model to obtain a sixth image; calculating the loss value through a loss function based on the sixth image and the fourth image; and adjusting the parameters of the second image processing model so that the loss value is within a preset range.
在此，第五图像可以包括下文中的低分辨率参照图像和/或新视角下的低分辨率参照图像。第六图像可以是下文中新视角下的超分辨率参照图像。Here, the fifth image may include a low-resolution reference image described below and/or a low-resolution reference image at a new viewing angle. The sixth image may be a super-resolution reference image at a new viewing angle described below.
具体地，基于高分辨率参照图像可向第二图像处理模型生成的图像引入边缘细节、纹理细节等高频信息、基于低分辨率参照图像和/或新视角下的低分辨率参照图像可向第二图像处理模型生成的图像引入残差特征以进一步修复因退化过程而丢失或扭曲的高频信息。基于前述内容可提高NeRF模型生成的低分辨率参照图像的清晰度。可以理解，第二图像处理模型可以是任何能基于高分辨率参照图像和/或低分辨率参照图像向NeRF模型生成的低分辨率参照图像引入高频信息的图像处理模型。前述高频信息或称高频细节，包括但不限于边缘信息、纹理信息、高光信息等。Specifically, high-frequency information such as edge details and texture details can be introduced into the image generated by the second image processing model based on the high-resolution reference image, and residual features can be introduced into the image generated by the second image processing model based on the low-resolution reference image and/or the low-resolution reference image under the new perspective to further repair the high-frequency information lost or distorted due to the degradation process. Based on the foregoing, the clarity of the low-resolution reference image generated by the NeRF model can be improved. It can be understood that the second image processing model can be any image processing model that can introduce high-frequency information into the low-resolution reference image generated by the NeRF model based on the high-resolution reference image and/or the low-resolution reference image. The foregoing high-frequency information or high-frequency details includes but is not limited to edge information, texture information, highlight information, etc.
在上述第二方面的一种可能实现中，第二图像处理模型包括RefSR模型。In a possible implementation of the second aspect, the second image processing model includes a RefSR model.
在此，本申请提供的图像处理方法可基于RefSR模型提高ReRF模型生成的图像的分辨率及图像质感。Here, the image processing method provided in the present application can improve the resolution and image texture of the image generated by the ReRF model based on the RefSR model.
在上述第二方面的一种可能实现中，第二图像处理模型包括第一模型、第二模型、第三模型，并且，向第二图像处理模型输入第四图像、第五图像、以及第四图像的偏振参数，得到第六图像，包括：将第四图像、第五图像、第四图像的偏振参数输入第一模型得到第一训练特征向量；将第五图像输入第二模型得到第二训练特征向量；将第一训练特征向量及第二训练特征向量输入第三模型，得到第六图像。In a possible implementation of the second aspect above, the second image processing model includes a first model, a second model, and a third model, and the fourth image, the fifth image, and the polarization parameters of the fourth image are input into the second image processing model to obtain the sixth image, including: inputting the polarization parameters of the fourth image, the fifth image, and the fourth image into the first model to obtain a first training feature vector; inputting the fifth image into the second model to obtain a second training feature vector; and inputting the first training feature vector and the second training feature vector into the third model to obtain the sixth image.
在此，第一模型可以是下文中的RefSR模型通过高频建模得到的高频模型，第二模型可以是下文中的RefSR模型通过退化建模得到的退化模型，第三模型可以是下文中RefSR模型的融合模块。Here, the first model may be a high-frequency model obtained by high-frequency modeling of the RefSR model described below, the second model may be a degradation model obtained by degradation modeling of the RefSR model described below, and the third model may be a fusion module of the RefSR model described below.
具体地，第一训练特征向量可以是下文中，将高分辨率参照图像、新视角的低分辨率参照图像、新视角对应的偏振参数输入高频模型中，得到的高频特征。可以理解，新视角对应的偏振参数也可是高分辨率参照图像对应的偏振参数，相当于，新视角的低分辨率参照图像即为高分辨率参照图像输入NeRF模型后，退化得到的低分辨率参照图像。Specifically, the first training feature vector may be a high-frequency feature obtained by inputting a high-resolution reference image, a low-resolution reference image of a new perspective, and a polarization parameter corresponding to the new perspective into a high-frequency model as described below. It can be understood that the polarization parameter corresponding to the new perspective may also be a polarization parameter corresponding to the high-resolution reference image, which is equivalent to the low-resolution reference image of the new perspective being a low-resolution reference image degraded after the high-resolution reference image is input into the NeRF model.
对应于第五图像包括低分辨率参照图像及新视角下的低分辨率参照图像的场景，将第五图像输入第二模型得到第二训练特征向量，可以是，将低分辨率参照图像及新视角下的低分辨率参照图像输入退化模型中，得到残差特征。Corresponding to the scene where the fifth image includes a low-resolution reference image and a low-resolution reference image at a new perspective, the fifth image is input into the second model to obtain a second training feature vector. Alternatively, the low-resolution reference image and the low-resolution reference image at a new perspective are input into a degradation model to obtain residual features.
将第一训练特征向量及第二训练特征向量输入第三模型，得到第六图像，可以是，将基于高频模型得到的高频特征以及基于退化模型得到的残差特征输入融合模块，得到新视角下的超分辨率参照图像。The first training feature vector and the second training feature vector are input into the third model to obtain the sixth image, which can be that the high-frequency features obtained based on the high-frequency model and the residual features obtained based on the degradation model are input into the fusion module to obtain the super-resolution reference image under the new perspective.
在上述第二方面的一种可能实现中，将第四图像、第五图像、第四图像的偏振参数输入第一模型得到第一训练特征向量，包括：将第五图像的上采样结果和第四图像进行空间至深度的重排操作，得到第三训练特征向量；将第三训练特征向量输入第一模型中的编码器，得到第四训练特征向量；将第四图像的偏振参数拼接至第四训练特征向量，得到第五训练特征向量；将第五训练特征向量输入第一模型中的解码器，得到第一训练特征向量。In a possible implementation of the second aspect above, the fourth image, the fifth image, and the polarization parameters of the fourth image are input into the first model to obtain a first training feature vector, including: performing a space-to-depth rearrangement operation on the upsampling result of the fifth image and the fourth image to obtain a third training feature vector; inputting the third training feature vector into the encoder in the first model to obtain a fourth training feature vector; splicing the polarization parameters of the fourth image to the fourth training feature vector to obtain a fifth training feature vector; and inputting the fifth training feature vector into the decoder in the first model to obtain the first training feature vector.
在此，对应于第五图像包括低分辨率参照图像及新视角下的低分辨率参照图像的场景，对第五图像的上采样，包括，对第五图像中的新视角下的低分辨率参照图像的上采样。基于该方式，可以初步提升NeRF模型生成的新视角下的低分辨率参照图像的图像质量。Here, corresponding to the scenario where the fifth image includes a low-resolution reference image and a low-resolution reference image under a new perspective, upsampling the fifth image includes upsampling the low-resolution reference image under a new perspective in the fifth image. Based on this approach, the image quality of the low-resolution reference image under a new perspective generated by the NeRF model can be preliminarily improved.
前述空间至深度的重排操作可以是下文中的空间至深度（space to depth，S2D)操作。通过该S2D操作，可实现新视角下的低分辨率参照图像的上采样结果和高分辨率参照图像之间的特征融合，其融合结果可为第三训练特征向量。可以理解，第三特征向量可为融合了新视角下的低分辨率参照图像的上采样结果和高分辨率参照图像的特征图。The aforementioned space-to-depth rearrangement operation may be a space-to-depth (S2D) operation hereinafter. Through the S2D operation, feature fusion between the upsampling result of the low-resolution reference image under the new perspective and the high-resolution reference image may be achieved, and the fusion result may be a third training feature vector. It is understood that the third feature vector may be a feature map that fuses the upsampling result of the low-resolution reference image under the new perspective and the high-resolution reference image.
进而，将第三训练特征向量输入第一模型中的编码器得到第四训练特征向量，可以包括，将融合了新视角下的低分辨率参照图像的上采样结果和高分辨率参照图像的特征图输入高频模型的编码器中，通过该编码器输出高频特征向量（作为第四训练特征向量的一种示例）。Furthermore, inputting the third training feature vector into the encoder in the first model to obtain the fourth training feature vector can include inputting the upsampling result of the low-resolution reference image at the new perspective and the feature map of the high-resolution reference image into the encoder of the high-frequency model, and outputting the high-frequency feature vector (as an example of the fourth training feature vector) through the encoder.
基于第一模型将第四图像的偏振参数拼接至第四训练特征向量中，得到第五训练特征向量，可以是，基于高频模型将第四图像的偏振参数拼接到编码器输出的高频特征向量中。对应地，第五训练特征向量可以是包含了第四图像的偏振参数的高频特征向量。在此，可以基于concat函数将第四图像的偏振参数拼接至高频特征向量中。本申请不对具体的拼接方式做限制性说明。Based on the first model, the polarization parameters of the fourth image are spliced into the fourth training feature vector to obtain the fifth training feature vector. The fifth training feature vector can be obtained by splicing the polarization parameters of the fourth image into the high-frequency feature vector output by the encoder based on the high-frequency model. Correspondingly, the fifth training feature vector can be a high-frequency feature vector containing the polarization parameters of the fourth image. Here, the polarization parameters of the fourth image can be spliced into the high-frequency feature vector based on the concat function. This application does not make a restrictive description of the specific splicing method.
进而，将第五特征训练向量输入第一模型中的解码器，得到第一训练特征向量，可以是将包含了第四图像的偏振参数的高频特征向量输入高频模型的解码器中，得到高频模型输出的高频特征。Furthermore, the fifth feature training vector is input into the decoder in the first model to obtain the first training feature vector, which can be input into the decoder of the high-frequency model containing the polarization parameters of the fourth image to obtain the high-frequency features output by the high-frequency model.
可以理解，高频模型输出的高频特征可用于建构第六图像的高频信息。It can be understood that the high-frequency features output by the high-frequency model can be used to construct the high-frequency information of the sixth image.
在上述第二方面的一种可能实现中，将第五图像输入第二模型得到第二训练特征向量，包括：向第二模型输入第五图像，得到第六训练特征向量；对第六训练特征向量进行深度至空间的重排操作，得到第二训练特征向量。In a possible implementation of the second aspect above, inputting the fifth image into the second model to obtain the second training feature vector includes: inputting the fifth image into the second model to obtain the sixth training feature vector; and performing a depth-to-space rearrangement operation on the sixth training feature vector to obtain the second training feature vector.
在此，向第二模型输入第五图像，得到第六训练特征向量，可以包括，向退化模型输入低分辨率参照图像以及新视角下的低分辨率参照图像，得到退化模型输出的残差特征向量（作为第六训练特征向量的一种示例）。Here, inputting the fifth image into the second model to obtain the sixth training feature vector may include inputting a low-resolution reference image and a low-resolution reference image at a new perspective into the degradation model to obtain a residual feature vector output by the degradation model (as an example of the sixth training feature vector).
在此，深度至空间的重排操作可以是下文中的深度至空间（depth to space，D2S)操作。进而，对第六训练特征向量进行D2S操作，可得到特征重排后的残差特征（作为第二特征训练向量的一种示例），以便于与高频特征进行特征融合。Here, the depth-to-space rearrangement operation may be a depth-to-space (D2S) operation described below. Further, the D2S operation is performed on the sixth training feature vector to obtain a residual feature after feature rearrangement (as an example of a second feature training vector) to facilitate feature fusion with high-frequency features.
第三方面，本申请提供了一种电子设备，包括：一个或多个处理器；一个或多个存储器；一个或多个存储器存储有一个或多个程序，当一个或者多个程序被一个或多个处理器执行时，使得电子设备执行前述第一方面以及第一方面的各种可能实现提供的图像处理方法、或者执行前述第二方面以及第二方面的各种可能实现提供的模型训练方法。In a third aspect, the present application provides an electronic device comprising: one or more processors; one or more memories; one or more memories storing one or more programs, wherein when one or more programs are executed by one or more processors, the electronic device executes the image processing method provided by the aforementioned first aspect and various possible implementations of the first aspect, or executes the model training method provided by the aforementioned second aspect and various possible implementations of the second aspect.
第四方面，本申请提供了一种计算机可读介质，该可读介质上存储有指令，指令在计算机上执行时使计算机执行前述第一方面以及第一方面的各种可能实现提供的图像处理方法、或者执行前述第二方面以及第二方面的各种可能实现提供的模型训练方法。In a fourth aspect, the present application provides a computer-readable medium having instructions stored thereon, which, when executed on a computer, causes the computer to execute the image processing method provided by the aforementioned first aspect and various possible implementations of the first aspect, or to execute the model training method provided by the aforementioned second aspect and various possible implementations of the second aspect.
第五方面，本申请提供了一种计算机程序产品，包括计算机程序/指令，计算机程序/指令被处理器执行时实现前述第一方面以及第一方面的各种可能实现提供的图像处理方法、或者实现前述第二方面以及第二方面的各种可能实现提供的模型训练方法。In a fifth aspect, the present application provides a computer program product, including a computer program/instructions, which, when executed by a processor, implements the image processing method provided by the aforementioned first aspect and various possible implementations of the first aspect, or implements the model training method provided by the aforementioned second aspect and various possible implementations of the second aspect.
上述第二方面至第五方面的有益效果，可以参考上述第一方面以及第一方面的各种可能实现中的相关描述，在此不做赘述。The beneficial effects of the second to fifth aspects mentioned above can be referred to the relevant descriptions in the first aspect mentioned above and various possible implementations of the first aspect, and will not be elaborated here.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1所示为本申请提供的一种基于NeRF模型生成目标视角下的拍摄图像的效果示意图；FIG1 is a schematic diagram showing the effect of generating a captured image at a target viewing angle based on a NeRF model provided by the present application;
图2所示为本申请实施例提供的一种基于NeRF模型生成具有高光效果的目标视角下的拍摄图像的过程示意图；FIG2 is a schematic diagram showing a process of generating a captured image with a highlight effect at a target viewing angle based on a NeRF model provided in an embodiment of the present application;
图3所示为本申请实施例提供的一种图像处理方法的流程示意图；FIG3 is a schematic diagram showing a flow chart of an image processing method provided in an embodiment of the present application;
图4a所示为本申请实施例提供的一种用户输入目标方向参数及目标偏振参数的效果示意图；FIG4a is a schematic diagram showing the effect of a user inputting a target direction parameter and a target polarization parameter provided in an embodiment of the present application;
图4b所示为本申请实施例提供的另一种用户输入目标方向参数及目标偏振参数的效果示意图；FIG4 b is a schematic diagram showing another effect of a user inputting a target direction parameter and a target polarization parameter provided in an embodiment of the present application;
图4c所示为本申请实施例提供的图像处理方法的一种应用场景；FIG4c shows an application scenario of the image processing method provided in an embodiment of the present application;
图5所示为本申请实施例提供的另一种图像处理方法的流程示意图；FIG5 is a schematic diagram showing a flow chart of another image processing method provided in an embodiment of the present application;
图6所示为本申请实施例提供的一种用户输入目标偏振参数的效果示意图；FIG6 is a schematic diagram showing the effect of a user inputting a target polarization parameter according to an embodiment of the present application;
图7a所示为本申请实施例提供的一种NeRF模型的训练过程示意图；FIG7a is a schematic diagram showing a training process of a NeRF model provided in an embodiment of the present application;
图7b所示为本申请实施例提供的一种RefSR模型的训练过程示意图；FIG7 b is a schematic diagram showing a training process of a RefSR model provided in an embodiment of the present application;
图7c所示为本申请实施例提供的一种基于NeRF模型及RefSR模型生成高分辨拍摄图像的过程示意图；FIG7c is a schematic diagram showing a process of generating a high-resolution captured image based on a NeRF model and a RefSR model provided in an embodiment of the present application;
图8所示为本申请实施例提供的一种手机100的结构示意图；FIG8 is a schematic diagram showing the structure of a mobile phone 100 provided in an embodiment of the present application;
图9所示为本申请实施例提供的一种手机100的软件结构框图。FIG9 is a software structure block diagram of a mobile phone 100 provided in an embodiment of the present application.
具体实施方式Detailed ways
为了使本申请实施例的目的、技术方案和优点更加清楚，下面将结合说明书附图以及具体的实施方式对本申请实施例中的技术方案进行详细的说明。In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
可以理解，本申请实施例所提供的图像处理方法，所适用的电子设备可以包括但不限于手机、平板电脑、桌面型、膝上型、手持计算机、上网本，以及增强现实（augmentedreality，AR）\虚拟现实（virtual reality，VR）设备、智能电视、智能手表等可穿戴设备、服务器、移动电子邮件设备、车机设备、及其他嵌入或耦接有一个或多个处理器的电视机或其他电子设备。It can be understood that the image processing method provided in the embodiments of the present application may be applicable to electronic devices including but not limited to mobile phones, tablet computers, desktops, laptops, handheld computers, netbooks, as well as augmented reality (AR) and virtual reality (VR) devices, smart TVs, smart watches and other wearable devices, servers, mobile email devices, car equipment, and other televisions or other electronic devices embedded or coupled with one or more processors.
为了便于本领域技术人员理解本申请实施例中的方案，下面首先对基于NeRF模型生成目标视角下的拍摄图像的过程及原理进行解释说明。In order to facilitate those skilled in the art to understand the solutions in the embodiments of the present application, the process and principle of generating the captured image under the target perspective based on the NeRF model are first explained below.
在训练NeRF模型的过程中，NeRF模型的训练数据可以为多组拍摄图像，其中，各组拍摄图像中可以包括以多个视角拍摄相同对象的多个拍摄图像。NeRF模型的输入数据可包括拍摄图像的各个像素点的位置参数(x，y，z)以及表征目标视角方向的方向参数(θ，Φ)。其中，前述像素点的位置参数(x，y，z)可以是该像素点对应的空间点在世界坐标系下的三维位置坐标。其中，目标视角方向可以是需要基于NeRF模型得到的拍摄视角，θ可以是该目标视角方向对应的相机位姿参数，Φ可以是该目标视角方向对应的相机内参，前述相机内参可以包括能够表征目标视角方向的焦距、畸变系数、缩放因子等参数。In the process of training the NeRF model, the training data of the NeRF model may be multiple groups of captured images, wherein each group of captured images may include multiple captured images of the same object captured from multiple perspectives. The input data of the NeRF model may include the position parameters (x, y, z) of each pixel point of the captured image and the direction parameters (θ, Φ) representing the target viewing direction. Among them, the position parameters (x, y, z) of the aforementioned pixel points may be the three-dimensional position coordinates of the spatial point corresponding to the pixel point in the world coordinate system. Among them, the target viewing direction may be the shooting viewing angle that needs to be obtained based on the NeRF model, θ may be the camera pose parameter corresponding to the target viewing direction, Φ may be the camera intrinsic parameter corresponding to the target viewing direction, and the aforementioned camera intrinsic parameter may include parameters such as focal length, distortion coefficient, and scaling factor that can represent the target viewing direction.
在每个训练迭代过程中，可从训练数据中选择一批像素点，在由相机到各像素点对应的空间点的光线上选取若干个采样点，基于NeRF模型预测各个采样点的色彩值和体密度值。进而，基于体渲染技术将若干个采样点的色彩值和体密度值进行积分整合，最终得到的积分结果即可作为该光线投射生成的像素点的预测像素颜色。In each training iteration, a batch of pixels can be selected from the training data, and several sampling points can be selected on the light from the camera to the spatial point corresponding to each pixel, and the color value and volume density value of each sampling point can be predicted based on the NeRF model. Then, the color values and volume density values of several sampling points are integrated based on the volume rendering technology, and the final integrated result can be used as the predicted pixel color of the pixel generated by the projection of the light.
在此，NeRF模型可基于损失函数计算预测像素颜色和实际像素颜色之间的损失，通过梯度下降法等方式调整NeRF模型中的参数，基于多测训练迭代将该损失收敛至满意水平，以此提高NeRF模型的预测准确性。Here, the NeRF model can calculate the loss between the predicted pixel color and the actual pixel color based on the loss function, adjust the parameters in the NeRF model through methods such as gradient descent, and converge the loss to a satisfactory level based on multiple training iterations, thereby improving the prediction accuracy of the NeRF model.
基于以上训练过程，在应用NeRF模型生成目标视角下的拍摄图像的场景中，可向训练好的NeRF模型输入拍摄图像的各个像素点的位置参数(x，y，z)以及表征目标视角的方向参数（θ，Φ）。NeRF模型可以基于输入的五维数据（x，y，z，θ，Φ）预测像素点对应的若干个采样点的色彩值和体密度值。进而，NeRF模型可以基于体渲染技术将若干个采样点的色彩值和体密度值进行积分整合，得到该光线投射生成的像素点的像素颜色。生成的各个像素点的像素颜色即可用于生成该目标视角下的拍摄图像。可以理解，目标视角下的拍摄图像可为基于NeRF模型将拍摄图像的拍摄视角改变为目标视角后的图像。Based on the above training process, in the scenario where the NeRF model is used to generate a captured image under the target perspective, the position parameters (x, y, z) of each pixel of the captured image and the direction parameters (θ, Φ) representing the target perspective can be input into the trained NeRF model. The NeRF model can predict the color values and volume density values of several sampling points corresponding to the pixel points based on the input five-dimensional data (x, y, z, θ, Φ). Furthermore, the NeRF model can integrate the color values and volume density values of several sampling points based on the volume rendering technology to obtain the pixel color of the pixel generated by the ray projection. The pixel colors of each pixel generated can be used to generate the captured image under the target perspective. It can be understood that the captured image under the target perspective can be an image after the shooting perspective of the captured image is changed to the target perspective based on the NeRF model.
基于前述NeRF模型的训练过程及应用过程可知，NeRF模型对于各个像素点的像素颜色的确定过程是相互独立的，并未考虑不同像素点之间的空间关系对像素颜色的影响。进而，可能导致生成的目标视角下的拍摄图像缺乏高光细节等。Based on the training and application process of the NeRF model, it can be seen that the NeRF model determines the pixel color of each pixel independently, and does not consider the impact of the spatial relationship between different pixels on the pixel color. As a result, the generated image captured at the target perspective may lack highlight details.
下面结合前述对NeRF模型生成的目标视角下的拍摄图像缺乏高光细节的说明，详细介绍本申请提供的可提升目标视角下的拍摄图像中的高光细节的图像处理方法。In combination with the above description of the lack of highlight details in the captured images at the target perspective generated by the NeRF model, the image processing method provided by the present application for improving the highlight details in the captured images at the target perspective is described in detail.
可以理解，具有不同偏振参数的偏振滤镜或偏振镜头可以过滤不同方向的反射光线，因此，采用偏振相机拍摄得到的偏振图像可呈现出该偏振相机的偏振参数对应的高光效果。在此，偏振相机包括设置了偏振滤镜或内置了偏振镜头的电子设备，偏振相机的偏振参数可为偏振滤镜或偏振镜头的偏振参数，前述偏振参数可包括偏振方向等。It can be understood that polarization filters or polarization lenses with different polarization parameters can filter reflected light in different directions. Therefore, the polarization image captured by the polarization camera can present a highlight effect corresponding to the polarization parameter of the polarization camera. Here, the polarization camera includes an electronic device with a polarization filter or a built-in polarization lens. The polarization parameter of the polarization camera can be the polarization parameter of the polarization filter or the polarization lens, and the aforementioned polarization parameter can include polarization direction, etc.
具体地，以拍摄人像为例，可以通过调整偏振滤镜或偏振镜头的偏振参数，来过滤部分投射至人像非高光区域的光线。可以理解，过滤部分投射至人像非高光区域的光线后，人像的非高光区域的像素颜色可能发生变化，例如，非高光区域的像素纯度、明度、亮度可能变低等。因此，通过调整偏振参数可扩大上述非高光区域与高光区域的差异，包括两区域之间的像素纯度、明度、亮度差异等。进而，能够凸显人像的高光区域，使得拍摄得到的偏振图像中的人像呈现出明显的高光效果。Specifically, taking portrait photography as an example, you can filter out part of the light projected onto the non-highlight area of the portrait by adjusting the polarization parameters of the polarization filter or polarization lens. It is understandable that after filtering out part of the light projected onto the non-highlight area of the portrait, the pixel color of the non-highlight area of the portrait may change, for example, the pixel purity, brightness, and luminance of the non-highlight area may become lower. Therefore, by adjusting the polarization parameters, the difference between the above-mentioned non-highlight area and the highlight area can be enlarged, including the difference in pixel purity, brightness, and luminance between the two areas. Furthermore, the highlight area of the portrait can be highlighted, so that the portrait in the captured polarized image presents an obvious highlight effect.
因此，为解决前述NeRF模型生成的目标视角下的拍摄图像缺乏高光细节的技术问题，本申请提供了一种图像处理方法。该方法为了提升目标视角下的拍摄图像中的高光效果，可将多组偏振图像作为NeRF模型的训练数据，以使NeRF模型可以通过多个训练迭代过程学习到偏振图像的不同偏振参数对应的不同高光效果。例如，NeRF模型可以学习到不同偏振参数对偏振图像中各像素点的像素颜色的影响程度等。进而，在使用训练完成的NeRF模型调整拍摄图像的视角时，可向该NeRF模型输入拍摄图像以及目标偏振参数，以使得最终生成的目标视角下的拍摄图像可呈现出目标偏振参数对应的高光效果。通过此方式，可有效提高基于NeRF模型生成的目标视角下的拍摄图像的高光效果，提升目标视角下的拍摄图像的图像质感。Therefore, in order to solve the technical problem that the captured images under the target viewing angle generated by the aforementioned NeRF model lack highlight details, the present application provides an image processing method. In order to improve the highlight effect in the captured images under the target viewing angle, the method can use multiple groups of polarized images as training data for the NeRF model, so that the NeRF model can learn different highlight effects corresponding to different polarization parameters of the polarized images through multiple training iterations. For example, the NeRF model can learn the degree of influence of different polarization parameters on the pixel color of each pixel in the polarized image. Furthermore, when the trained NeRF model is used to adjust the viewing angle of the captured image, the captured image and the target polarization parameters can be input into the NeRF model, so that the captured image under the target viewing angle finally generated can present the highlight effect corresponding to the target polarization parameters. In this way, the highlight effect of the captured image under the target viewing angle generated based on the NeRF model can be effectively improved, and the image texture of the captured image under the target viewing angle can be improved.
具体地，图2示出了应用本申请提供的NeRF模型生成具有高光效果的目标视角下的拍摄图像的过程示意图。Specifically, FIG2 shows a schematic diagram of a process of applying the NeRF model provided in the present application to generate a captured image under a target viewing angle with a highlight effect.
参见图2，为使NeRF模型能够生成具有高光效果的目标视角下的拍摄图像，可以将偏振参数拼接至NeRF模型的输入数据中，使得NeRF模型的输入数据扩展为位置参数、方向参数以及偏振参数。对应地，在使用训练完成的NeRF模型调整拍摄图像的视角时，NeRF模型的输入数据可为拍摄图像中的像素点对应的六维数据（x，y，z，θ，Φ，p）。其中，位置参数(x，y，z)可以为该拍摄图像中的各个像素点对应的空间点的空间位置坐标、（θ，Φ）可以为用于表征调整后的目标视角的目标方向参数、p可以为对应于想要呈现的高光效果的目标偏振参数。Referring to FIG. 2 , in order to enable the NeRF model to generate a captured image with a highlight effect at a target viewing angle, the polarization parameters can be spliced into the input data of the NeRF model, so that the input data of the NeRF model is expanded into position parameters, direction parameters, and polarization parameters. Correspondingly, when the trained NeRF model is used to adjust the viewing angle of the captured image, the input data of the NeRF model can be six-dimensional data (x, y, z, θ, Φ, p) corresponding to the pixel points in the captured image. Among them, the position parameters (x, y, z) can be the spatial position coordinates of the spatial points corresponding to each pixel point in the captured image, (θ, Φ) can be the target direction parameters used to characterize the adjusted target viewing angle, and p can be the target polarization parameters corresponding to the highlight effect that you want to present.
进而，NeRF模型中的位姿多层感知器(positional multilayer perceptron，positional MLP)可基于像素点对应的空间点的空间位置坐标(x，y，z)得到该空间点对应的若干个采样点的体密度值。NeRF模型中的方向多层感知器(directional multilayerperceptron，directional MLP)可基于目标方向参数(θ，Φ)、目标偏振参数p、以及positional MLP输出的采样点的特征向量，得到该采样点的色彩值。其中，前述空间点对应的若干个采样点可包括从相机到该空间点的光线上选取的若干个采样点。如此，基于NeRF模型中的体渲染模型对该空间点对应的若干个采样点的体密度值和色彩值进行积分，便可以得到该空间点在目标视角下对应的像素点的像素颜色。Furthermore, the positional multilayer perceptron (positional MLP) in the NeRF model can obtain the volume density values of several sampling points corresponding to the spatial point based on the spatial position coordinates (x, y, z) of the spatial point corresponding to the pixel point. The directional multilayer perceptron (directional MLP) in the NeRF model can obtain the color value of the sampling point based on the target direction parameter (θ, Φ), the target polarization parameter p, and the feature vector of the sampling point output by the positional MLP. Among them, the several sampling points corresponding to the aforementioned spatial point may include several sampling points selected on the light from the camera to the spatial point. In this way, based on the volume rendering model in the NeRF model, the volume density values and color values of several sampling points corresponding to the spatial point are integrated, and the pixel color of the pixel point corresponding to the spatial point under the target viewing angle can be obtained.
基于前述内容可以理解，根据本申请提供的NeRF模型得到的像素点的像素颜色受到了目标偏振参数的影响。进而，基于各个像素点的像素颜色生成的目标视角下的拍摄图像可呈现出该目标偏振参数对应的高光效果。例如，向图2所示的本申请提供的NeRF模型输入拍摄图像及目标偏振参数，输出的目标视角下的拍摄图像可以呈现出该目标偏振参数对应的高光效果。因此，通过本申请提供的NeRF模型及图像处理方法，可有效弥补NeRF模型生成的目标视角下的拍摄图像缺乏高光细节的缺陷，提高基于NeRF模型生成的目标视角下的拍摄图像的图像质感。Based on the foregoing, it can be understood that the pixel color of the pixel point obtained according to the NeRF model provided by the present application is affected by the target polarization parameter. Furthermore, the captured image under the target perspective generated based on the pixel color of each pixel point can present the highlight effect corresponding to the target polarization parameter. For example, the captured image and the target polarization parameter are input to the NeRF model provided by the present application shown in Figure 2, and the captured image under the output target perspective can present the highlight effect corresponding to the target polarization parameter. Therefore, the NeRF model and image processing method provided by the present application can effectively compensate for the defect of lack of highlight details in the captured image under the target perspective generated by the NeRF model, and improve the image texture of the captured image under the target perspective generated based on the NeRF model.
基于前述对本申请提供的图像处理方法的原理性阐释，下面，将结合附图及不同实施例对本申请提供的图像处理方法在不同应用场景的具体实现过程进行详细说明。Based on the above explanation of the principles of the image processing method provided by the present application, the specific implementation process of the image processing method provided by the present application in different application scenarios will be described in detail below in combination with the accompanying drawings and different embodiments.
还需声明，本申请实施例中对方法、流程中的步骤进行编号是为了便于引用，而不是限定先后顺序，各步骤之间如有先后顺序，当以文字说明为准。It should also be stated that the steps in the methods and processes in the embodiments of the present application are numbered for ease of reference, rather than to limit the order of precedence. If there is an order between the steps, the text description shall prevail.
实施例Example
本申请实施例将详细介绍在用户需要调整拍摄照片视角的应用场景下，本申请实施例提供的图像处理方法的具体实现过程。The embodiments of the present application will introduce in detail the specific implementation process of the image processing method provided by the embodiments of the present application in an application scenario where the user needs to adjust the viewing angle of the taken photo.
首先，图3示出了一种本申请实施例提供的图像处理方法的流程示意图，下面将结合图3对电子设备应用本申请实施例提供的图像处理方法的具体过程进行详细说明。First, FIG3 shows a flow chart of an image processing method provided in an embodiment of the present application. The specific process of applying the image processing method provided in an embodiment of the present application to an electronic device will be described in detail below in conjunction with FIG3 .
可以理解，实施图3所述流程的各个步骤的电子设备可以是前述手机100。为了便于描述，以下在介绍各个步骤时，均以手机100为执行主体，下文将不再对执行主体进行赘述。It is understood that the electronic device implementing each step of the process shown in FIG3 may be the aforementioned mobile phone 100. For ease of description, the mobile phone 100 is used as the execution subject when introducing each step below, and the execution subject will not be described in detail below.
具体地，本申请实施例提供的图像处理方法的实施流程可以包括以下步骤：Specifically, the implementation process of the image processing method provided in the embodiment of the present application may include the following steps:
300：响应于第一用户操作获取拍摄图像。300: Acquire a captured image in response to a first user operation.
在一种示例方式中，拍摄图像可以由手机100基于用户的拍摄操作实时获取。在该示例方式中，第一用户操作可以是用户基于手机100中的拍摄应用实施的拍摄操作，例如，该拍摄操作可以是用户对拍摄应用中的快门控件的点击操作、可实现拍摄的便捷操作等。前述便捷操作可以是用户对手机100的音量键、电源键等实体按键或虚拟按键的点击操作、口述指定口令的语音操作、实施指定手势动作的手势操作等。在此，本申请不对获取用户拍摄图像的第一用户操作进行限制性说明。In an example, the captured image can be acquired in real time by the mobile phone 100 based on the user's shooting operation. In this example, the first user operation can be a shooting operation performed by the user based on the shooting application in the mobile phone 100. For example, the shooting operation can be a user's click operation on the shutter control in the shooting application, a convenient operation that can achieve shooting, etc. The aforementioned convenient operation can be a user's click operation on a physical button or virtual button such as a volume button or a power button of the mobile phone 100, a voice operation of orally specifying a command, a gesture operation of performing a specified gesture action, etc. Here, this application does not provide a restrictive description of the first user operation of acquiring the user's captured image.
在该示例方式中，本申请提供的图像处理方法可以集成在前述拍摄应用中，以在用户基于该拍摄应用完成拍摄操作后，直接在该拍摄应用中基于下述步骤301至步骤304实现对拍摄图像的自动调节或向用户提供手动调节方案。在此，拍摄应用可以是手机100中的系统应用，例如相机应用，也可以是手机100中安装的第三方应用或手机100可使用的小程序等。本申请不对集成该图像处理方法的拍摄应用做限制性说明。In this example, the image processing method provided by the present application can be integrated into the aforementioned shooting application, so that after the user completes the shooting operation based on the shooting application, the automatic adjustment of the shot image can be directly implemented in the shooting application based on the following steps 301 to 304, or a manual adjustment solution can be provided to the user. Here, the shooting application can be a system application in the mobile phone 100, such as a camera application, or a third-party application installed in the mobile phone 100 or a small program that can be used by the mobile phone 100. This application does not make a restrictive description of the shooting application that integrates the image processing method.
在另一种示例方式中，拍摄图像可以是已经存储在手机100中的图像。在该示例方式中，第一用户操作可以是用户对存储在手机100中的拍摄图像的选择操作。在此，本申请提供的图像处理方法可以集成在图像处理应用中，用户可以基于图像处理应用提供的图像选择功能实施对拍摄图像的选择操作。进而，可在该图像处理应用中基于下述步骤301至步骤304实现对拍摄图像的自动调节或向用户提供手动调节方案。In another example, the captured image may be an image that has been stored in the mobile phone 100. In this example, the first user operation may be a user's selection operation of the captured image stored in the mobile phone 100. Here, the image processing method provided in the present application may be integrated in an image processing application, and the user may perform a selection operation on the captured image based on the image selection function provided by the image processing application. Furthermore, in the image processing application, automatic adjustment of the captured image may be implemented based on the following steps 301 to 304, or a manual adjustment solution may be provided to the user.
在此，图像处理应用可以是前述拍摄应用，即，拍摄应用可以对用户实时拍摄的拍摄图像进行即时优化，也可以对用户选择的存储在手机100中的拍摄图像进行后期优化。图像处理应用也可以是用于对存储在手机100中的拍摄图像进行后期优化的其他应用。具体地，图像处理应用可以是手机100的系统应用，例如图库应用（或称相册应用）。也可以是手机100中安装的第三方应用或手机100可使用的小程序等。本申请不对集成该图像处理方法的图像处理应用做限制性说明。Here, the image processing application may be the aforementioned shooting application, that is, the shooting application may perform instant optimization on the captured images captured by the user in real time, or may perform post-optimization on the captured images selected by the user and stored in the mobile phone 100. The image processing application may also be other applications for post-optimization of the captured images stored in the mobile phone 100. Specifically, the image processing application may be a system application of the mobile phone 100, such as a gallery application (or album application). It may also be a third-party application installed in the mobile phone 100 or a small program that can be used by the mobile phone 100. This application does not make a restrictive description of the image processing application that integrates the image processing method.
301：获取拍摄图像对应的目标方向参数及目标偏振参数。301: Obtain target direction parameters and target polarization parameters corresponding to the captured image.
示例性地，拍摄图像对应的目标方向参数和目标偏振参数可以由手机100预设，可以理解，用户也可以修改由手机100预设的目标方向参数和目标偏振参数。此外，拍摄图像对应的目标方向参数和目标偏振参数也可基于用户的输入确定。Exemplarily, the target direction parameter and target polarization parameter corresponding to the captured image may be preset by the mobile phone 100. It is understood that the user may also modify the target direction parameter and target polarization parameter preset by the mobile phone 100. In addition, the target direction parameter and target polarization parameter corresponding to the captured image may also be determined based on the user's input.
在一种示例方式中，手机100可以预设目标偏振参数和目标方向参数。例如，手机100可以预设目标偏振参数以及目标方向参数。以手机100基于前述拍摄应用对用户实时拍摄的拍摄图像进行即时优化为例，当用户基于前述步骤300拍摄了拍摄图像后，拍摄应用可以在向用户显示该拍摄图像前，先获取手机100中预设的目标偏振参数及目标方向参数，并根据该目标偏振参数及目标方向参数基于下述步骤302至步骤303生成具有高光细节的目标视角下的超分辨率拍摄图像。进而，可基于下述步骤304将该具有高光细节的目标视角下的超分辨率拍摄图像显示给用户。In an example manner, the mobile phone 100 may preset the target polarization parameters and the target direction parameters. For example, the mobile phone 100 may preset the target polarization parameters and the target direction parameters. Taking the mobile phone 100 as an example, based on the aforementioned shooting application, the instant optimization of the captured image captured by the user in real time is performed. After the user captures the captured image based on the aforementioned step 300, the shooting application may first obtain the preset target polarization parameters and target direction parameters in the mobile phone 100 before displaying the captured image to the user, and generate a super-resolution captured image under the target viewing angle with highlight details based on the target polarization parameters and target direction parameters based on the following steps 302 to 303. Furthermore, the super-resolution captured image under the target viewing angle with highlight details may be displayed to the user based on the following step 304.
在另一种示例方式中，手机100也可以仅预设目标偏振参数，并基于用户输入确定目标方向参数。例如，拍摄应用可以基于手机100预设的目标偏振参数对拍摄图像进行即时优化，以基于下述步骤302至步骤303生成具有高光细节的超分辨率拍摄图像。可以理解，此时，该具有高光细节的超分辨率拍摄图像的视角与原拍摄图像一致。进而，手机100可保存该具有高光细节的超分辨率拍摄图像，并且，可以基于用户输入的目标方向参数对该具有高光细节的超分辨率拍摄图像进行视角调整，以基于下述步骤302至步骤303生成具有高光细节的目标视角下的超分辨率拍摄图像。In another example, the mobile phone 100 may also only preset the target polarization parameters and determine the target direction parameters based on the user input. For example, the shooting application can instantly optimize the captured image based on the target polarization parameters preset by the mobile phone 100 to generate a super-resolution captured image with highlight details based on the following steps 302 to 303. It can be understood that at this time, the viewing angle of the super-resolution captured image with highlight details is consistent with the original captured image. Furthermore, the mobile phone 100 can save the super-resolution captured image with highlight details, and can adjust the viewing angle of the super-resolution captured image with highlight details based on the target direction parameters input by the user, so as to generate a super-resolution captured image at the target viewing angle with highlight details based on the following steps 302 to 303.
对应地，手机100也可以仅预设目标方向参数，并基于用户输入确定目标偏振参数。在此场景下，手机100生成具有高光细节的目标视角下的超分辨率拍摄图像的过程与前述内容实质相同，在此不做赘述。Correspondingly, the mobile phone 100 may also only preset the target direction parameter and determine the target polarization parameter based on the user input. In this scenario, the process of the mobile phone 100 generating a super-resolution image captured from a target perspective with high light details is substantially the same as the above content, and will not be repeated here.
在再一种示例方式中，手机100可基于用户输入确定目标方向参数和目标偏振参数。以手机100基于前述图像处理应用对存储在手机100中的拍摄图像进行后期优化为例，图4a示出了一种用户输入目标方向参数及目标偏振参数的效果示意图。In another example, the mobile phone 100 may determine the target direction parameter and the target polarization parameter based on the user input. Taking the mobile phone 100 performing post-optimization on the captured image stored in the mobile phone 100 based on the aforementioned image processing application as an example, FIG4a shows a schematic diagram of the effect of the user inputting the target direction parameter and the target polarization parameter.
示例性地，参见图4a，当用户基于图像处理应用提供的图像选择功能选择了拍摄图像400a后，图像处理应用可将拍摄图像400a显示在调节界面400中。进而，用户可根据调节界面400中的视角调节控件400b输入目标方向参数。例如，视角调节控件400b可为滑动条控件，滑动条不同的位置可对应不同的目标方向参数。因此，用户可通过滑动视角调节控件400b以进行视角调节。当用户滑动视角调节控件400b后，图像处理应用可获取此时滑动位置对应的目标方向参数，并根据该目标方向参数基于下述步骤302至步骤304将生成的目标视角的超分辨率拍摄图像400c显示在调节界面400中。Exemplarily, referring to FIG. 4a, after the user selects the captured image 400a based on the image selection function provided by the image processing application, the image processing application may display the captured image 400a in the adjustment interface 400. Furthermore, the user may input the target direction parameter according to the viewing angle adjustment control 400b in the adjustment interface 400. For example, the viewing angle adjustment control 400b may be a slider control, and different positions of the slider may correspond to different target direction parameters. Therefore, the user may adjust the viewing angle by sliding the viewing angle adjustment control 400b. After the user slides the viewing angle adjustment control 400b, the image processing application may obtain the target direction parameter corresponding to the sliding position at this time, and display the generated super-resolution captured image 400c of the target viewing angle in the adjustment interface 400 based on the target direction parameter based on the following steps 302 to 304.
进而，用户可继续根据调节界面400中的高光调节控件400d输入目标偏振参数。例如，高光调节控件400d可为滑动条控件，滑动条不同的位置可对应不同的目标偏振参数。因此，用户可通过滑动高光调节控件400d以对目标视角下的超分辨率拍摄图像400c进行高光调节。当用户滑动高光调节控件400d后，图像处理应用可获取此时滑动位置对应的目标偏振参数，并根据该目标偏振参数基于下述步骤302至步骤304将生成的具有高光细节的目标视角下的超分辨率拍摄图像400e显示在调节界面400中。该超分辨率拍摄图像400e可具有高光区域400f。Then, the user may continue to input the target polarization parameter according to the highlight adjustment control 400d in the adjustment interface 400. For example, the highlight adjustment control 400d may be a slider control, and different positions of the slider may correspond to different target polarization parameters. Therefore, the user may adjust the highlights of the super-resolution captured image 400c under the target viewing angle by sliding the highlight adjustment control 400d. When the user slides the highlight adjustment control 400d, the image processing application may obtain the target polarization parameter corresponding to the sliding position at this time, and display the generated super-resolution captured image 400e under the target viewing angle with highlight details in the adjustment interface 400 based on the target polarization parameter based on the following steps 302 to 304. The super-resolution captured image 400e may have a highlight area 400f.
此外，图4b示出了另一种用户输入目标方向参数及目标偏振参数的效果示意图。参见图4b，图像处理应用也可在用户滑动视角调节控件400b以及高光调节控件400d后，获取对应的目标偏振参数和目标方向参数，并基于下述步骤302至步骤304将生成的具有高光细节的目标视角下的超分辨率拍摄图像400e显示在调节界面400中。In addition, FIG4b shows another schematic diagram of the effect of the user inputting the target direction parameter and the target polarization parameter. Referring to FIG4b, the image processing application can also obtain the corresponding target polarization parameter and target direction parameter after the user slides the viewing angle adjustment control 400b and the highlight adjustment control 400d, and display the generated super-resolution captured image 400e with highlight details at the target viewing angle in the adjustment interface 400 based on the following steps 302 to 304.
此外，可以理解，用户也可以根据调节界面400先输入目标偏振参数以进行高光调节，再输入目标方向参数以进行视角调节。在此，本申请不对用户根据调节界面400进行的调节过程进行限制性说明。In addition, it is understood that the user can also first input the target polarization parameter to adjust the highlight according to the adjustment interface 400, and then input the target direction parameter to adjust the viewing angle. Here, the present application does not restrictively describe the adjustment process performed by the user according to the adjustment interface 400.
302：确定拍摄图像对应匹配的高分辨率参照图像。302: Determine a high-resolution reference image that matches the captured image.
示例性地，基于对前述图2的相关说明可知，基于本申请提供的NeRF模型可加强拍摄图像的高光细节。但由于NeRF模型并未考虑不同像素点之间的空间关系对像素颜色的影响，可能导致生成的目标视角下的拍摄图像除了缺乏高光细节以外，还可能缺乏边缘细节、纹理细节等。可以理解，缺乏边缘细节、纹理细节等将导致NeRF模型生成的目标视角下的拍摄图像分辨率较低，即，NeRF模型生成的目标视角下的拍摄图像可能为具有高光细节的目标视角下的低分辨率拍摄图像。因此，本申请提供的图像处理方法还可结合参考帧的超分辨率(reference-based super resolution，RefSR)模型，基于高分辨率参照图像补充NeRF模型输出的目标视角下的低分辨率拍摄图像中的边缘细节及纹理细节，以提高该图像的分辨率。在此，为叙述连贯性，基于RefSR模型提高图像的分辨率的具体内容将在下文对图7a的说明中具体阐述，在此不做赘述。Exemplarily, based on the relevant description of the aforementioned FIG. 2, it can be known that the NeRF model provided by the present application can enhance the highlight details of the captured image. However, since the NeRF model does not consider the influence of the spatial relationship between different pixels on the pixel color, the generated captured image under the target perspective may lack edge details, texture details, etc. in addition to the lack of highlight details. It can be understood that the lack of edge details, texture details, etc. will result in a low resolution of the captured image under the target perspective generated by the NeRF model, that is, the captured image under the target perspective generated by the NeRF model may be a low-resolution captured image under the target perspective with highlight details. Therefore, the image processing method provided by the present application can also be combined with a reference frame super-resolution (reference-based super resolution, RefSR) model to supplement the edge details and texture details in the low-resolution captured image under the target perspective output by the NeRF model based on the high-resolution reference image to improve the resolution of the image. Here, for the sake of narrative coherence, the specific content of improving the resolution of the image based on the RefSR model will be specifically explained in the following description of FIG. 7a, and will not be repeated here.
在此，手机100中可以存储一个或多个高分辨率参照图像，高分辨率参照图像可以是由内置了偏振镜头或外设了偏振滤镜的电子设备拍摄得到的偏振图像。不同高分辨率参照图像对应的偏振参数可以不同，例如，拍摄不同高分辨率参照图像所使用的偏振镜头或偏振滤镜可以具有不同的偏振参数。该偏振参数可以包括偏振方向等参数。拍摄高分辨率参照图像的电子设备可以是手机100，也可以是手机100以外的其他电子设备。Here, one or more high-resolution reference images may be stored in the mobile phone 100, and the high-resolution reference image may be a polarized image captured by an electronic device with a built-in polarization lens or an external polarization filter. The polarization parameters corresponding to different high-resolution reference images may be different. For example, the polarization lens or polarization filter used to capture different high-resolution reference images may have different polarization parameters. The polarization parameters may include parameters such as polarization direction. The electronic device that captures the high-resolution reference image may be the mobile phone 100, or it may be another electronic device other than the mobile phone 100.
示例性地，手机100确定的拍摄图像对应的高分辨率参照图像可以是，拍摄内容与该拍摄图像相同或相似的高分辨率参照图像。例如，手机100基于前述步骤300获取了拍摄图像后，可以对拍摄图像及高分辨率参照图像的拍摄场景和/或拍摄对象进行识别，进而，可以将拍摄场景和/或拍摄对象与拍摄图像相同或相似的高分辨率参照图像作为该拍摄图像对应的高分辨率参照图像。前述拍摄场景包括夜间场景、白天场景、蓝天场景、雨天场景、星空场景、自拍场景、弱光场景、强光场景等，前述拍摄对象包括人物、动物、花卉、树木、湖泊、山川等。本申请不对拍摄场景及拍摄对象做限制性说明。Exemplarily, the high-resolution reference image corresponding to the captured image determined by the mobile phone 100 may be a high-resolution reference image whose captured content is the same or similar to that of the captured image. For example, after the mobile phone 100 acquires the captured image based on the aforementioned step 300, it may identify the shooting scene and/or the shooting object of the captured image and the high-resolution reference image, and then, the high-resolution reference image whose shooting scene and/or the shooting object are the same or similar to that of the captured image may be used as the high-resolution reference image corresponding to the captured image. The aforementioned shooting scenes include night scenes, day scenes, blue sky scenes, rainy scenes, starry sky scenes, selfie scenes, low-light scenes, strong-light scenes, etc., and the aforementioned shooting objects include people, animals, flowers, trees, lakes, mountains, etc. This application does not make any restrictive descriptions of shooting scenes and shooting objects.
在此，若前述步骤300获取的拍摄图像的拍摄对象为人物，由于手机100拍摄的人物通常具有重复性，因此，可对该拍摄图像中的人物进行人脸识别，并将手机100中存储的拍摄对象同样为该人物的一个或多个高分辨率参照图像作为该拍摄图像对应的高分辨率参照图像。可以理解，基于拍摄的人物相同的高分辨率参照图像对拍摄图像进行优化，可有效提高优化效果。Here, if the subject of the captured image acquired in the aforementioned step 300 is a person, since the persons captured by the mobile phone 100 are usually repetitive, face recognition can be performed on the person in the captured image, and one or more high-resolution reference images of the same person stored in the mobile phone 100 are used as high-resolution reference images corresponding to the captured image. It can be understood that optimizing the captured image based on the same high-resolution reference images of the captured person can effectively improve the optimization effect.
303：基于拍摄图像、高分辨率参照图像、目标方向参数、目标偏振参数生成超分辨率拍摄图像。303: Generate a super-resolution captured image based on the captured image, the high-resolution reference image, the target direction parameter, and the target polarization parameter.
示例性地，对于手机100基于前述步骤301获取了拍摄图像的目标方向参数以及目标偏振参数的场景，手机100可以基于拍摄图像、高分辨率参照图像、目标方向参数及目标偏振参数生成具有高光细节的目标视角下的超分辨率拍摄图像。Exemplarily, for a scenario in which the mobile phone 100 obtains the target direction parameters and target polarization parameters of the captured image based on the aforementioned step 301, the mobile phone 100 can generate a super-resolution captured image with high-light details under the target perspective based on the captured image, the high-resolution reference image, the target direction parameters and the target polarization parameters.
可以理解，前述手机100获取了拍摄图像的目标方向参数以及目标偏振参数的场景，包括但不限于，前述步骤301中手机100预设了目标偏振参数以及目标方向参数的场景、前述步骤301中图4b所示的基于用户操作获取对应的目标偏振参数和目标方向参数的场景。It can be understood that the scenario in which the aforementioned mobile phone 100 obtains the target direction parameters and target polarization parameters of the captured image includes, but is not limited to, the scenario in which the mobile phone 100 presets the target polarization parameters and target direction parameters in the aforementioned step 301, and the scenario in which the corresponding target polarization parameters and target direction parameters are obtained based on user operations as shown in Figure 4b of the aforementioned step 301.
下面以前述图4b所示的场景为例，对基于拍摄图像400a得到具有高光细节的目标视角下的超分辨率拍摄图像400e的过程进行说明。Taking the scene shown in FIG. 4 b as an example, the process of obtaining a super-resolution captured image 400 e with highlight details at a target viewing angle based on the captured image 400 a is described below.
具体地，手机100可以将基于前述步骤302确定的高分辨率参照图像输入NeRF模型中。以手机100基于前述图像处理应用对拍摄图像进行优化为例，图像处理应用可将前述步骤302确定的高分辨率参照图像输入前述NeRF模型中，以得到基于NeRF模型而被退化的低分辨率参照图像。可以理解，高分辨率参照图像与低分辨率参照图像实质上是一张图像，低分辨率参照图像是高分辨率参照图像经过NeRF模型的图像处理过程而得到的退化结果。在此，为叙述连贯性，基于高分辨率参照图像得到低分辨率参照图像的具体过程将在下文对图7a的描述中详细说明，在此不做赘述。Specifically, the mobile phone 100 can input the high-resolution reference image determined based on the aforementioned step 302 into the NeRF model. Taking the example of the mobile phone 100 optimizing the captured image based on the aforementioned image processing application, the image processing application can input the high-resolution reference image determined in the aforementioned step 302 into the aforementioned NeRF model to obtain a low-resolution reference image that is degraded based on the NeRF model. It can be understood that the high-resolution reference image and the low-resolution reference image are essentially one image, and the low-resolution reference image is the degradation result obtained by the high-resolution reference image through the image processing process of the NeRF model. Here, for the sake of narrative coherence, the specific process of obtaining the low-resolution reference image based on the high-resolution reference image will be described in detail in the description of Figure 7a below, and will not be repeated here.
并且，图像处理应用可将基于前述步骤300获取的拍摄图像400a、基于前述步骤301获取的目标方向参数及目标偏振参数输入NeRF模型中，以得到具有高光细节的目标视角下的低分辨率拍摄图像。在此，为叙述连贯性，基于拍摄图像、目标方向参数及目标偏振参数得到具有高光细节的目标视角下的低分辨率拍摄图像的具体过程将在下文对图7c的描述中详细说明，在此不做赘述。Furthermore, the image processing application may input the captured image 400a obtained based on the aforementioned step 300, the target direction parameter and the target polarization parameter obtained based on the aforementioned step 301 into the NeRF model to obtain a low-resolution captured image under the target viewing angle with high light details. Here, for the sake of narrative coherence, the specific process of obtaining a low-resolution captured image under the target viewing angle with high light details based on the captured image, the target direction parameter and the target polarization parameter will be described in detail in the description of FIG. 7c below, and will not be repeated here.
进而，图像处理应用可将前述低分辨率参照图像、具有高光细节的目标视角下的低分辨率拍摄图像、高分辨率参照图像、偏振参数输入RefSR模型中，以基于高分辨率参照图像引入边缘细节、纹理细节等高频信息、基于低分辨率参照图像引入残差特征以进一步修复因退化过程而丢失或扭曲的高频信息。可以理解，基于前述内容可提高该具有高光细节的目标视角下的低分辨率拍摄图像的清晰度，生成具有高光细节的目标视角下的超分辨率拍摄图像。例如，基于前述方式可得到RefSR模型输出的具有高光细节的目标视角下的超分辨率拍摄图像400e。在此，为叙述连贯性，基于低分辨率参照图像、具有高光细节的目标视角下的低分辨率拍摄图像、高分辨率参照图像得到具有高光细节的目标视角下的超分辨率拍摄图像的具体过程将在下文对图7b的描述中详细说明，在此不做赘述。Furthermore, the image processing application may input the aforementioned low-resolution reference image, the low-resolution captured image at the target perspective with highlight details, the high-resolution reference image, and the polarization parameters into the RefSR model, so as to introduce high-frequency information such as edge details and texture details based on the high-resolution reference image, and introduce residual features based on the low-resolution reference image to further repair the high-frequency information lost or distorted due to the degradation process. It can be understood that based on the aforementioned content, the clarity of the low-resolution captured image at the target perspective with highlight details can be improved, and a super-resolution captured image at the target perspective with highlight details can be generated. For example, based on the aforementioned method, a super-resolution captured image 400e at the target perspective with highlight details output by the RefSR model can be obtained. Here, for the sake of narrative coherence, the specific process of obtaining a super-resolution captured image at the target perspective with highlight details based on the low-resolution reference image, the low-resolution captured image at the target perspective with highlight details, and the high-resolution reference image will be described in detail in the description of Figure 7b below, and will not be repeated here.
因此，基于上述内容，本申请提供的图像处理方法可基于NeRF模型对拍摄图像进行视角调整以及高光调整，基于RefSR模型进行超分辨率(super resolution，SR)重建，以得到分辨率更高、具有更多高频细节的超分辨率拍摄图像。在基于NeRF模型满足了用户调整拍摄图像的视角，生成目标视角下的拍摄图像的需求的同时，弥补了NeRF模型生成的目标视角下的拍摄图像缺乏高频细节的缺陷，得到了具有高光细节的目标视角下的超分辨率拍摄图像。例如，对于前述图4b所示的场景，基于前述内容可调整拍摄图像400a的视角及高光，得到具有高光细节的目标视角下的超分辨率拍摄图像400e。Therefore, based on the above content, the image processing method provided in the present application can adjust the perspective and highlight of the captured image based on the NeRF model, and perform super-resolution (SR) reconstruction based on the RefSR model to obtain a super-resolution captured image with higher resolution and more high-frequency details. While satisfying the user's need to adjust the perspective of the captured image and generate the captured image at the target perspective based on the NeRF model, it makes up for the defect that the captured image at the target perspective generated by the NeRF model lacks high-frequency details, and obtains a super-resolution captured image at the target perspective with highlight details. For example, for the scene shown in Figure 4b above, the perspective and highlight of the captured image 400a can be adjusted based on the above content to obtain a super-resolution captured image 400e at the target perspective with highlight details.
下面以前述图4a所示的场景为例，对基于拍摄图像400a得到具有高光细节的目标视角下的超分辨率拍摄图像400e的过程进行说明。Taking the scene shown in FIG. 4 a as an example, the process of obtaining a super-resolution captured image 400 e with highlight details at a target viewing angle based on the captured image 400 a is described below.
示例性地，对于前述图4a所示的场景，当图像处理应用先基于用户滑动视角调节控件400b的操作获取了对应的目标方向参数时，图像处理应用可以基于拍摄图像400a、高分辨率参照图像、目标方向参数生成目标视角下的超分辨率拍摄图像400c。当图像处理应用又基于用户滑动高光调节控件400d的操作获取了对应的目标偏振参数时，图像处理应用可以基于目标视角下的超分辨率拍摄图像400c、高分辨率参照图像、目标偏振参数生成具有高光细节的目标视角下的超分辨率拍摄图像400e。在此，由拍摄图像400a得到超分辨率拍摄图像400c，以及由超分辨率拍摄图像400c得到超分辨率拍摄图像400e的过程与前述基于拍摄图像400a得到超分辨率拍摄图像400e的过程实质相同，在此不做赘述。Exemplarily, for the scene shown in FIG. 4a above, when the image processing application first obtains the corresponding target direction parameter based on the user's operation of sliding the viewing angle adjustment control 400b, the image processing application can generate a super-resolution captured image 400c at the target viewing angle based on the captured image 400a, the high-resolution reference image, and the target direction parameter. When the image processing application obtains the corresponding target polarization parameter based on the user's operation of sliding the highlight adjustment control 400d, the image processing application can generate a super-resolution captured image 400e at the target viewing angle with highlight details based on the super-resolution captured image 400c at the target viewing angle, the high-resolution reference image, and the target polarization parameter. Here, the process of obtaining the super-resolution captured image 400c from the captured image 400a, and obtaining the super-resolution captured image 400e from the super-resolution captured image 400c is substantially the same as the process of obtaining the super-resolution captured image 400e based on the captured image 400a, and will not be repeated here.
304：显示所生成的目标视角下的超分辨率拍摄图像。304: Display the generated super-resolution image at the target viewing angle.
示例性地，手机100基于上述步骤300至步骤303得到的超分辨率拍摄图像包括：基于拍摄图像、高分辨率参照图像、目标方向参数及目标偏振参数生成的具有高光细节的目标视角下的超分辨率拍摄图像，例如，前述超分辨率拍摄图像400e；基于拍摄图像、高分辨率参照图像、目标方向参数生成的目标视角下的超分辨率拍摄图像，例如，前述超分辨率拍摄图像400c。在此，不对目标视角下的超分辨率拍摄图像的具体内容做限制性说明。Exemplarily, the super-resolution captured image obtained by the mobile phone 100 based on the above steps 300 to 303 includes: a super-resolution captured image with highlight details at a target viewing angle generated based on the captured image, the high-resolution reference image, the target direction parameter, and the target polarization parameter, for example, the super-resolution captured image 400e; a super-resolution captured image at a target viewing angle generated based on the captured image, the high-resolution reference image, and the target direction parameter, for example, the super-resolution captured image 400c. Here, no restrictive description is given to the specific content of the super-resolution captured image at the target viewing angle.
具体地，对于前述图4a所示的场景，图像处理应用可以将基于拍摄图像400a、高分辨率参照图像、目标方向参数生成的目标视角下的超分辨率拍摄图像400c显示在调节界面400中。可以将基于超分辨率拍摄图像400c、高分辨率参照图像、目标偏振参数生成的具有高光细节的目标视角下的超分辨率拍摄图像400e显示在调节界面400中。Specifically, for the scene shown in FIG. 4a above, the image processing application may display a super-resolution captured image 400c at a target viewing angle generated based on the captured image 400a, the high-resolution reference image, and the target direction parameter in the adjustment interface 400. A super-resolution captured image 400e at a target viewing angle with highlight details generated based on the super-resolution captured image 400c, the high-resolution reference image, and the target polarization parameter may be displayed in the adjustment interface 400.
具体地，对于前述图4b所示的场景，图像处理应用可以将基于拍摄图像400a、高分辨率参照图像、目标方向参数、目标偏振参数生成的具有高光细节的目标视角下的超分辨率拍摄图像400e显示在调节界面400中。Specifically, for the scene shown in the aforementioned FIG. 4b , the image processing application may display in the adjustment interface 400 a super-resolution captured image 400e at the target perspective with high light details generated based on the captured image 400a , the high-resolution reference image, the target direction parameters, and the target polarization parameters.
此外，参见图4c示出的本申请提供的图像处理方法的一种应用场景，以手机100基于前述图像处理应用对存储在手机100中的拍摄图像进行后期优化为例，图像处理应用可以提取拍摄图像中的部分图像，进而，对该部分图像进行视角和高光的调整。In addition, referring to FIG. 4c , an application scenario of the image processing method provided by the present application is shown. Taking the example of the mobile phone 100 performing post-optimization on the captured image stored in the mobile phone 100 based on the aforementioned image processing application, the image processing application can extract part of the image from the captured image, and then adjust the perspective and highlight of the part of the image.
例如，基于前述步骤300获取的拍摄图像401a中可能包括人像401b及背景401c。进而，图像处理应用可以基于用户操作提取拍摄图像401a中的人像401b，并将该人像401b显示在调节界面400中。在此，提取拍摄图像401a中的人像401b的用户操作可以是对人像401b进行单指或多指操作，例如，单指长按人像401b、双指点击人像401b等。还可以是对调节界面400中的抠图控件（图中未示出）的点击操作等。在此，本申请不对提取人像401b的用户操作做限制性说明。此外，图像处理应用基于用户操作提取的部分图像还可以是人像401b中的人脸部分等，本申请不对基于用户操作提取的部分图像的具体内容做限制性说明。For example, the captured image 401a obtained based on the aforementioned step 300 may include a portrait 401b and a background 401c. Furthermore, the image processing application may extract the portrait 401b in the captured image 401a based on the user operation, and display the portrait 401b in the adjustment interface 400. Here, the user operation of extracting the portrait 401b in the captured image 401a may be a single-finger or multi-finger operation on the portrait 401b, for example, long pressing the portrait 401b with one finger, clicking the portrait 401b with two fingers, etc. It may also be a click operation on the cutout control (not shown in the figure) in the adjustment interface 400, etc. Here, the present application does not make a restrictive description on the user operation of extracting the portrait 401b. In addition, the partial image extracted by the image processing application based on the user operation may also be the face part of the portrait 401b, etc., and the present application does not make a restrictive description on the specific content of the partial image extracted based on the user operation.
进而，图像处理应用可以基于前述步骤301至步骤303对人像401b进行高光调节及视角调节，生成具有高光区域401c的目标视角下的超分辨率拍摄图像401d，并将超分辨率拍摄图像401d显示在调节界面400中。Furthermore, the image processing application can perform highlight adjustment and viewing angle adjustment on the portrait 401b based on the aforementioned steps 301 to 303, generate a super-resolution captured image 401d at a target viewing angle with a highlight area 401c, and display the super-resolution captured image 401d in the adjustment interface 400.
在此，用户可以单独保存该超分辨率拍摄图像401d，示例性地，用户可将超分辨率拍摄图像401d制作为头像、表情包、壁纸、将其与其他图像进行组合等。本申请不对用户保存超分辨率拍摄图像401d后的具体应用场景做限制性说明。可以理解，用户也可以将该超分辨率拍摄图像401d回贴至拍摄图像401a中。Here, the user can save the super-resolution captured image 401d separately. For example, the user can make the super-resolution captured image 401d into an avatar, an emoticon pack, a wallpaper, or combine it with other images. This application does not limit the specific application scenarios after the user saves the super-resolution captured image 401d. It is understood that the user can also paste the super-resolution captured image 401d back to the captured image 401a.
实施例Example
本申请实施例将详细介绍在用户需要仅需要提升拍摄照片的质感，而不需调整视角的应用场景下，本申请实施例提供的图像处理方法的具体实现过程。The embodiments of the present application will introduce in detail the specific implementation process of the image processing method provided by the embodiments of the present application in an application scenario where the user only needs to improve the texture of the taken photos without adjusting the viewing angle.
首先，图5示出了另一种本申请实施例提供的图像处理方法的流程示意图，下面将结合图5对电子设备应用本申请实施例提供的图像处理方法的具体过程进行详细说明。First, FIG5 shows a flow chart of another image processing method provided in an embodiment of the present application. The specific process of applying the image processing method provided in an embodiment of the present application to an electronic device will be described in detail below in conjunction with FIG5 .
可以理解，实施图5所述流程的各个步骤的电子设备可以是前述手机100。为了便于描述，以下在介绍各个步骤时，均以手机100为执行主体，下文将不再对执行主体进行赘述。It is understood that the electronic device implementing each step of the process described in Figure 5 can be the aforementioned mobile phone 100. For ease of description, when introducing each step below, the mobile phone 100 is used as the execution subject, and the execution subject will not be repeated below.
具体地，本申请实施例提供的图像处理方法的实施流程可以包括以下步骤：Specifically, the implementation process of the image processing method provided in the embodiment of the present application may include the following steps:
500：响应于第一用户操作获取拍摄图像。500: Acquire a captured image in response to a first user operation.
在此，手机100基于第一用户操作获取拍摄图像的具体过程可参见前述步骤300中的具体说明，在此不做赘述。Here, the specific process of the mobile phone 100 acquiring the captured image based on the first user operation can refer to the specific description in the aforementioned step 300, which will not be repeated here.
501：获取拍摄图像对应的目标偏振参数。501: Obtain target polarization parameters corresponding to the captured image.
在一种示例方式中，手机100可以预设目标偏振参数。例如，以手机100基于前述拍摄应用对用户实时拍摄的拍摄图像进行即时优化为例，当用户基于前述步骤500拍摄了拍摄图像后，拍摄应用可以在向用户显示该拍摄图像前，先获取手机100中预设的目标偏振参数及目标方向参数，并根据该目标偏振参数基于下述步骤502至步骤503生成具有高光细节的超分辨率拍摄图像。进而，可基于下述步骤504将该具有高光细节的超分辨率拍摄图像显示给用户。可以理解，由于本实施例并未对拍摄图像的拍摄视角进行调整，因此，生成的超分辨率拍摄图像可以为具有高光细节的原始视角下的超分辨率拍摄图像。在此，原始视角包括基于步骤500获取的拍摄图像的拍摄视角。同样地，原始视角包括拍摄图像的方向参数对应的拍摄视角。In an exemplary manner, the mobile phone 100 may preset target polarization parameters. For example, taking the mobile phone 100 as an example of instantly optimizing the captured image captured by the user in real time based on the aforementioned shooting application, after the user captures the captured image based on the aforementioned step 500, the shooting application may first obtain the target polarization parameters and target direction parameters preset in the mobile phone 100 before displaying the captured image to the user, and generate a super-resolution captured image with highlight details based on the target polarization parameters based on the following steps 502 to 503. Furthermore, the super-resolution captured image with highlight details may be displayed to the user based on the following step 504. It can be understood that since the shooting angle of the captured image is not adjusted in this embodiment, the generated super-resolution captured image may be a super-resolution captured image under the original angle of view with highlight details. Here, the original angle of view includes the shooting angle of view of the captured image acquired based on step 500. Similarly, the original angle of view includes the shooting angle of view corresponding to the direction parameter of the captured image.
在另一种示例方式中，手机100可基于用户输入确定目标偏振参数。以手机100基于前述图像处理应用对存储在手机100中的拍摄图像进行后期优化为例，图6示出了一种用户输入目标偏振参数的效果示意图。In another example, the mobile phone 100 may determine the target polarization parameter based on user input. Taking the mobile phone 100 performing post-optimization on the captured image stored in the mobile phone 100 based on the aforementioned image processing application as an example, FIG6 shows a schematic diagram of the effect of the user inputting the target polarization parameter.
示例性地，参见图6，当用户基于图像处理应用提供的图像选择功能选择了拍摄图像600a后，图像处理应用可将拍摄图像600a显示在调节界面600中。进而，用户可继续根据调节界面600中的高光调节控件600b输入目标偏振参数。例如，高光调节控件600b可为滑动条控件，滑动条不同的位置可对应不同的目标偏振参数。因此，用户可通过滑动高光调节控件600b以对拍摄图像进行高光调节。当用户滑动高光调节控件600b后，图像处理应用可获取此时滑动位置对应的目标偏振参数，并根据该目标偏振参数基于下述步骤502至步骤504将生成的具有高光细节的超分辨率拍摄图像600c显示在调节界面600中。该超分辨率拍摄图像600c可具有高光区域600d。Exemplarily, referring to FIG. 6 , after the user selects the captured image 600a based on the image selection function provided by the image processing application, the image processing application may display the captured image 600a in the adjustment interface 600. Then, the user may continue to input the target polarization parameter according to the highlight adjustment control 600b in the adjustment interface 600. For example, the highlight adjustment control 600b may be a slider control, and different positions of the slider may correspond to different target polarization parameters. Therefore, the user may adjust the highlights of the captured image by sliding the highlight adjustment control 600b. After the user slides the highlight adjustment control 600b, the image processing application may obtain the target polarization parameter corresponding to the sliding position at this time, and display the generated super-resolution captured image 600c with highlight details in the adjustment interface 600 based on the target polarization parameter based on the following steps 502 to 504. The super-resolution captured image 600c may have a highlight area 600d.
502：确定拍摄图像对应匹配的高分辨率参照图像。502: Determine a high-resolution reference image that matches the captured image.
示例性地，基于对前述图2的相关说明可知，由于NeRF模型并未考虑不同像素点之间的空间关系对像素颜色的影响，可能导致生成的目标视角下的拍摄图像除了缺乏高光细节以外，还可能缺乏边缘细节、纹理细节等。可以理解，缺乏边缘细节、纹理细节等将导致NeRF模型生成的目标视角下的拍摄图像分辨率较低，即，基于本申请实施例提供的NeRF模型生成的具有高光效果的图像可能为低分辨率图像。因此，本申请提供的图像处理方法还可结合RefSR模型，基于高分辨率参照图像补充NeRF模型输出的具有高光效果的低分辨率图像中的边缘细节及纹理细节，提高该图像的分辨率。在此，为叙述连贯性，基于RefSR模型提高图像的分辨率的具体内容将在下文对图7c的说明中具体阐述，在此不做赘述。Exemplarily, based on the relevant description of the aforementioned FIG. 2, it can be known that since the NeRF model does not consider the influence of the spatial relationship between different pixel points on the pixel color, the generated captured image under the target viewing angle may lack edge details, texture details, etc. in addition to the lack of highlight details. It can be understood that the lack of edge details, texture details, etc. will result in a lower resolution of the captured image under the target viewing angle generated by the NeRF model, that is, the image with highlight effect generated by the NeRF model provided in the embodiment of the present application may be a low-resolution image. Therefore, the image processing method provided in the present application can also be combined with the RefSR model to supplement the edge details and texture details in the low-resolution image with highlight effect output by the NeRF model based on the high-resolution reference image, thereby improving the resolution of the image. Here, for the sake of narrative coherence, the specific content of improving the resolution of the image based on the RefSR model will be specifically explained in the description of FIG. 7c below, and will not be repeated here.
在此，手机100确定拍摄图像对应的高分辨率参照图像的具体过程可参见前述步骤302中的具体说明，在此不做赘述。Here, the specific process of the mobile phone 100 determining the high-resolution reference image corresponding to the captured image can be referred to the specific description in the aforementioned step 302, which will not be repeated here.
503：基于拍摄图像、高分辨率参照图像、目标偏振参数生成超分辨率拍摄图像。503: Generate a super-resolution captured image based on the captured image, the high-resolution reference image, and the target polarization parameter.
示例性地，对于手机100基于前述步骤501获取了拍摄图像的目标偏振参数的场景，手机100可以基于拍摄图像、高分辨率参照图像、目标偏振参数生成具有高光细节的超分辨率拍摄图像。该具有高光细节的超分辨率拍摄图像例如可以是图6中的具有高光区域600d的超分辨率拍摄图像600c。Exemplarily, for a scene in which the mobile phone 100 obtains the target polarization parameters of the captured image based on the aforementioned step 501, the mobile phone 100 can generate a super-resolution captured image with highlight details based on the captured image, the high-resolution reference image, and the target polarization parameters. The super-resolution captured image with highlight details can be, for example, the super-resolution captured image 600c with the highlight area 600d in FIG. 6 .
具体地，手机100可以基于拍摄图像、高分辨率参照图像、目标偏振参数生成具有高光细节的超分辨率拍摄图像的过程，与前述步骤303中手机100基于拍摄图像、高分辨率参照图像、目标方向参数及目标偏振参数生成具有高光细节的目标视角下的超分辨率拍摄图像的过程实质相同，在此不做赘述。Specifically, the process in which the mobile phone 100 generates a super-resolution captured image with highlight details based on the captured image, the high-resolution reference image, and the target polarization parameters is essentially the same as the process in which the mobile phone 100 generates a super-resolution captured image with highlight details under the target perspective based on the captured image, the high-resolution reference image, the target direction parameters and the target polarization parameters in the aforementioned step 303, and will not be repeated here.
可以理解，在本实施例中，由于用户不需要对拍摄视角进行调整，因此，输入NeRF模型中的方向参数可以是拍摄图像的方向参数。具体地，可以是表征拍摄图像的拍摄视角的方向参数（作为第一方向参数的一种示例）。进而，根据拍摄图像及目标偏振参数通过NeRF模型可得到具有高光细节的原始视角下的低分辨率拍摄图像。前述原始视角可以是拍摄图像的拍摄视角。进一步地，根据拍摄图像、目标偏振参数、高分辨率参照图像通过RefSR模型可得到具有高光细节的原始视角下的超分辨率拍摄图像。It can be understood that in this embodiment, since the user does not need to adjust the shooting angle, the direction parameter input into the NeRF model can be a direction parameter of the captured image. Specifically, it can be a direction parameter that characterizes the shooting angle of the captured image (as an example of a first direction parameter). Furthermore, based on the captured image and the target polarization parameter, a low-resolution captured image with highlight details at the original perspective can be obtained through the NeRF model. The aforementioned original perspective can be the shooting perspective of the captured image. Furthermore, based on the captured image, the target polarization parameter, and the high-resolution reference image, a super-resolution captured image with highlight details at the original perspective can be obtained through the RefSR model.
504：显示超分辨率拍摄图像。504: Display the super-resolution captured image.
示例性地，手机100基于上述步骤500至步骤503得到的超分辨率拍摄图像包括，基于拍摄图像、高分辨率参照图像、目标偏振参数生成的具有高光细节的原始视角下的超分辨率拍摄图像，例如，前述具有高光区域600d的超分辨率拍摄图像600c。在此，本申请不对超分辨率拍摄图像的具体内容做限制性说明。Exemplarily, the super-resolution captured image obtained by the mobile phone 100 based on the above steps 500 to 503 includes a super-resolution captured image with highlight details at the original viewing angle generated based on the captured image, the high-resolution reference image, and the target polarization parameters, for example, the super-resolution captured image 600c with the highlight area 600d mentioned above. Here, this application does not make a restrictive description on the specific content of the super-resolution captured image.
具体地，对于前述图6所示的场景，图像处理应用可以将基于拍摄图像600a、高分辨率参照图像、目标偏振参数生成的超分辨率拍摄图像600c显示在调节界面600中。Specifically, for the scene shown in FIG. 6 , the image processing application may display the super-resolution captured image 600 c generated based on the captured image 600 a , the high-resolution reference image, and the target polarization parameters in the adjustment interface 600 .
在此，本申请实施例提供的NeRF模型强化了拍摄图像600a的高光区域600d，此外，本申请实施例提供的RefSR模型在保留高光区域600d的同时，增加了超分辨率拍摄图像600c的纹理细节、边缘细节等。可以理解，超分辨率拍摄图像600c可以是提高了高光细节、纹理细节、边缘细节后的拍摄图像600a。即，基于前述步骤500至步骤504，有效提高了拍摄图像600a的高光效果及清晰度。Here, the NeRF model provided in the embodiment of the present application strengthens the highlight area 600d of the captured image 600a. In addition, the RefSR model provided in the embodiment of the present application retains the highlight area 600d while increasing the texture details, edge details, etc. of the super-resolution captured image 600c. It can be understood that the super-resolution captured image 600c can be the captured image 600a with the highlight details, texture details, and edge details increased. That is, based on the aforementioned steps 500 to 504, the highlight effect and clarity of the captured image 600a are effectively improved.
下面，结合附图对本申请提供的图像处理方法能够基于NeRF模型以及RefSR模型生成具有高光细节的目标视角下的超分辨率拍摄图像的具体过程进行详细说明。Below, in conjunction with the accompanying drawings, the specific process by which the image processing method provided by the present application can generate a super-resolution image captured under a target perspective with highlight details based on the NeRF model and the RefSR model is described in detail.
首先，对本申请提供的图像处理方法中的NeRF模型以及RefSR模型的训练过程进行详细说明。First, the training process of the NeRF model and the RefSR model in the image processing method provided in this application is described in detail.
示例性地，图7a示出了一种本申请提供的NeRF模型的训练过程示意图。Exemplarily, FIG7a shows a schematic diagram of a training process of a NeRF model provided in the present application.
参见图7a，NeRF模型的训练数据可以为多组高分辨率参照图像（作为前述多组偏振图像的一种示例）。其中，各组高分辨率参照图像中可以包括以多个视角拍摄相同对象的多个偏振图像，前述偏振图像由设置了偏振滤镜或内置了偏振镜头的相机拍摄得到。不同的拍摄视角可对应不同的方向参数，不同的偏振滤镜或偏振镜头可对应不同的偏振参数。Referring to FIG. 7a, the training data of the NeRF model may be a plurality of sets of high-resolution reference images (as an example of the aforementioned plurality of sets of polarization images). Each set of high-resolution reference images may include a plurality of polarization images of the same object taken at multiple viewing angles, and the aforementioned polarization images are taken by a camera with a polarization filter or a built-in polarization lens. Different shooting viewing angles may correspond to different directional parameters, and different polarization filters or polarization lenses may correspond to different polarization parameters.
NeRF模型的输入数据可包括各个高分辨率参照图像的各个像素点的位置参数(x，y，z)、表征该高分辨率参照图像的拍摄视角的方向参数(θ，Φ)、表征拍摄该高分辨率参照图像的偏振滤镜或偏振镜头的偏振方向的偏振参数p。其中，前述像素点的位置参数(x，y，z)可以是该像素点对应的空间点在世界坐标系下的三维位置坐标。θ可以是该高分辨率参照图像的拍摄视角对应的相机位姿，Φ可以是该拍摄视角对应的相机内参。p可以是拍摄该高分辨率参照图像的偏振滤镜或偏振镜头的偏振方向。The input data of the NeRF model may include the position parameters (x, y, z) of each pixel of each high-resolution reference image, the direction parameters (θ, Φ) representing the shooting angle of view of the high-resolution reference image, and the polarization parameter p representing the polarization direction of the polarization filter or polarization lens used to shoot the high-resolution reference image. Among them, the position parameters (x, y, z) of the aforementioned pixel points may be the three-dimensional position coordinates of the spatial point corresponding to the pixel point in the world coordinate system. θ may be the camera pose corresponding to the shooting angle of view of the high-resolution reference image, and Φ may be the camera internal parameter corresponding to the shooting angle of view. p may be the polarization direction of the polarization filter or polarization lens used to shoot the high-resolution reference image.
进而，NeRF模型中的positional MLP可基于像素点对应的空间点的空间位置坐标(x，y，z)得到该空间点对应的若干个采样点的体密度值，以及各个采样点的特征向量。NeRF模型中的directional MLP可基于该高分辨率参照图像的方向参数(θ，Φ)、该高分辨率参照图像的偏振参数p、以及positional MLP输出的采样点的特征向量，得到该采样点的色彩值。其中，前述空间点对应的若干个采样点可包括从相机到该空间点的光线上选取的若干个采样点。Furthermore, the positional MLP in the NeRF model can obtain the volume density values of several sampling points corresponding to the spatial point based on the spatial position coordinates (x, y, z) of the spatial point corresponding to the pixel point, as well as the feature vector of each sampling point. The directional MLP in the NeRF model can obtain the color value of the sampling point based on the directional parameters (θ, Φ) of the high-resolution reference image, the polarization parameter p of the high-resolution reference image, and the feature vector of the sampling point output by the positional MLP. Among them, the several sampling points corresponding to the aforementioned spatial point may include several sampling points selected on the light from the camera to the spatial point.
进而，可基于NeRF模型中的体渲染模型对该空间点对应的若干个采样点的体密度值和色彩值进行积分，以得到NeRF模型输出的该空间点在该高分辨率参照图像的拍摄视角下对应的像素点的像素颜色。可以理解，基于前述方式得到的像素点的像素颜色受到了高分辨率参照图像的偏振参数的影响，因此，NeRF模型基于各个像素点的像素颜色输出的图像可呈现出该偏振参数对应的高光效果。Furthermore, the volume density values and color values of several sampling points corresponding to the spatial point can be integrated based on the volume rendering model in the NeRF model to obtain the pixel color of the pixel corresponding to the spatial point output by the NeRF model under the shooting angle of the high-resolution reference image. It can be understood that the pixel color of the pixel obtained based on the above method is affected by the polarization parameter of the high-resolution reference image. Therefore, the image output by the NeRF model based on the pixel color of each pixel can present a highlight effect corresponding to the polarization parameter.
可以理解，由于NeRF模型对于各个像素点的像素颜色的确定过程是相互独立的，并未考虑不同像素点之间的空间关系对像素颜色的影响，因此，还可能导致NeRF模型输出的结果缺乏边缘细节、纹理细节等高频信息，因此，高分辨率参照图像经过该NeRF模型可能会被退化为低分辨率参照图像。即，高分辨率参照图像经过NeRF模型可输出保留了高光信息的退化结果，例如，可输出具有高光效果的低分辨率参照图像。It can be understood that since the NeRF model determines the pixel color of each pixel independently, and does not consider the impact of the spatial relationship between different pixels on the pixel color, the result output by the NeRF model may lack high-frequency information such as edge details and texture details. Therefore, the high-resolution reference image may be degraded to a low-resolution reference image after passing through the NeRF model. That is, the high-resolution reference image can output a degraded result that retains the highlight information after passing through the NeRF model, for example, a low-resolution reference image with a highlight effect can be output.
进而，可基于损失函数计算NeRF模型得到像素颜色与该高分辨率参照图像中的实际像素颜色之间的损失，并可以通过梯度下降法等方式调整NeRF模型中的参数，经多次训练迭代将该损失收敛至满意水平，以此提高NeRF模型最终确定的像素颜色的准确性。Furthermore, the loss between the pixel color obtained by the NeRF model and the actual pixel color in the high-resolution reference image can be calculated based on the loss function, and the parameters in the NeRF model can be adjusted by gradient descent method and other methods. After multiple training iterations, the loss is converged to a satisfactory level, thereby improving the accuracy of the pixel color finally determined by the NeRF model.
可以理解，NeRF模型确定的像素颜色的准确性越高，代表其生成的低分辨率参照图像的高光效果与其对应的偏振参数可呈现的高光效果越相似。相应地，在应用该NeRF模型时，可基于输入的偏振参数使拍摄图像较为准确的呈现出与输入的偏振参数对应的高光效果。It can be understood that the higher the accuracy of the pixel color determined by the NeRF model, the more similar the highlight effect of the low-resolution reference image generated by it is to the highlight effect that can be presented by the corresponding polarization parameters. Accordingly, when the NeRF model is applied, the captured image can more accurately present the highlight effect corresponding to the input polarization parameters based on the input polarization parameters.
基于前述内容可知，NeRF模型输出的图像为低分辨率图像，因此，为进一步提高NeRF模型输出的图像的分辨率，图7b示出了一种本申请提供的RefSR模型的训练过程示意图。Based on the foregoing, it can be seen that the image output by the NeRF model is a low-resolution image. Therefore, in order to further improve the resolution of the image output by the NeRF model, FIG7b shows a schematic diagram of the training process of the RefSR model provided in the present application.
参见图7b，RefSR模型的训练数据可以包括高分辨率参照图像、低分辨率参照图像、新视角下的低分辨率参照图像、以及偏振参数。在此，低分辨率参照图像可以是高分辨率参照图像输入训练好的NeRF模型得到的输出结果。新视角下的低分辨率参照图像可以是高分辨率参照图像以及与该图像不同的方向参数输入训练好的NeRF模型得到的输出结果。偏振参数可以是该高分辨率参照图像的偏振参数。可以理解，前述低分辨率参照图像以及新视角下的低分辨率参照图像均具有高光细节。Referring to FIG. 7b, the training data of the RefSR model may include a high-resolution reference image, a low-resolution reference image, a low-resolution reference image under a new perspective, and polarization parameters. Here, the low-resolution reference image may be the output result obtained by inputting a high-resolution reference image into a trained NeRF model. The low-resolution reference image under a new perspective may be the output result obtained by inputting a high-resolution reference image and a trained NeRF model with directional parameters different from the image. The polarization parameter may be the polarization parameter of the high-resolution reference image. It can be understood that the aforementioned low-resolution reference image and the low-resolution reference image under a new perspective both have highlight details.
具体地，一方面，RefSR模型可基于低分辨率参照图像和新视角下的低分辨率参照图像进行退化建模。以此学习低分辨率参照图像和新视角下的低分辨率参照图像的退化过程，以得到有助于修复因退化过程而丢失或扭曲的高频信息的残差(residual)特征。并基于深度至空间（depth to space，D2S)操作进行特征向量的重排，以便于进行后续的特征融合。Specifically, on the one hand, the RefSR model can perform degradation modeling based on the low-resolution reference image and the low-resolution reference image under the new perspective. In this way, the degradation process of the low-resolution reference image and the low-resolution reference image under the new perspective is learned to obtain residual features that help repair the high-frequency information lost or distorted due to the degradation process. The feature vector is rearranged based on the depth to space (D2S) operation to facilitate subsequent feature fusion.
另一方面，RefSR模型可基于新视角下的低分辨率参照图像和高分辨率参照图像进行高频建模。例如，可对前述新视角下的低分辨率参照图像进行上采样(upsampling)以初步提升图像质量。进一步地，可对高分辨率参照图像以及上采样后的新视角下的低分辨率参照图像进行空间至深度(space to depth，S2D)操作以实现特征融合。进而，可基于编码器-解码器(encoder-decoder)架构进行高频建模。具体地，可根据编码器提取特征融合后的高分辨率参照图像以及新视角的低分辨率参照图像的高频特征向量。并将偏振参数拼接至该高频特征向量中，并通过解码器基于高频特征向量输出高频建模的结果。On the other hand, the RefSR model can perform high-frequency modeling based on the low-resolution reference image and the high-resolution reference image under the new perspective. For example, the low-resolution reference image under the aforementioned new perspective can be upsampled (upsampling) to preliminarily improve the image quality. Furthermore, the high-resolution reference image and the upsampled low-resolution reference image under the new perspective can be subjected to a space-to-depth (S2D) operation to achieve feature fusion. Furthermore, high-frequency modeling can be performed based on an encoder-decoder architecture. Specifically, the high-frequency feature vectors of the high-resolution reference image after feature fusion and the low-resolution reference image of the new perspective can be extracted according to the encoder. The polarization parameters are spliced into the high-frequency feature vector, and the decoder outputs the result of the high-frequency modeling based on the high-frequency feature vector.
进而，RefSR模型中的融合模块可将高频建模输出的结果以及退化建模输出的结果进行融合，以生成新视角下的超分辨率参照图像。可以理解，该新视角下的超分辨率参照图像具有偏振参数对应的高光细节。Furthermore, the fusion module in the RefSR model can fuse the results of the high-frequency modeling output and the results of the degradation modeling output to generate a super-resolution reference image under a new perspective. It can be understood that the super-resolution reference image under the new perspective has highlight details corresponding to the polarization parameters.
在此，基于对RefSR模型的多个训练迭代过程，RefSR模型输出的新视角下的超分辨率参照图像可具有从高分辨率参照图像中引入的高频信息，例如边缘细节、纹理细节等，并可呈现该高分辨率参照图像的偏振参数对应的高光效果。Here, based on multiple training iterations of the RefSR model, the super-resolution reference image under the new perspective output by the RefSR model can have high-frequency information introduced from the high-resolution reference image, such as edge details, texture details, etc., and can present the highlight effect corresponding to the polarization parameters of the high-resolution reference image.
进而，基于前述NeRF模型及RefSR模型的训练过程，图7c提供了一种基于训练完成的NeRF模型及RefSR模型生成具有高光细节的目标视角下的超分辨拍摄图像的过程示意图。Furthermore, based on the training process of the aforementioned NeRF model and RefSR model, FIG7c provides a schematic diagram of a process of generating a super-resolution image captured under a target perspective with highlight details based on the trained NeRF model and RefSR model.
参见图7c，在对拍摄图像进行视角及高光调节的过程中，可向训练好的NeRF模型输入基于前述步骤300或步骤500获取的拍摄图像，以及基于前述步骤301或步骤501获取的目标方向参数和/或目标偏振参数。可以理解，由于训练好的NeRF模型可基于输入的方向参数调整拍摄图像的视角，生成目标视角下的拍摄图像，可基于输入的偏振参数调整拍摄图像的高光细节，使得拍摄图像具有该偏振参数对应的高光效果。因此，将拍摄图像、目标方向参数、目标偏振参数输入训练好的NeRF模型后，可生成具有高光细节的目标视角下的低分辨率拍摄图像。Referring to FIG. 7c, in the process of adjusting the viewing angle and highlight of the captured image, the captured image obtained based on the aforementioned step 300 or step 500, and the target direction parameter and/or target polarization parameter obtained based on the aforementioned step 301 or step 501 can be input into the trained NeRF model. It can be understood that since the trained NeRF model can adjust the viewing angle of the captured image based on the input direction parameter to generate the captured image under the target viewing angle, the highlight details of the captured image can be adjusted based on the input polarization parameter, so that the captured image has the highlight effect corresponding to the polarization parameter. Therefore, after the captured image, the target direction parameter, and the target polarization parameter are input into the trained NeRF model, a low-resolution captured image under the target viewing angle with highlight details can be generated.
可以理解，基于拍摄图像、目标方向参数、目标偏振参数生成低分辨率拍摄图像的过程，与前述图7a中基于高分辨率参照图像、该高分辨率参照图像的方向参数、该高分辨率参照图像的偏振参数生成低分辨率参照图像的过程实质相同，在此不做赘述。It can be understood that the process of generating a low-resolution captured image based on the captured image, target direction parameters, and target polarization parameters is essentially the same as the process of generating a low-resolution reference image based on the high-resolution reference image, the direction parameters of the high-resolution reference image, and the polarization parameters of the high-resolution reference image in the aforementioned Figure 7a, and will not be repeated here.
并且，可向训练好的NeRF模型输入基于前述步骤302或步骤502确定的拍摄图像对应的高分辨率参照图像。具体地，可向该NeRF模型输入该高分辨率参照图像的位置参数、方向参数以及偏振参数以得到对应的低分辨率参照图像。在此，高分辨率参照图像的位置参数、方向参数以及偏振参数的具体内容，以及生成低分辨率参照图像的具体过程可参见前述对图7a的具体描述，在此不做赘述。Furthermore, a high-resolution reference image corresponding to the captured image determined based on the aforementioned step 302 or step 502 may be input into the trained NeRF model. Specifically, the position parameters, direction parameters, and polarization parameters of the high-resolution reference image may be input into the NeRF model to obtain a corresponding low-resolution reference image. Here, the specific contents of the position parameters, direction parameters, and polarization parameters of the high-resolution reference image, as well as the specific process of generating the low-resolution reference image, can be found in the aforementioned specific description of FIG. 7a, which will not be repeated here.
进而，可将前述具有高光细节的目标视角下的低分辨率拍摄图像、低分辨率参照图像、高分辨率参照图像、目标偏振参数输入训练好的RefSR模型中，以根据低分辨率参照图像及低分辨率拍摄图像通过RefSR模型中的退化模型输出残差特征、根据高分辨率参照图像及低分辨率拍摄图像结合目标偏振参数通过RefSR模型中的高频模型输出高频特征。进而，基于RefSR模型中的融合模块将残差特征及高频特征进行融合，得到具有高光细节的目标视角下的超分辨率拍摄图像。在此，输出残差特征、高频特征的具体内容可参见前述图7b中的具体描述，在此不做赘述。Furthermore, the aforementioned low-resolution captured images under the target perspective with highlight details, low-resolution reference images, high-resolution reference images, and target polarization parameters can be input into the trained RefSR model, so as to output residual features according to the low-resolution reference images and the low-resolution captured images through the degradation model in the RefSR model, and output high-frequency features according to the high-resolution reference images and the low-resolution captured images combined with the target polarization parameters through the high-frequency model in the RefSR model. Furthermore, the residual features and high-frequency features are fused based on the fusion module in the RefSR model to obtain a super-resolution captured image under the target perspective with highlight details. Here, the specific content of outputting residual features and high-frequency features can be found in the specific description in the aforementioned FIG. 7b, which will not be repeated here.
图8以本申请提供的图像处理方法适用的电子设备为手机100为例，示出了手机100的结构示意图。FIG8 shows a schematic structural diagram of a mobile phone 100 , taking a mobile phone 100 as an example in which the electronic device to which the image processing method provided in the present application is applicable.
如图8所示，手机100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线（universal serial bus，USB）接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块（subscriber identification module，SIM）卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L等。As shown in FIG8 , the mobile phone 100 may include a processor 110 , an external memory interface 120 , an internal memory 121 , a universal serial bus (USB) interface 130 , a charging management module 140 , a power management module 141 , a battery 142 , an antenna 1 , an antenna 2 , a mobile communication module 150 , a wireless communication module 160 , an audio module 170 , a speaker 170A , a receiver 170B , a microphone 170C , an earphone interface 170D , a sensor module 180 , a button 190 , a motor 191 , an indicator 192 , a camera 193 , a display screen 194 , and a subscriber identification module (SIM) card interface 195 , etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and the like.
可以理解的是，本发明实施例示意的结构并不构成对手机100的具体限定。在本申请另一些实施例中，手机100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器（application processor，AP），调制解调处理器，图形处理器（graphics processingunit，GPU），图像信号处理器（image signal processor，ISP），控制器，视频编解码器，数字信号处理器（digital signal processor，DSP），基带处理器，和/或神经网络处理器（neural-network processing unit，NPU）等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路（inter-integrated circuit，I2C）接口，移动产业处理器接口（mobile industryprocessor interface，MIPI），通用输入输出（general-purpose input/output，GPIO）接口等。In some embodiments, the processor 110 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, and the like.
手机100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。The mobile phone 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏（liquid crystal display，LCD），有机发光二极管（organic light-emittingdiode，OLED），有源矩阵有机发光二极体或主动矩阵有机发光二极体（active-matrixorganic light emitting diode，AMOLED），柔性发光二极管（flex light-emittingdiode，FLED），Mini-LED，Micro-LED，Micro-OLED，量子点发光二极管（quantum dot lightemitting diodes，QLED）等。在一些实施例中，手机100可以包括1个或N个显示屏194，N为大于1的正整数。The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-LED, Micro-LED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the mobile phone 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
手机100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。The mobile phone 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将该电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件（charge coupled device，CCD）或互补金属氧化物半导体（complementary metal-oxide-semiconductor，CMOS）光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，手机100可以包括1个或N个摄像头193，N为大于1的正整数。The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile phone 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当手机100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the mobile phone 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
视频编解码器用于对数字视频压缩或解压缩。手机100可以支持一种或多种视频编解码器。这样，手机100可以播放或录制多种编码格式的视频，例如：动态图像专家组（moving picture experts group，MPEG）1，MPEG2，MPEG3，MPEG4等。Video codecs are used to compress or decompress digital videos. Mobile phone 100 may support one or more video codecs. Thus, mobile phone 100 may play or record videos in various coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
NPU为神经网络（neural-network ，NN）计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现手机100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, the intelligent cognition of the mobile phone 100 can be realized, such as: image recognition, face recognition, voice recognition, text understanding, etc.
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。压力传感器180A的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器180A，电极之间的电容改变。手机100根据电容的变化确定压力的强度。当有触摸操作作用于显示屏194，手机100根据压力传感器180A检测该触摸操作的强度。手机100也可以根据压力传感器180A的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The mobile phone 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the mobile phone 100 detects the intensity of the touch operation based on the pressure sensor 180A. The mobile phone 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.
触摸传感器180K，也称“触控器件”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于手机100的表面，与显示屏194所处的位置不同。The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the mobile phone 100, which is different from the position of the display screen 194.
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。手机100可以接收按键输入，产生与手机100的用户设置以及功能控制有关的键信号输入。The key 190 includes a power key, a volume key, etc. The key 190 can be a mechanical key or a touch key. The mobile phone 100 can receive key input and generate key signal input related to the user settings and function control of the mobile phone 100.
图9以本申请提供的图像处理方法适用的电子设备为手机100为例，示出了手机100的软件结构框图。FIG9 takes a mobile phone 100 as an example of an electronic device to which the image processing method provided in the present application is applicable, and shows a software structure block diagram of the mobile phone 100 .
手机100的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本发明实施例以分层架构的Android系统为例，示例性说明手机100的软件结构。分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时（Android runtime）和系统库，以及内核层。The software system of the mobile phone 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the mobile phone 100. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, respectively, the application layer, the application framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.
应用程序层可以包括一系列应用程序包。The application layer can include a series of application packages.
如图9所示，应用程序包可以包括前述图像处理应用、拍摄应用等应用程序。As shown in FIG. 9 , the application package may include the aforementioned image processing application, shooting application, and other applications.
应用程序框架层为应用程序层的应用程序提供应用编程接口（applicationprogramming interface，API）和编程框架。应用程序框架层包括一些预先定义的函数。The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.
如图9所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。As shown in FIG. 9 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。该数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
电话管理器用于提供手机100的通信功能。例如通话状态的管理（包括接通，挂断等）。The phone manager is used to provide communication functions of the mobile phone 100, such as management of call status (including answering, hanging up, etc.).
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，电子设备振动，指示灯闪烁等。The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
系统库可以包括多个功能模块。例如：表面管理器（surface manager），媒体库（media libraries），三维图形处理库（例如：openGL ES），2D图形引擎（例如：SGL）等。The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as openGL ES), 2D graphics engine (such as SGL), etc.
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如: MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
2D图形引擎是2D绘图的绘图引擎。A 2D graphics engine is a drawing engine for 2D drawings.
内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动。The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
本申请实施例还提供了一种计算机程序产品，用于实现上述各实施例提供的图像处理方法。The embodiments of the present application also provide a computer program product for implementing the image processing methods provided in the above embodiments.
本申请公开的机制的各实施例可以被实现在硬件、软件、固件或这些实现方法的组合中。本申请的实施例可实现为在可编程系统上执行的计算机程序模块或模块代码，该可编程系统包括至少一个处理器、存储系统（包括易失性和非易失性存储器和/或存储元件）、至少一个输入设备以及至少一个输出设备。The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program module or module code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and/or storage elements), at least one input device and at least one output device.
计算机程序模块或模块代码，可以应用于输入指令，以执行本申请描述的各功能并生成输出信息。可以按已知方式将输出信息应用于一个或多个输出设备。为了本申请的目的，处理系统包括具有诸如例如数字信号处理器（digital signal processor，DSP）、微控制器、专用集成电路（application specific integrated circuit，ASIC）或微处理器之类的处理器的任何系统。A computer program module or module code may be applied to input instructions to perform the functions described herein and generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.
模块代码可以用高级模块化语言或面向对象的编程语言来实现，以便与处理系统通信。在需要时，也可用汇编语言或机器语言来实现模块代码。事实上，本申请中描述的机制不限于任何特定编程语言的范围。在任一情形下，该语言可以是编译语言或解释语言。Module code can be implemented with high-level modular language or object-oriented programming language to communicate with the processing system. When necessary, module code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.
在一些情况下，所公开的实施例可以以硬件、固件、软件或其任何组合来实现。所公开的实施例还可以被实现为由一个或多个暂时或非暂时性机器可读（例如，计算机可读）存储介质承载或存储在其上的指令，其可以由一个或多个处理器读取和执行。例如，指令可以通过网络或通过其他计算机可读介质分发。因此，机器可读介质可以包括用于以机器（例如，计算机）可读的形式存储或传输信息的任何机制，包括但不限于，软盘、光盘、光碟、磁光盘、只读存储器（Read Only Memory，ROM）、随机存取存储器（random access memory，RAM）、可擦除可编程只读存储器（erasable programmable read only memory，EPROM）、电可擦除可编程只读存储器（electrically erasable programmable read-only memory，EEPROM）、磁卡或光卡、闪存、或用于利用因特网以电、光、声或其他形式的传播信号来传输信息（例如，载波、红外信号数字信号等）的有形的机器可读存储器。因此，机器可读介质包括适合于以机器（例如计算机）可读的形式存储或传输电子指令或信息的任何类型的机器可读介质。In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).
在说明书对“一个实施例”或“实施例”的引用意指结合实施例所描述的具体特征、结构或特性被包括在根据本申请实施例公开的至少一个范例实施方案或技术中。说明书中的各个地方的短语“在一个实施例中”的出现不一定全部指代同一个实施例。References to "one embodiment" or "an embodiment" in the specification mean that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least one exemplary implementation or technology disclosed according to the embodiment of the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily all refer to the same embodiment.
本申请实施例的公开还涉及用于执行文本中的操作装置。该装置可以专门处于所要求的目的而构造或者其可以包括被存储在计算机中的计算机程序选择性地激活或者重新配置的通用计算机。这样的计算机程序可以被存储在计算机可读介质中，诸如，但不限于任何类型的盘，包括软盘、光盘、CD-ROM、磁光盘、只读存储器（ROM）、随机存取存储器（RAM）、EPROM、EEPROM、磁或光卡、专用集成电路（ASIC）或者适于存储电子指令的任何类型的介质，并且每个可以被耦合到计算机系统总线。此外，说明书中所提到的计算机可以包括单个处理器或者可以是采用针对增加的计算能力的多个处理器涉及的架构。The disclosure of the embodiments of the present application also relates to an operating device for executing the text. The device can be constructed specifically for the required purpose or it can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer-readable medium, such as, but not limited to, any type of disk, including a floppy disk, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic or optical card, an application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, and each can be coupled to a computer system bus. In addition, the computer mentioned in the specification may include a single processor or may be an architecture involving multiple processors for increased computing power.
另外，在本说明书所使用的语言已经主要被选择用于可读性和指导性的目的并且可能未被选择为描绘或限制所公开的主题。因此，本申请实施例公开旨在说明而非限制本文所讨论的概念的范围。In addition, the language used in this specification has been primarily selected for readability and instructional purposes and may not be selected to describe or limit the disclosed subject matter. Therefore, the present application embodiment disclosure is intended to illustrate rather than limit the scope of the concepts discussed herein.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410148313.5ACN117689545B (en) | 2024-02-02 | 2024-02-02 | Image processing method, electronic device and computer readable storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410148313.5ACN117689545B (en) | 2024-02-02 | 2024-02-02 | Image processing method, electronic device and computer readable storage medium |
| Publication Number | Publication Date |
|---|---|
| CN117689545A CN117689545A (en) | 2024-03-12 |
| CN117689545Btrue CN117689545B (en) | 2024-07-16 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410148313.5AActiveCN117689545B (en) | 2024-02-02 | 2024-02-02 | Image processing method, electronic device and computer readable storage medium |
| Country | Link |
|---|---|
| CN (1) | CN117689545B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111027415A (en)* | 2019-11-21 | 2020-04-17 | 杭州凌像科技有限公司 | Vehicle detection method based on polarization image |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7149262B1 (en)* | 2000-07-06 | 2006-12-12 | The Trustees Of Columbia University In The City Of New York | Method and apparatus for enhancing data resolution |
| CN109118460B (en)* | 2018-06-27 | 2020-08-11 | 河海大学 | A method and system for synchronizing processing of spectroscopic polarization spectrum information |
| WO2021071992A1 (en)* | 2019-10-07 | 2021-04-15 | Boston Polarimetrics, Inc. | Systems and methods for augmentation of sensor systems and imaging systems with polarization |
| CN115565212B (en)* | 2022-01-20 | 2023-08-04 | 荣耀终端有限公司 | Image processing method, neural network model training method and device |
| CN115424115B (en)* | 2022-09-11 | 2025-06-24 | 西北农林科技大学 | A transparent target detection method based on polarization imaging and deep learning |
| CN115661320B (en)* | 2022-11-28 | 2023-07-25 | 荣耀终端有限公司 | Image processing method and electronic device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111027415A (en)* | 2019-11-21 | 2020-04-17 | 杭州凌像科技有限公司 | Vehicle detection method based on polarization image |
| Publication number | Publication date |
|---|---|
| CN117689545A (en) | 2024-03-12 |
| Publication | Publication Date | Title |
|---|---|---|
| CN112712470B (en) | Image enhancement method and device | |
| CN113099146B (en) | Video generation method and device and related equipment | |
| CN113538227B (en) | Image processing method based on semantic segmentation and related equipment | |
| CN113938602B (en) | Image processing method, electronic device, chip and readable storage medium | |
| CN113709355B (en) | Sliding zoom shooting method and electronic equipment | |
| CN113706414A (en) | Training method of video optimization model and electronic equipment | |
| CN114463191A (en) | Image processing method and electronic equipment | |
| CN115272138B (en) | Image processing method and related device | |
| CN113705665A (en) | Training method of image transformation network model and electronic equipment | |
| CN117499779B (en) | Image preview method, device and storage medium | |
| WO2021180046A1 (en) | Image color retention method and device | |
| CN114926351B (en) | Image processing method, electronic device, and computer storage medium | |
| CN116916151A (en) | Photography methods, electronic equipment and storage media | |
| CN113452969A (en) | Image processing method and device | |
| CN117132515A (en) | Image processing method and electronic device | |
| CN115643485B (en) | Photography methods and electronic equipment | |
| CN115587938A (en) | Video distortion correction method and related equipment | |
| CN113518172B (en) | Image processing method and device | |
| CN117499797B (en) | Image processing method and related equipment | |
| CN108495038A (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN116055895B (en) | Image processing method and its device, chip system and storage medium | |
| CN117689545B (en) | Image processing method, electronic device and computer readable storage medium | |
| CN118264766A (en) | A video generation method and related equipment | |
| CN115460343A (en) | Image processing method, device and storage medium | |
| CN113891008A (en) | Exposure intensity adjusting method and related equipment |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address | Address after:Unit 3401, unit a, building 6, Shenye Zhongcheng, No. 8089, Hongli West Road, Donghai community, Xiangmihu street, Futian District, Shenzhen, Guangdong 518040 Patentee after:Honor Terminal Co.,Ltd. Country or region after:China Address before:3401, unit a, building 6, Shenye Zhongcheng, No. 8089, Hongli West Road, Donghai community, Xiangmihu street, Futian District, Shenzhen, Guangdong Patentee before:Honor Device Co.,Ltd. Country or region before:China |